OpenAI and Google Take Steps to Avoid Abusive AI Imagery After Grok Scandal
The year 2026 began with a horrific example of the abuse potential of generative AI. Grok, the AI tool from Elon Musk’s xAI, was used to strip or nudify photos of people shared on X (formerly Twitter) at an alarming rate. Grok made 3 million sexualized images over an 11-day period in January, about 23,000 of which contained images of children, according to a study by the Center for Countering Digital Hate.
Now, competitors like OpenAI and Google are beefing up their security to avoid being the next Grok.
Security advocates and researchers have long been concerned about AI’s ability to create abusive and illegal content. Creating and sharing non-consensual intimate images, sometimes called revenge porn, was a big problem before AI. Generative AI only allows anyone to target and victimize people faster, easier, and cheaper.
On January 14, two weeks after the scandal began, the X’s Safety account confirmed in a post that it would suspend Grok’s ability to edit images on the social media app. Grok’s image generation capabilities are still available to paid subscribers in its standalone app and website. X did not respond to multiple requests for comment.
Most large companies have safeguards in place to prevent the type of large-scale abuse we saw possible with Grok. But cybersecurity is never a solid metal wall of protection; it’s a brick wall that’s constantly under repair. Here’s how OpenAI and Google tried to strengthen their security protections to circumvent Grok-like outages.
Learn more: AI Slop is destroying the Internet. These are the people who are fighting to save him
OpenAI fixes image generation vulnerabilities
At their core, all AI companies have policies prohibiting the creation of illegal images, such as child sexual abuse images, also known as CSAM. Many tech companies have implemented safeguards to completely prevent the creation of intimate images. Grok is an exception, with “spicy” modes for image and video.
Yet anyone intending to create non-consensual intimate images can try to trick AI models into doing so.
Researchers at Mindgard, an AI-focused cybersecurity company, discovered a vulnerability in ChatGPT that allowed users to bypass its guardrails and create intimate images. They used a tactic called “adversarial prodding,” in which testers attempt to poke holes in an AI with specially designed instructions. In this case, it involved tricking the chatbot’s memory with personalized prompts and then copying the nude style onto images of known people.
Mindgard alerted OpenAI of its findings in early February, and developer ChatGPT confirmed on February 10 – before Mindgard made its report public – that it had fixed the issue.
“We are grateful to the researchers who shared their findings,” an OpenAI spokesperson told CNET and Mindgard. “We acted quickly to fix a bug that allowed the model to generate these images. We appreciate this type of collaboration and remain focused on strengthening safeguards to keep users safe.”
This process is how cybersecurity often works. External red team researchers, like Mindgard, test software for weaknesses or workarounds, mimicking strategies that bad actors might use. When they identify security vulnerabilities, they alert the software vendor so that fixes can be deployed.
“To think that motivated users will not attempt to circumvent protection measures is a strategic miscalculation. Attackers repeat. Guardrails must assume perseverance,” Mindgard wrote in a blog post.
While tech companies boast that they can use their AI for any purpose, they must also strongly promise that they can prevent AI from being used for abusive purposes. For AI image generation, this means having a solid repertoire of prompts that will be declined and returned to users.
When OpenAI launched its Sora 2 video model, it promised to be more conservative in moderating its content for this very reason. But it’s important to ensure that your moderation practices are always effective, and not just at a product launch. This makes AI security testing an ongoing process for cybersecurity researchers and AI developers.
Watch this: AI is indistinguishable from reality. How to spot fake videos?
Google updates search reports
For its part, Google is taking steps to ensure that abusive images don’t spread as easily. The tech giant has simplified its process for requesting the removal of explicit images from Google Search. You can click the three dots in the upper right corner of an image, click report, then tell Google that you want the photo removed because it “shows a sexual image of me.” The new changes also allow you to select multiple images at once and track your reports more easily.
“We hope this new removal process will reduce the burden faced by victims of explicit, non-consensual images,” the company said in a blog post.
Asked what additional steps the company is taking to prevent AI-related abuse, Google brought CNET’s attention to its policy prohibiting the use of generative AI. Google’s policy, like many other tech companies, prohibits the use of AI for illegal or potentially abusive purposes, such as creating intimate images.
There are laws that aim to help victims when these images are shared online, like the Take It Down Act of 2025. But the scope of this law is limited, which is why many advocacy groups, like the National Center on Sexual Exploitation, are pushing for better rules.
There is no guarantee that these changes will prevent anyone from using AI for harassment and abuse. That’s why it’s so important that developers remain vigilant to ensure we are all protected and act quickly when reports or issues arise.
(Disclosure: Ziff Davis, CNET’s parent company, filed a lawsuit in 2025 against OpenAI, alleging that it violated Ziff Davis’ copyrights in the training and operation of its AI systems.)
