The global outcry over the sexualisation and nudification of photographs – including of children – by Grok, the chatbot developed by Elon Musk’s artificial intelligence (AI) company xAI, has led to urgent discussions about how such technology should be more strictly regulated.
But to what extent can technology also be used to prevent this explosion in the generation and sharing of deepfake content of real people, without their knowledge or consent?
On January 10, Indonesia became the first country to announce it was temporarily blocking access to Grok, followed soon after by Malaysia. Other governments, including the UK’s, have promised to take action against the chatbot and its related social media site X (formerly Twitter), on which the sexualised images have been shared.
But while outright national bans can limit casual use of the chatbot, such bans are easily bypassed using virtual private networks (VPNs) or alternative routing services. These mask the user’s real location and make it appear they originate from a location that allows access to the service.
As a result, country-level bans tend to reduce visibility rather than eliminate access. Their primary impact is symbolic and regulatory, placing pressure on companies such as xAI rather than preventing determined misuse. And content generated elsewhere can still circulate freely across borders via encrypted social media platforms and on the dark web.
In response to the controversy, X moved Grok’s image-generation features behind a paywall, making them only available to subscribers. X subsequently posted that it takes “action against illegal content on X, including child sexual abuse material, by removing it, permanently suspending accounts, and working with local governments and law enforcement as necessary”. Grok itself apologised for “the incident”, describing it as a “serious lapse”.
Some people are saying Grok is creating inappropriate images.
But that’s like blaming a pen for writing something bad.
A pen doesn’t decide what gets written. The person holding it does.
Grok works the same way. What you get depends a lot on what you put in.
Think about it!
— DogeDesigner (@cb_doge) January 3, 2026
How the technology works
While not all chatbots have image generation capabilities, most of the mainstream providers including OpenAI, xAI, Meta and Google provide this service.
Modern AI image generators are typically built using diffusion models, which are trained by taking real images and gradually adding random visual distortion, known as noise, until the original image is no longer recognisable. The model then learns how to reverse this process, step by step, reconstructing an image by removing noise.
Over time, it will learn statistical patterns representing faces, bodies, clothing, lighting and other visual features. These patterns are organised within the model so that visually similar concepts sit close together. Because clothed and unclothed human bodies share very similar shapes and structures, the changes required to move between them can be relatively small.
So, when an existing image is used as the starting point and identity-preserving features are retained, transforming a clothed photograph into an unclothed one becomes technically straightforward. Of course, the AI model itself has no understanding of identity, consent or harm. It simply produces images that resemble what it has learned, in response to user requests.
However, after the core model has been trained, companies can apply “retrospective alignment” – rules, filters and policies that are layered on top of the trained system to block certain outputs and align its behaviour with the company’s ethical, legal and commercial principles.
But retrospective alignment does not remove capability; it simply limits what the AI image generator is allowed to output. Those limits are primarily a design and policy choice made by the company operating the chatbot, although these may also be shaped by legal or regulatory requirements imposed by governments – for example, requiring companies to disable or restrict certain features such as identity-preserving image generation.
Large, centrally hosted social media platforms could also play an important role here. All have the power to restrict the sharing of sexual imagery involving real people, and to require explicit consent mechanisms from those featured in images. But to date, the big tech companies have tended to drag their feet when it comes to labour-intensive moderation of their users’ content.
‘Jailbreaking’
Research by Nana Nwachukwu, a PhD candidate at Trinity College Dublin’s Centre for AI-Driven Digital Content Technology, highlighted the frequency of requests for sexualised images on Grok. Other research has estimated that before the service went behind a paywall, up to 6,700 undressed images were being produced every hour.
This has prompted regulatory scrutiny in Europe and beyond. French officials described some outputs as manifestly illegal and referred them to prosecutors. The UK’s communications watchdog, Ofcom, has launched an investigation into X and xAI over the issue.
But this problem is not limited to one platform. In early 2024, non-consensual AI-generated sexual images of Taylor Swift, produced using publicly available tools, spread widely on X before being removed because of a combination of legal risk, platform policy enforcement and reputational pressure.
Some platforms explicitly market minimal or no content restrictions as a feature rather than a risk. It is simple enough to find websites promoting “unrestricted” image generation and privacy focused use, relying largely on open-source models and offering far fewer moderation controls than mainstream providers. Furthermore, there is an even larger number of self-hosted image and video generation tools where safeguards can be removed entirely.
While precise figures are unavailable, independent estimates suggest tens of millions of AI generated images are created daily across platforms, with video generation rapidly accelerating.
Another potential issue is that some AI chatbots, including Meta’s Llama and Google’s Gemma, can be downloaded onto computers (even those with relatively light processing power), after which these models are completely free of oversight or moderation when run offline.
Even tightly controlled systems can be bypassed through “jailbreaking” – a way of constructing prompts to fool the generative AI system into breaking its own ethics filters.
Jailbreaking exploits the fact that retrospective alignment systems depend on contextual judgment, rather than absolute rules. Rather than directly asking for prohibited content, users reframe their prompts so the same underlying action appears to fall within an allowed category such as fiction, education, journalism or hypothetical analysis.
An early example was known as the “grandma hack”, because it involved a recently deceased grandmother recounting experiences from her technical profession in chemical engineering, leading the model to generate step-by-step descriptions of prohibited activities.
Speed and scale
The internet already contains an enormous quantity of illegal and non-consensual sexual imagery, far beyond the capacity of authorities to remove. What generative AI systems change is the speed and scale at which new material can be produced. Law enforcement agencies have warned that this could lead to a dramatic increase in volume, overwhelming moderation and investigative resources.
Laws that may apply in one country can be ambiguous or unenforceable when services are hosted elsewhere. This mirrors longstanding challenges in policing child sexual abuse material and other illegal pornography, where content is frequently hosted offshore and rapidly redistributed. Once images spread, attribution and removal are slow and often ineffective.
By making countless millions more people aware of the possibility of sexualising and nudifiying images, high-profile AI chatbots make it possible for large numbers of users to generate illegal and abusive sexual imagery through simple plain English prompts. Estimates suggest Grok alone currently has anywhere from 35 million to 64 million monthly active users.
If companies can build systems capable of generating such imagery, they can also stop it being generated – in theory, at least. In practice, however, the technology exists and there is a demand for it – so this capability can never now be eliminated.
- is a Senior Lecturer in Computing and Information Systems, Cardiff Metropolitan University
- This article first appeared in The Conversation
