The online network X has caused significant anger after individuals employed its AI chatbot Grok to alter images of females by taking off their attire.
Numerous cases have surfaced where the tool has been used to virtually strip women to show them in swimwear without consent, as well as to put them in inappropriate scenarios.
XAI, the organization behind Grok, offered no comment to questions, aside from an automatically-generated message stating "legacy media lies".
A Home Office spokesperson stated that legislation is being brought forward to ban image stripping software. Under a new criminal offence, anyone who provided such technology would "receive a prison sentence and significant fines".
"X or Grok could prevent these forms of misuse if they wanted to," said Clare McGlynn, adding they "seem to have a lack of consequences."
The regulator Ofcom said tech firms must "assess the risk" of people accessing illegal content on their networks, but did not confirm whether it was currently looking into X or Grok in regarding AI-generated images.
Grok is a no-cost AI assistant—with some premium functions—which answers X users' queries when they reference it in a tweet.
It is often used to provide reactions or further context to other posters' comments, but people on X are also can modify an uploaded image through its AI image editing feature.
The tool has been criticised for allowing users to create photos and videos with explicit content and sexually suggestive content, and it was earlier accused of making sexually explicit material of celebrities.
"The platform has been permitting the creation and spread of these images for a long time without taking any measures and we have not seen any challenge by regulators," Professor McGlynn added.
XAI's own acceptable use policy bans "portraying images of people in a pornographic way".
In a statement, Ofcom said it was unlawful to "create or share deepfakes or illegal content" and clarified this applied to explicit AI images generated with AI.
It said services such as X were required to take "appropriate measures" to "reduce the risk" of users encountering illegal material on their platforms and to remove it promptly when they become aware of it.
Elena Hartwell is a tech enthusiast and lifestyle writer exploring the intersection of innovation and well-being.