The social media network X has provoked major backlash after people utilized its AI chatbot Grok to alter images of females by taking off their clothing.
Numerous cases have surfaced where the system has been employed to remove clothing from women to render them in bikinis without permission, as well as to put them in inappropriate contexts.
XAI, the company behind Grok, offered no comment to requests for comment, aside from an auto-reply response stating "traditional news outlets lies".
A government representative stated that new laws is being brought forward to prohibit nudification software. Under a new law, anyone who supplied such tools would "face a jail term and significant penalties".
"X or Grok has the ability to stop these types of misuse if they desired to," said a law professor, adding they "operate with impunity."
The communications watchdog Ofcom said platforms must "evaluate the danger" of people accessing illegal content on their services, but stopped short of confirming whether it was at this time examining X or Grok in concerning artificial intelligence images.
Grok is a free artificial intelligence—with some premium functions—which responds to X users' queries when they tag it in a post.
It is often used to give reactions or further context to other posters' remarks, but people on X are also able to modify an posted photo through its artificial intelligence picture modification tool.
The chatbot has been condemned for allowing users to produce photos and footage with nudity and sexualised content, and it was previously accused of creating sexually explicit content of celebrities.
"The platform has been permitting the production and sharing of these pictures for months without taking any action and we have not seen any action by oversight bodies," Professor McGlynn added.
XAI's own terms of service bans "showing likenesses of individuals in a sexually explicit manner".
In a comment, Ofcom said it was against the law to "create or circulate deepfakes or child sexual abuse material" and confirmed this applied to explicit AI images made with AI.
It said networks such as X were mandated to take "necessary measures" to "reduce the risk" of users encountering illegal content on their platforms and to delete it promptly when they learn of it.
A certified personal trainer and nutrition coach with over 8 years of experience in holistic wellness and strength training.