The rise of artificial intelligence (AI) has brought incredible innovation, but also complex challenges, notably concerning the creation and dissemination of non-consensual intimate imagery, often referred to as deepfakes. Recent scrutiny is focused on X, formerly known as Twitter, and its handling of AI-generated content, specifically its new Grok chatbot.Concerns are mounting that the platform may be facilitating the creation of illegal material, prompting investigations and calls for stricter regulation.
The Growing Concern over AI-Generated Abuse
Regulators are increasingly concerned about the potential for AI tools like Grok to be misused. The UK’s media regulator, Ofcom, is currently engaged in urgent talks
with X regarding these issues, signaling the seriousness of the situation. This follows reports suggesting the platform isn’t adequately addressing the creation of harmful content using its own AI technology. I’ve found that the speed at which these AI tools can generate realistic images and videos makes it incredibly arduous to control the spread of abuse.
Elon Musk, owner of X, has publicly stated that anyone utilizing Grok to produce illegal content will face the same repercussions as those who directly upload such material.He posted on X earlier this month: “Anyone using Grok to make illegal content will suffer the same consequences as if they upload illegal content.” However, critics argue that simply stating a policy isn’t enough; effective enforcement and proactive measures are crucial.
X’s safety team has also affirmed its commitment to removing illegal content, including Child Sexual Abuse Material (CSAM), through account suspensions and collaboration with law enforcement. They emphasize a zero-tolerance approach, but the sheer volume of content on the platform presents a significant challenge. Here’s what works best: a multi-layered approach combining AI detection, human moderation, and user reporting.
The legal landscape is evolving to address the threat of AI-generated abuse.In 2024, the UK government criminalized the creation of sexually explicit deepfakes, a direct response to high-profile cases like the targeting of taylor Swift with non-consensual AI-generated images. This legislation aims to provide victims with greater legal recourse and deter the creation of such harmful content.
Anyone using Grok to make illegal content will suffer the same consequences as if they upload illegal content
– Elon Musk (@elonmusk) January 3, 2026
The Impact on Individuals and the Entertainment Industry
The impact of AI-generated abuse extends beyond high-profile figures. Jenna Ortega, for example, reportedly left X after encountering AI-generated images of herself from her teenage years. This highlights the deeply personal and distressing nature of this type of abuse. Moreover, numerous artists, including Jess Glynne, Mumford & Sons, and Robert Smith, signed an open letter in 2024 advocating for stronger government protections against AI deepfakes
Keep reading