England’s Children’s Commissioner is calling for a total ban on artificial intelligence (AI) apps that generate sexually explicit images of children.
Dame Rachel de Souza said the government must outlaw “nudification” apps, which use AI to edit photos of real people to make them appear naked.
She warned that these tools are going unchecked and pose extreme real-world consequences for children.
In a report published Monday, Dame Rachel said the technology disproportionately targets girls and young women, with many apps designed to work only on female bodies.
The report found that girls are avoiding posting images or engaging online to reduce the risk of being targeted, similar to how they follow safety rules offline.
Children fear being targeted by strangers, classmates or even friends using AI tools found on popular search engines and social media platforms.
Dame Rachel said the rapid evolution of these tools makes it difficult to control the dangers they present.
She added that bespoke AI apps are having a dangerous impact on children’s lives and must be stopped.
Under the Online Safety Act, it is illegal to share or threaten to share explicit deepfake images.
In February, the government announced new laws to criminalise the possession, creation or distribution of AI tools designed to generate child sexual abuse material.
However, Dame Rachel said these measures do not go far enough.
Her office stated that all nudifying apps should be banned, not just those classified as child sexual abuse generators.
The Internet Watch Foundation (IWF) reported a 380% increase in AI-generated child sexual abuse cases in 2024, with 245 reports compared to 51 in 2023.
IWF Interim Chief Executive Derek Ray-Hill said these apps are being misused in schools and that the imagery quickly spreads out of control.
A spokesperson for the Department for Science, Innovation and Technology said creating, possessing or distributing AI-generated child sexual abuse material is “abhorrent and illegal.”
They added that under the Online Safety Act, platforms must remove such content or face significant fines.
The UK is the first country to introduce specific AI child sexual abuse offences.
Dame Rachel also urged the government to impose legal obligations on developers of generative AI tools to assess and mitigate risks to children.
She called for a systemic process to remove sexually explicit deepfake images of children from the internet.
She also wants deepfake sexual abuse to be recognised as a form of violence against women and girls.
Paul Whiteman, general secretary of the school leaders’ union NAHT, said members share the commissioner’s concerns and that the issue needs urgent review.
He warned that technology risks outpacing both the law and education.
On Friday, media regulator Ofcom released the final version of its Children’s Code, requiring platforms hosting pornography or harmful content to implement stricter age checks or face fines.
Dame Rachel criticised the code, saying it prioritises the business interests of tech companies over children’s safety.
Dame Rachel de Souza calls for a total ban on AI nudification apps
Report shows girls are disproportionately targeted by AI-generated abuse
IWF reports 380% rise in AI child sexual abuse cases in 2024
Source: BBC
