The Children’s Commissioner is calling on the government to introduce a total ban on apps that use AI to generate sexually explicit ‘deepfake’ images of children.

Dame Rachel de Souza’s new report exposes how Generative AI is being misused to create sexually explicit deepfake images of real people, and the alarming effect these ‘nudification’ tools are already having on children’s safety, well-being, and participating online.

“In our lifetime, we have seen the rise and power of Artificial Intelligence – once the stuff of science fiction – to shape the way we learn, connect and experience the world,” Dame de Souza said.

“It has enormous potential to enhance our lives, but in the wrong hands it also brings alarming risks to children’s safety online.”

Despite being relatively new technology, GenAI – which is often free to use and widely available – has supercharged the growth of these tools.

While it is illegal to create or share a sexually explicit image of a child, the technology enabling them remains legal – and it is no longer confined to corners of the dark web but now accessible through large social media platforms and search engines.

The Commissioner’s new report analyses the threat of nudifcation technology, assessing its use online and the impact on children.

The Commissioner found that children’s biggest concerns revolved around the high risk of harm, the change in girls’ online participation, and the easy access of harmful tools.

“Children have told me they are frightened by the very idea of this technology even being available, let alone used. They fear that anyone – a stranger, a classmate, or even a friend – could use a smartphone as a way of manipulating them by creating a naked image using these bespoke apps,” de Souza said.

Children want the government to take action to tackle the misuse of AI tools, with many questioning why these tools were made in the first place, given their penchant for harm.

“Girls have told me they now actively avoid posting images or engaging online to reduce the risk of being targeted by this technology – we cannot allow sit back and allow these bespoke AI apps to have such a dangerous hold over children’s lives,” de Souza added.

Recommended reading

• Deepfake Fraud Explodes 2,000% In Three Years
• Is It Now Practically Impossible To Identify Deepfakes?
• UK Gov Cracks Down on Explicit Deepfake Creators
• Teens Now Fear Deepfakes More Than Real Nudes

The Children’s Commissioner is therefore calling for urgent action, including banning bespoke nudification apps that enable users to generate sexually explicit images of real people.

Further, the government should make specific legal responsibilities for developers of GenAI tools to identify and address the risks their products pose and to mitigate the risks to children.

The Commissioner is also calling for effective systems to remove sexually explicit deepfake images of children from the internet, as well as recognising deepfake sexual abuse as a form of violence against women and girls and taking it seriously in law and policy.

“The online world is revolutionary and quickly evolving, but there is no positive reason for these particular apps to exist. They have no place in our society,” de Souza said.

“Tools using deepfake technology to create naked images of children should not be legal and I’m calling on the government to take decisive action to ban them, instead of allowing them to go unchecked with extreme real-world consequences.”