San Francisco orders Apple, Google to remove nudify apps

San Francisco orders Apple and Google to remove 13 "nudification" apps from their digital storefronts, a move aimed at combating the proliferation of AI tools that create explicit images from ordinary photos.

AI Photo Manipulation Apps Targeted

David Chiu has sent cease-and-desist letters to Apple and Google. Remove those apps immediately. They're using artificial intelligence to transform standard photographs into explicit content by removing clothing, altering facial features, and manipulating poses into sexualized positions or superimposing victims' faces onto nude bodies. But Chiu's office cited California laws that prohibit supporting services enabling deepfake pornography creation, and that's the legal foundation for these demands.

Harm to Victims Outlined

Chiu expressed deep concern over the widespread availability and use of this technology. It's hitting women and children the hardest. And he explained that these AI-generated images are frequently employed for bullying, humiliation, and threats, which can cause severe psychological tolls like reputation damage, mental health crises, and a profound loss of personal autonomy. Some individuals experience suicidal ideation.

"Generating non-consensual intimate images is illegal, harmful, and completely unacceptable," Chiu stated, emphasizing his office's stance on the matter.

App Store Revenue Under Scrutiny

San Francisco's attorney general's office estimates that Apple and Google may have profited significantly from these applications through app store fees, suggesting that the companies have potentially garnered millions of dollars by allowing these harmful tools to persist rather than implementing more robust detection mechanisms or taking proactive enforcement actions.

Company Responses and App Store Policies

Google confirmed it. Five apps flagged by Chiu's office have already been suspended from the Google Play store due to violations of their policies against harmful content. Dan Jackson, a Google spokesperson, stated the platform strictly prohibits apps containing sexual content and actively works to identify and remove such applications. Swift action's been taken. This includes suspending hundreds of violating apps and restricting search terms like "nudify." But Apple didn't immediately respond to requests for comment on the matter.

Evolving Tactics and Detection Challenges

A recent research paper highlighted that a large percentage of apps advertised as simple photo editing tools could, in fact, be used to create sexually explicit images. But app store detection systems are struggling. Some malicious applications are now evading removal by advertising only generic face-swapping features while concealing their nudifying capabilities. It's a growing concern.

The Grok Controversy and Broader Implications

xAI recently filed a lawsuit, acknowledging that its chatbot Grok generated child sexual abuse material and nonconsensual intimate imagery targeting adults. But it's going after the alleged user, not the machine. That raises tough questions. The company's own liability for such outputs now faces intensified scrutiny, sparking debates over whether AI chatbots like Grok violate policies against harmful content, and app stores are caught in the middle.

Future Enforcement and Legal Options

Chiu hopes Apple and Google will promptly remove the flagged apps. So they should also enhance their screening processes to prevent similar applications from appearing on their platforms in the future. It's a clear demand. But if the companies don't take satisfactory action, his office will explore all available legal avenues.

Specific App Removals Demanded

• Google was asked to remove five applications.
• Apple was asked to remove eight applications.

The attorney general's office refrained from naming specific apps in its communication to avoid inadvertently promoting them. Don't be fooled, though. It noted that at least one of those apps had surpassed one million downloads and advertised features for sexualizing images of women or creating "free and uncensored" videos.

Chiu stressed that the continued presence of any such apps is unacceptable given the illegal and harmful nature of generating non-consensual intimate imagery. But they can't stay. He anticipates that the pressure from these demands will lead to a broader tightening of enforcement across both app stores, beginning with the immediate removal of the identified nudification apps.