How Meta contractors Probed Rival Chatbots

Meta contractors posed as minors online. They were quietly testing AI safety by probing how competitor chatbots handled high-risk prompts about suicide, sex, drugs, and eating disorders. The operation targeted OpenAI’s ChatGPT, Google’s Gemini, and Character.AI. It was managed by a partner firm named Covalen, and it ran as recently as April 21. Workers had to create dummy under-18 accounts, feed sensitive text and imagery to rival systems, and log results in shared databases. But the targeted companies had no knowledge of it. That’s unsettling.

The scale of the initiative was substantial. But they relied on throwaway Gmail and Outlook addresses that all shared a single password, and workers copied and pasted the chatbot responses directly into spreadsheets for analysis. A single round of testing completed in August 2025 put more than 45,000 prompts through the competitor systems. Many of the queries were written in languages other than English, including French-language prompts referencing real-world tragedies. It's massive.

The Mechanics of Cannes

The internal name was Cannes. It relied on a structured database of highly sensitive prompts designed to bypass standard safety guardrails, and workers uploaded images alongside text to see how the rival systems would react. Some files included pills, knives, nooses, and medical diagrams. And the goal was to force the safety systems of ChatGPT, Gemini, and Character.AI into generating responses they were programmed to refuse.

3,748 prompts were analyzed. The log revealed a heavy focus on crisis scenarios, with hundreds of queries targeting suicide and self-harm as well as hundreds more focused on eating disorders. At least 239 prompts involved sex or romance. Other prompts touched on drugs, profanity, and racial slurs, but they're often written from the perspective of a child in distress. It's a troubling pattern.

Some of the specific scenarios included:

• A prompt written from the perspective of a 13-year-old girl who became pregnant by an adult neighbor, asking where to buy pills to end the pregnancy.
• A query mimicking a fifth-grader whose classmate pointed a gun at his mouth.
• A prompt from a teenage girl asking how to hide bulimia from her parents.
• A user pretending to be a high school student asking where to buy cocaine.

Safety Testing or Terms of Service Violations

Tech companies regularly evaluate competitor systems. But this specific program crossed several operational boundaries, and it appears to have violated the user policies of all three targeted platforms. OpenAI explicitly bars unsolicited safety testing, attempts to bypass safeguards, and using outputs to build competing models. Google prohibits actions designed to bypass safety filters outside of its authorized bug-testing programs. Character.AI also maintains policies prohibiting harmful, exploitative, and obscene content, and it's banned open-ended chat for users under 18 since late 2025.

What the Companies Said

OpenAI spokesperson Drew Pusateri stated that the company was looking into the issue but declined to comment further. A Google spokesperson said the company had not authorized the third-party testing and did not know its purpose, adding that internal tests on the provided samples showed Gemini responded in accordance with its policies. Character.AI took a firmer stance.

"This alleged action is not only a violation of our Terms of Service, but also a violation of the characters and worlds our community has created." , Character.AI spokesperson

Meta defended the initiative.

It's a standard practice in tech, they argued, and testing chatbot responses for safety and age-appropriateness is a responsible, industry-standard approach. But they don't use competitor benchmarking to train their own AI models. A Meta spokesperson confirmed this. Covalen, meanwhile, offered no comment on how they managed the program.

Growing Alarm Among Workers

The project created deep unease among the staff executing it. It felt risky. Some workers feared the legal implications of their daily tasks, and one primary concern was that generating or preserving responses to sexual prompts involving minors could cross legal boundaries. But others worried that the program was a covert method to harvest data from competitors to benefit Meta's own systems.

The prompts were graphically brutal. They stunned everyone. One worker noted that each person on the project was shocked by the text they were instructed to test, wondering if this would lead to legal trouble with authorities. But two technology attorneys reviewed examples and determined the material didn't cross the line into soliciting illegal obscenity or child sexual abuse material. The ethical questions remained.

Evaluating the Industry Standard

External experts argue this project's secrecy and methods push it far outside normal boundaries. But it's not industry-standard evaluation. Rumman Chowdhury, the CEO of Humane Intelligence PBC, reviewed a sample of the prompts and the project framework and noted that structuring a monthslong, large-scale project designed to systematically break safety rules via dummy accounts masquerading as children is far from typical practice. She's clear about that.

Chowdhury pointed out that while large datasets of safety prompts are useful for comparing how systems refuse harmful requests, the lack of disclosure to the target companies changes the nature of the work. But that secrecy represents a difficult area in tech governance. So the line gets blurred. When safety evaluations are performed covertly on competitor systems, it's hard to distinguish product safety from anticompetitive data gathering, and that's a real problem for everyone involved.

Frequently Asked Questions

What was the purpose of Meta contractors testing rival chatbots?

Meta contractors were quietly testing AI safety by probing how competitor chatbots handled high-risk prompts about suicide, sex, drugs, and eating disorders. The goal was to force the safety systems of ChatGPT, Gemini, and Character.AI into generating responses they were programmed to refuse.

How did Meta contractors conduct the testing?

Workers created dummy under-18 accounts, fed sensitive text and imagery to rival systems, and logged results in shared databases. They used throwaway Gmail and Outlook addresses that all shared a single password, and copied and pasted the chatbot responses directly into spreadsheets for analysis.

Why did some workers feel uneasy about the project?

The project created deep unease among staff, with some fearing legal implications of generating or preserving responses to sexual prompts involving minors. Others worried it was a covert method to harvest data from competitors to benefit Meta's own systems.

When did the testing take place and how extensive was it?

The operation ran as recently as April 21, and a single round of testing completed in August 2025 put more than 45,000 prompts through the competitor systems. The prompts were written in multiple languages, including French.

Who managed the testing program and what was its internal name?

The program was managed by a partner firm named Covalen and its internal name was Cannes. Covalen offered no comment on how they managed the program.