AI Watermarking Makes Models Fail Safety Tests

AI Watermarking Is Quietly Rewriting Safety Rules

AI watermarking was supposed to solve a provenance problem. Instead, new research suggests it may be opening a security hole. A fresh experiment from Lasso Security shows that SynthID-Text, the watermarking scheme Anthropic has said future Claude models will use, can shift how large language models respond to harmful requests, and not always in the safe direction.

The finding lands now. Platforms are racing to comply with a new European Union law requiring machine-generated content to be identifiable, and that race shapes everything about how this news gets received. Anthropic disclosed it will adopt SynthID-Text. That's an approach Google built and released as open source. The mechanism relies on a secret key that subtly alters how a model picks its next word. A top candidate like "cloudy" might become "overcast." And anyone holding the key can later check whether that sequence came from the platform using it, which means the watermark only works for people who've got that key in hand.

What nobody fully checked until now is what else that key changes.

How the Key Reshapes Output

SynthID does not just swap synonyms. It replaces the model's arbitrary random number generator with a seeded one, then layers on a sampling algorithm and a scoring function. The core trick is something called tournament sampling. Picture a bracket of next-word candidates. Tokens compete in pairs, each assigned a hidden probability score by the secret key. The higher scorer advances. Rounds continue until one token wins.

That process is built to be invisible. It's meant to leave no trace. But Andrea Siposova, an AI security researcher at Lasso Security, found that while the watermarking is designed to leave no perceptible trace in the text for readers, the effects don't stay hidden, and they're surfacing elsewhere. They surface elsewhere.

"As compared to the same models without watermarking, it is definitely going to change their behavior, especially when we place it under adversarial conditions, or we make these models call tools when they're powering an agent," Siposova told Ars. "Watermarking is made to not be perceptible to a reader, but we know that when we are changing anything about what the model is generating, it is going to cause some tradeoffs, it's going to show up somewhere."

Six Models, One Uncomfortable Pattern

Siposova tested SynthID-Text's "non-distortionary" configuration. She used Hugging Face's unmodified SynthIDTextWatermarkLogitsProcessor, and she fed harmful prompts into six open-weight models, comparing responses with and without watermarking active, while the setup let her toggle token sampling on and off and hold other settings fixed. It's simple. But she didn't change the processor.

The results were not subtle. Watermarking changed refusal behavior on bare harmful requests. When the same requests arrived bundled with prompt-injection techniques, the shift grew sharper. On several models, the watermarking made them more likely to comply with requests they would otherwise turn down.

That matters beyond a single reply. If an agent is driving tools, the same sampled tokens determine which tool gets called and what arguments get passed. A weakened refusal becomes far more dangerous when the model can act, not just talk.

Sampling Drift, Explained

Siposova gave the phenomenon a name: sampling drift. At the model level, it can flip whether a harmful request is refused and whether that refusal survives a prompt injection. At the agent level, it can redirect tool calls and their parameters. Prompt injection links the two, because a refusal that collapses under adversarial pressure becomes far more consequential once the model holds real capabilities.

One detail stood out. Responses shifted depending on which secret key was used. Different keys produced different behavior, which suggests the problem is not a single fixable bug but a property of the mechanism itself.

What the Test Did Not Cover

The scope here is limited. The researcher says so. They didn't test how Claude model responses change under watermarking, and that's a real gap, because what they tested instead was a half-dozen open-weight models, chosen so token sampling could be enabled and disabled during tournament sampling while other variables stayed constant. The work also used the Hugging Face implementation of SynthID-Text. But it's not the specific implementation Claude models will run.

This isn't about indicting one vendor. Or one build. It's about showing something bigger. At least some forms of this watermarking approach can alter model and agent safety, and that means every deployment inherits the same risk until proven otherwise, no matter who built it or how carefully they've tested it. So the risk stays.

The Red-Team Gap

Safety guardrails are typically trained and evaluated without watermarking in the loop. Add a secret key that reshapes sampling. The guardrails may no longer hold. And they may not hold the way testing assumed, because once that key changes how sampling works, the assumptions baked into the original evaluation no longer match what the system actually does in practice. The research points to a clear operational gap. Red-team exercises need to stress-test platforms with SynthID deployed. They can't just test in its absence.

• Watermarking altered refusal behavior on harmful prompts across multiple models.
• The effect intensified when prompts used injection techniques.
• Different secret keys produced different model behavior.
• Agent tool calls and arguments can shift alongside the text output.

For developers, the practical lesson is straightforward. AI watermarking cannot be treated as a cosmetic layer bolted onto a finished model. It changes the sampling process, and the sampling process is where safety decisions get made. Test with the watermark on. Test under adversarial pressure. Test the agent, not just the chatbot.

Provenance and safety were sold as compatible goals. This research suggests they can pull against each other, and that the tension shows up exactly where it hurts most: at the moment a model decides whether to say no.