SynthID-Text Watermarking Alters LLM Safety Behavior

SynthID-Text Watermarking Changes How LLMs Handle Harmful Requests

SynthID-Text watermarking does more than mark text as machine-made. Google open-sourced the method, and Anthropic has said future Claude models will use it, but new research from Lasso Security shows it can shift a model's willingness to refuse harmful prompts, especially when an attacker pairs those prompts with injection tricks. It changes refusal behavior. That's the finding. And it's not just about labeling output as machine-made, because the research shows the watermark can influence whether a model refuses a harmful prompt at all, particularly when injection tricks are added alongside those prompts.

The finding lands now. AI platforms are rolling out watermarking schemes to comply with a new European Union law, and they're doing it under pressure. Provenance is the stated goal. It's the ability to trace output back to the system that produced it. But what the watermark quietly alters along the way, the subtle changes it makes to the very output it's supposed to track and protect, is now the open question.

How the Watermark Actually Works

Watermarking embeds a signal. It lets anyone with the right key identify output as AI generated. SynthID-Text does this by adding a random seed generator, a sampling algorithm, and a scoring function to the normal next-word selection process, which means the whole pipeline changes just enough that the output carries a detectable trace. Instead of pulling from an arbitrary random number generator, the model draws on a secret key. So a top candidate like "cloudy" might become "overcast.

The centerpiece is a technique called tournament sampling. Picture a bracket. SynthID lines up large numbers of candidate tokens, assigns each a hidden probability score using the secret key, and runs them head to head. Higher score wins, advances, and the rounds continue until one token takes the final. Word selection stays random, but anyone with the key can later check a sequence and judge how likely it came from that pipeline.

Six Models, One Uncomfortable Pattern

Andrea Siposova is an AI security researcher at Lasso Security. She tested SynthID-Text's non-distortionary configuration through Hugging Face's unmodified SynthIDTextWatermarkLogitsProcessor, and this specific setup matters because it is the version researchers would actually use when they check whether a watermark changes what a model says. She ran harmful prompts through six open-weight models. But she didn't just run them once. She ran them once with the watermark active and once without, then compared what came back, so we've got a clear before-and-after picture of what the watermark does and doesn't change.

The watermark changed refusal behavior on bare harmful requests. The effect grew sharper when the same requests arrived wrapped in prompt injection, the technique where an attacker slips instructions into input to steer a model toward an action it should reject, such as leaking a password or other sensitive data.

"On several models, watermarking then makes the model more likely to answer harmful requests that it would otherwise refuse."

Instructions went unfollowed. That's the norm. But once watermarking was in place, some of those same instructions were, in some cases, carried out, which is the part that matters here. Siposova named it. Sampling drift.

When the Model Stops Just Talking

At the model level, the shift touches whether a refusal happens at all and whether it holds under injection. At the agent level, the stakes climb. The same sampled tokens that pick a word can also determine which tool an agent calls and what arguments it passes along. A weakened refusal matters far more when the model has hands to act with.

It changes behavior. Siposova said that. Compared to the same models without watermarking, this shift is definitely going to show up, especially when we place it under adversarial conditions or make these models call tools when they're powering an agent. Watermarking is made to not be perceptible to a reader. But we know that when we're changing anything about what the model is generating, it's going to cause some tradeoffs, and it's going to show up somewhere.

One detail complicates any simple story about the technique. Responses varied. It's about the key. Same model, same prompt, different key, different outcome, and that's the part we can't ignore, because it's not the prompt that shapes what you get back but the secret key you happen to be holding when you ask.

What the Experiment Did Not Cover

The tests didn't examine how Claude models respond under the watermarking. They covered six open-weight models, chosen because the researcher could toggle token sampling during tournament sampling while holding other settings fixed, a decision that shaped what the study could actually show about these systems. And the work also exercised the Hugging Face implementation of SynthID-Text tournament sampling, not the specific implementation Claude models will run. That distinction matters. It's not the same thing.

Those boundaries matter for anyone tempted to read the results as a verdict on Anthropic's rollout. They do not erase the pattern either. At least some forms of this watermarking approach can move model and agent safety, and the researcher's own conclusion points one direction.

Red-team hacking exercises need to stress-test platforms. They confirm they behave as expected once SynthID is deployed. But without that step, a provenance feature meant to build trust, one that's supposed to sit quietly and do its job, could quietly reshape the guardrails it was never supposed to touch. And that's the risk. We've got to test it.

• SynthID-Text uses a secret key during token sampling, with tournament sampling narrowing candidates round by round until one wins.
• Six open-weight models were tested with and without the watermark, plus a prompt injection variant.
• Refusal behavior weakened, and the effect intensified under injection.
• Agent tool calls and their arguments depend on the same sampled tokens, so the drift can extend past text.

Platforms weighing SynthID-Text watermarking now face a narrow but real choice: ship the provenance layer and assume safety behavior carries over, or measure the drift first and find out what the key changes.