# Tests suggest OpenAI's textGrain watermark weakens on short or edited text

> OpenAI's new textGrain watermark, built to help meet EU AI Act rules, often fails to survive editing or texts under 150 words, according to tests reported by IEEE Spectrum.

- Status: reported (One outlet, not independently verified yet.)
- Topic: Policy & Regulation
- Published: 2026-11-05T15:00:03+00:00
- Updated: 2026-11-07T12:05:32.967871+00:00
- Web page: https://ai-news.ikangai.com/story/tests-suggest-openai-s-textgrain-watermark-weakens-on-short-or-edited-text/
- Seen in: IEEE Spectrum
- Change: capability (New capability)
- Applies to: textGrain

## What changed

OpenAI released the textGrain text watermark in October, and tests reported by IEEE Spectrum suggest it often fails to survive editing and weakens on texts under 150 words.

OpenAI's textGrain watermark was released in October and is meant to help the company meet the European Union's new AI transparency rules. Tests reported by IEEE Spectrum found it often fails to survive editing and tends to break down on short text. In OpenAI's own figures, the detector caught the watermark in about 80 percent of 200-token passages, roughly 150 words. For 400-token passages, roughly 300 words, the figure was 95 percent. The detector is set to wrongly flag about one in 100 unwatermarked passages, a 1 percent false-positive rate.

Article 50 of the EU AI Act took effect on 2 August. It requires providers of systems that generate synthetic audio, images, video or text to mark outputs in a machine-readable format so they can be detected as artificially generated or manipulated. The law says technical solutions must be "effective, interoperable, robust and reliable as far as this is technically feasible." The EU's Code of Practice offers a voluntary way for providers to show compliance, and it does not require watermarks on outputs shorter than 200 tokens.

Text watermarks work at the token level. Language models choose each next word or word fragment from a set of probabilities that depends on the text already written. A watermark uses a secret key to generate pseudorandom values that nudge which token gets chosen, leaving a statistical pattern that a detector holding the same key can look for. Xiang Li, a coauthor of the textGrain technical report and a postdoctoral researcher at the University of Pennsylvania, said textGrain treats entropy, the measure of how unpredictable a model's word choices are, differently from other schemes. Developers specify how much unpredictability to allow, and textGrain adjusts to that target. Li described this as giving developers more control over the trade-off between a detectable signal and variety in the output.

Short passages seem to carry too little evidence of the watermark. Li called this a "basic limitation" of text watermarking that would be complicated to solve. He said that even with additional computation, short passages may not contain enough signal for reliable detection, while longer passages generally provide more. The 80 and 95 percent figures came from psychology questions drawn from the ELI5 database, which collects questions and answers from the r/explainlikeimfive subreddit.

Editing weakens the signal further. In OpenAI's evaluation of 400-token passages, replacing just 10 percent of words with synonyms reduced detection from 92 percent to 66 percent. Replacing up to 25 percent of words cut detection to 17 percent. Li explained that because each token's watermark depends on the tokens before it, changing one word can disrupt the signal at several positions.

The results also show what a detection does and doesn't establish. A positive result does not show how much a person contributed to a passage, and a negative result does not prove a text was human-written. OpenAI itself says watermarking alone is not enough to determine whether content is AI-generated. It says textGrain will complement other provenance tools, such as content credentials in metadata and the SynthID watermarks Google DeepMind has developed for images and audio.

OpenAI says textGrain matched or outperformed Google's SynthID Text in its own tests, but those results have not been independently verified. Thomas Souverain, a postdoctoral fellow on AI Ethics at the Institute of Advanced Education in Paris, said the early methodology behind textGrain seems reliable and could work better than competitors at identifying AI-generated text. He also said the legislation is open to interpretation. The AI Act's criteria still lack measurable thresholds, so he said it is too early to say whether textGrain meets legal requirements. If textGrain were found not to comply, Souverain said OpenAI could face a fine of up to 7 percent of its annual income.

Independent evaluation is still at an early stage. OpenAI is working with a limited group that includes John Thickstun, an assistant professor of computer science at Cornell University, and researchers at the Kempelen Institute of Intelligent Technologies (KInIT) in Bratislava, Slovakia. Thickstun and Dominik Macko, a senior researcher at KInIT, said they had only just gained access and had not yet begun evaluating the system. Macko said his team plans to test textGrain's robustness across subject areas, languages and text lengths. Souverain noted that so far there is only a technical study, with no peer-reviewed article, and that he does not yet have access to a precise protocol. He called on OpenAI to make textGrain open-source so researchers outside the company can try to break it. He said he would like the kind of access to the methods that Google has given with SynthID.

## Why it matters

Companies that generate or publish AI text in the EU may not be able to rely on a single watermark to show compliance, especially for short outputs or text that gets edited. An expert quoted by the article says a non-compliance finding could carry a fine of up to 7% of annual income.

## What you can do

Do not treat a text watermark as sufficient proof of AI-content labelling; plan for layered provenance measures and track independent evaluations of textGrain.

## Further reporting

- [EU's AI Watermark Rules Challenge OpenAI's TextGrain - IEEE Spectrum](https://spectrum.ieee.org/ai-watermark-openai-textgrain-europe)

---

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it.

Cite as: IKANGAI AI News, "Tests suggest OpenAI's textGrain watermark weakens on short or edited text", 2026-11-05, https://ai-news.ikangai.com/story/tests-suggest-openai-s-textgrain-watermark-weakens-on-short-or-edited-text/
