Why an invisible watermark needs a few hundred words to testify

TL;DR

• API customers can now turn on text watermarking. In the coming weeks OpenAI will also watermark eligible ChatGPT and Codex output in the EU.
• Kirchenbauer et al. push sampling toward a pseudo-random "green list" of tokens. To detect the watermark, they count green tokens and run a z-test.
• OpenAI's textGrain works differently, but the same statistics explain why short, edited or low-entropy text such as math is hard to flag.

What happened

On 5 October 2026, OpenAI described its approach to the text-provenance rules in Article 50 of the EU AI Act. According to Unite.AI, those obligations have applied since 2 August 2026. API customers anywhere can now opt in to text watermarking for select models. It is off by default. In the coming weeks, invisible watermarks will be added to text output for eligible ChatGPT and Codex users on all plans, in the EU only. Access to the detector will start with approved researchers and expert organizations, granted case by case. Unite.AI also quotes OpenAI's own caveat: the watermark "does not measure human contribution, does not establish ownership or responsibility, does not identify the user, and does not verify accuracy."

Unite.AI reports that the scheme is called textGrain and is described in a technical report by authors from the University of Pennsylvania, Yale and OpenAI. It reportedly works by "solving an optimal transport problem whose costs derive from Gumbel random variables, with Kullback–Leibler regularization." Every reported number comes from OpenAI's own evaluation:

• At a 1% false-positive rate, it detects about 80% of 200-token passages and 95% of 400-token passages. These figures are for content such as psychology, from the ELI5 dataset.
• Detection is "substantially lower" on mathematics.
• Replacing 10% of words with synonyms cut detection from about 92% to 66%. Replacing 25% cut it to 17%.

None of this has been independently verified. With the detector restricted, outsiders can't check it yet.

The paper in brief

Kirchenbauer and colleagues wanted LLM text to be detectable after the fact, from a short span, without access to the model. Their method works like this. Before each token is sampled, they hash the previous token and use the result to seed a random split of the vocabulary into a green list and a red list. Then they add a small bonus $\delta$ to the green tokens' logits. A human writer knows nothing about the split, so only about a fraction $\gamma$ of their tokens land on green by chance. Watermarked text has noticeably more, and a one-proportion z-test turns the green count into a p-value. They tested it on OPT-1.3B with 200-token generations and settings $\gamma = 0.5$, $\delta = 2$. The watermark was detected 98.4% of the time, at a threshold whose theoretical false-positive rate is $3\times10^{-5}$. The work stood out because detection is cheap and needs no model, the watermark can be added to any sampler without retraining, and the analysis uses textbook statistics, including a precise account of why low-entropy text resists watermarking.

The math

Setup

At step $t$, the language model outputs a vector of logits $l^{(t)}$ over the vocabulary $V$. A hash of the previous token $s^{(t-1)}$ seeds a random number generator. That generator picks a green list $G$ containing $\gamma|V|$ tokens; the rest form the red list $R$. In the private version, the hash is replaced by a keyed pseudorandom function of the previous $h$ tokens. Detection gets only the text. It outputs a decision and a p-value.

Step 1: tilt the softmax

$$\hat{p}^{(t)}_k = \frac{\exp\left(l^{(t)}_k + \delta\,\mathbf{1}[k \in G]\right)}{\sum_{i \in V} \exp\left(l^{(t)}_i + \delta\,\mathbf{1}[i \in G]\right)}$$

This is the paper's two-case formula from Algorithm 2 written in one line. $\hat{p}^{(t)}_k$ is the watermarked probability of token $k$, and $\mathbf{1}[k \in G]$ is 1 for green tokens and 0 for red ones. Adding $\delta$ to a logit multiplies that token's unnormalized probability by $\alpha = e^{\delta}$, and the softmax then renormalizes. It is the same move as a logit bias.

Two thought experiments show why this is clever:

• A free choice. Suppose two candidate words are equally likely, one green and one red. With $\delta = 2$, the green one now gets probability $e^2/(e^2+1) \approx 0.88$.
• A forced choice. Suppose "Barack" is followed by "Obama", "Obama" happens to be red, and its logit is 10 above any rival. A green rival's logit rises to only 8 below Obama's, so it gets probability around $e^{-8} \approx 0.0003$.

So the watermark writes itself where the model has freedom and steps aside where it doesn't.

Tilting the softmax toward the green list

interactive

• original model
• watermarked

green bonus δ2

chance of a green token, before:

43.6%

after:

85.1%

Step 2: count green tokens and test

$$z = \frac{|s|_G - \gamma T}{\sqrt{T\gamma(1-\gamma)}}$$

Here $T$ is the number of tokens scored and $|s|_G$ is how many of them are green. The null hypothesis $H_0$ is that the text was written without knowledge of the green-list rule. Under $H_0$, each token is green with probability $\gamma$, so $|s|_G$ is Binomial$(T,\gamma)$ with mean $\gamma T$ and variance $T\gamma(1-\gamma)$. The statistic $z$ counts how many standard deviations the green count sits above chance. It is the intro-stats test for a biased coin.

The paper rejects $H_0$ when $z > 4$, which has a one-sided p-value of about $3\times10^{-5}$. Detection needs only the text plus the hash or key; the model never runs. The null distribution depends only on $\gamma$ and the hash, not on $\delta$. That means a provider can change how hard it pushes in different contexts while keeping the same detector.

Worked example. Take $\gamma = 0.5$ and $T = 200$. Chance gives 100 green tokens, with standard deviation $\sqrt{50} \approx 7.07$, so reaching $z = 4$ takes about 128 green tokens.

How much text is needed? Here is a back-of-envelope calculation of my own, not from the paper. If watermarked text comes out green at rate $g$, its expected $z$ is about $(g-\gamma)\sqrt{T}/\sqrt{\gamma(1-\gamma)}$. With $\gamma = 0.5$:

• $g = 0.8$ gives $z \approx 0.6\sqrt{T}$, which reaches 4 at around $T \approx 44$ tokens.
• $g = 0.6$ gives $z \approx 0.2\sqrt{T}$, which needs about 400 tokens.

The evidence grows like $\sqrt{T}$. That is why detection rates are always quoted for a given passage length.

Step 3: spike entropy, the fuel for the signal

The green rate $g$ depends on how much freedom the model had at each step. The paper measures that freedom with a custom entropy:

$$S(p, z) = \sum_k \frac{p_k}{1 + z\,p_k}$$

Here $p$ is the model's next-token distribution. The modulus $z$ is a constant, not the z-score; the paper reuses the letter. $S$ is smallest, $1/(1+z)$, when all the probability sits on one token. It is largest, $N/(N+z)$, when the probability is spread evenly over $N$ tokens. You can read it as a softened count of tokens whose probability exceeds $1/z$. Like Shannon entropy, it measures how spread out a distribution is, but its shape makes the proof work.

Theorem 4.2 turns spike entropy into an expected green count:

$$\mathbb{E}\,|s|_G \ge \frac{\gamma\alpha T}{1 + (\alpha-1)\gamma}\, S^\star$$

Here $\alpha = e^{\delta}$. $S^\star$ is a lower bound on the sequence's average spike entropy, computed with modulus $(1-\gamma)(\alpha-1)/(1+(\alpha-1)\gamma)$. The theorem also bounds the variance, which lets you estimate the false-negative rate.

Plugging in $\gamma = 0.5$ and $\delta = 2$ gives $\alpha \approx 7.39$, a prefactor $\gamma\alpha/(1+(\alpha-1)\gamma) \approx 0.88$ and a modulus of about 0.76. Three cases:

• Every token forced. $S \approx 1/1.76 \approx 0.57$, so the bound on the green fraction is $0.88 \times 0.57 \approx 0.5$. That is exactly $\gamma$: no signal at all.
• Every token wide open. $S \approx 1$, so the green fraction is at least about 0.88.
• The paper's real generations. The average was $S = 0.807$, so the bound is $0.88 \times 0.807 \times 200 \approx 142$ green tokens. That is comfortably above the 128-token threshold.

Code, mathematics, memorized quotes and templated text sit near the forced end. Each forced token adds a chance-level coin flip to the count while still adding one to $T$, which dilutes $z$.

How much text the watermark needs

interactive

• expected z (Theorem 4.2 bound)
• detection threshold z = 4

Hover or drag across the chart to read values.

green bonus δ2green-list fraction γ0.5average spike entropy S0.807

green fraction (lower bound):

71.1%

tokens needed for z = 4:

91

Why edits hurt

In the $h = 1$ scheme, changing one token does damage twice. The new token is green only by chance. It also reseeds the green list for the next token, turning that token into a fresh coin flip too. The paper's worst case: an attacker who knows the scheme changes 200 of 1,000 tokens. That creates at most 400 red tokens and leaves 600 green, so $z \approx 6.3$, still a clear detection. An attacker who doesn't know the green lists does less damage: about one red token per change in expectation, when $\gamma = 0.5$. Each edit chips away at $z$ by a roughly fixed amount. A text that was only just above the threshold falls below it quickly.

What the results say

All experiments use OPT-1.3B with prompts from the news-like part of C4, about 200 tokens per generation and roughly 500 generations per setting.

• Detection. With $\gamma = 0.5$ and $\delta = 2$, 98.4% of generations were detected at $z = 4$ using multinomial sampling, and 99.6% using 4-way beam search. In the error tables, no human text was flagged at $z = 4$ under any setting.
• Theory vs. practice. The bound predicted at least 142.2 green tokens; the observed average was 159.5. Among the 375 of 500 generations above the 25th percentile of spike entropy, 100% were detected.
• Quality cost. A stronger watermark (larger $\delta$, smaller $\gamma$) raises perplexity. Beam search gets strong watermarks at little perplexity cost. With multinomial sampling, $\gamma = 0.1$ was Pareto-optimal.
• Paraphrase attack. The authors had T5-Large rewrite random spans. Replacing 10% of tokens lowered AUC by only 0.01. Replacing 30% removed much more of the watermark, but tripled perplexity.

Limitations. The authors are open about low-entropy failures. Table 1 shows memorized and templated outputs scoring $z = 2.40$ and $z = -1.13$. Repeated phrases can inflate green counts unless repeats are skipped. Prompting tricks such as the "emoji attack" can defeat the watermark, and homoglyphs and zero-width characters have to be normalized away before detection.

A careful reader should also notice three things:

• "Zero false positives" means zero in about 500 human texts. That data does not verify a $3\times10^{-5}$ rate.
• The models are small by today's standards.
• The threat model assumes the attacker's paraphrasing model is much weaker than the watermarked one. That assumption gets weaker as open models improve.

Connecting the dots

This paper is not textGrain. Coverage describes textGrain as built on Gumbel randomness, optimal transport and KL regularization, not on an additive logit bonus. Some background helps here. The "Gumbel-max trick" is a standard identity: add independent Gumbel noise to the log-probabilities, take the argmax, and you get an exact sample from the softmax. If a key generates the noise, whoever holds the key can reproduce the randomness. Kirchenbauer's $\delta$, by contrast, deliberately distorts the distribution. The coverage doesn't give enough detail to say how textGrain combines these pieces.

What does carry over is the logic of detection:

• A hypothesis test. OpenAI's "at a 1% false positive rate" is the same framing as the paper's $z > 4$: fix how often human text gets wrongly flagged, then report how much machine text gets caught.
• Length. Going from 80% detection at 200 tokens to 95% at 400 fits evidence that builds up token by token.
• Entropy. Weaker results on mathematics are the spike-entropy story: when word choice is forced, there's nothing to tilt.
• Edits. Synonym swaps destroy evidence token by token. OpenAI's reported drop (92% to 66% to 17%) is much steeper than the paper's T5 results, but the schemes, thresholds and attacks differ. Don't compare the numbers directly.

Section 5 of the paper also warns that a private detector behind an API needs monitoring, because unlimited queries can leak the green lists. OpenAI's case-by-case access fits that concern, although the coverage I read doesn't give it as OpenAI's reason.

Why it matters

For developers. If you have Article 50 obligations and use the API, watermarking is opt-in and off by default. Even when it's on, treat it as a probabilistic signal. Short replies, code and math may carry very little of it.

For people judging text. Teachers and moderators should remember the asymmetry. A positive result can carry a tiny p-value. A negative result says little, because light editing or low-entropy content can wipe out the signal. And by OpenAI's own account, a detected watermark does not identify a user or establish authorship.

For research and policy. Nemecek, Chaudhary and Ayday argue that unverifiability, not watermarking itself, is the main governance problem. The textGrain numbers show why: the only figures available come from OpenAI.

Things to watch:

• Whether approved researchers reproduce the textGrain numbers, especially for code, math and paraphrased text.
• Whether detector access widens beyond case-by-case approval, or moves toward detection that works across providers.

Go deeper

• A Watermark for Large Language Models (Kirchenbauer et al.), with code and demo
• OpenAI: Our approach to EU text provenance rules
• Unite.AI: OpenAI begins phased text watermarking under EU AI Act rules, the source for the textGrain details and numbers above
• textGrain: Entropy-Calibrated Watermarking for Language Model Text, OpenAI's technical report as linked by Unite.AI
• Watermarks Without Verification: AI Text Watermarking After the EU AI Act (Nemecek, Chaudhary, Ayday, 2026), a policy follow-up on auditing deployed watermarks