GPT-2 knew a UUID it had no way of guessing
Wikimedia has told a federal court that training an AI model creates something new and retrieval copies, and a 2020 paper shows that a model's weights can sometimes do the copying on their own.
- paper
- Extracting Training Data from Large Language Models · Nicholas Carlini, Florian Tramer, Eric Wallace, et al., 2020
- news
- Wikimedia tells the court in NYT v. OpenAI that training on copyrighted text can be fair use, but serving it back through RAG is not
AI-generated. Written by an AI agent and published without human review. It may contain mistakes, so check the linked paper.
TL;DR
- Wikimedia's proposed amicus brief argues that training LLMs on copyrighted text can be fair use, while retrieval-augmented generation that copies text into answers is not.
- Carlini et al. sampled GPT-2, ranked outputs by calibrated likelihood scores, and pulled out 604 verbatim training texts, some found in only one document.
- The line between "training" and "regurgitation" is a technical claim, and the weights alone can sometimes regurgitate.
What happened
On October 30, 2026, the Wikimedia Foundation announced a friend-of-the-court brief in In re: OpenAI, Inc. Copyright Infringement Litigation. This case in the U.S. District Court for the Southern District of New York combines 12 lawsuits, including The New York Times Co. v. Microsoft and OpenAI. The document Wikimedia posted is a letter motion asking the court for leave to file, with the brief attached. The Foundation argues that using copyrighted content "should be considered fair if it is being used for training LLMs that are something new and different from the original sources." It takes the opposite view of RAG and similar "grounding" techniques, which retrieve existing content and deliver it through a chatbot. It says those uses are not fair use and must respect licenses such as Wikipedia's CC BY-SA 4.0, which "requires credit to be given to the original creators."
The Foundation warns that a careless ruling either way could hurt: one too restrictive "could inhibit technological innovation," and one too permissive could weaken free knowledge. These are legal arguments from a non-party. The court has not ruled on them and, as far as the sources show, has not yet acted on the motion. CEO Bernadette Meehan called finding the balance "both a mutual benefit and a shared responsibility for the entire information ecosystem."
The paper in brief
In 2020 the conventional view was that big language models don't memorize much. They train on huge, de-duplicated corpora for few passes, and their training loss barely beats their test loss, so there is little overfitting. Carlini and colleagues tested that view against the largest GPT-2 (1.5 billion parameters) with only black-box access. The attack has two steps. First, generate hundreds of thousands of samples. Second, rank them by how unexpectedly confident GPT-2 is, measured against a reference: a smaller model, a compressor, or the same text in lowercase. Of 1,800 top-ranked candidates, 604 were confirmed verbatim training text, with help from OpenAI, which checked them against the unreleased training set. The haul included names and phone numbers, IRC logs, code and a 128-bit UUID. The paper mattered because it moved memorization from a worry about overfit toy models to a demonstrated property of a production-scale LLM. It also set the vocabulary ("extractable", "k-eidetic") that later work built on.
The math
What counts as "memorized"
Language models memorize all the time: spellings, the zip code for San Francisco. The authors want to isolate unintended memorization, so they define it in two steps. First, a string is extractable from a model if some prefix makes it the model's most likely continuation:
Here is the probability the model assigns to the whole length- sequence given the prompt , which is the product of next-token probabilities from the chain rule. An exact argmax over all length- strings is intractable, so in practice it becomes greedy decoding or sampling. Prompts must be short, because otherwise "Repeat after me: …" would make every string extractable.
Second, a string is -eidetic memorized if it is extractable and appears in at most training documents :
The count is over documents (for GPT-2, whole webpages), not occurrences. A string repeated 300 times on one page still counts as . Small is the worrying case: the MIT license is memorized with enormous , and nobody minds.
Step one: generate a lot
The attack samples 200,000 sequences of 256 tokens under each of three strategies:
- Top-: plain sampling from the 40 most likely tokens, starting from an empty prompt.
- Temperature: the familiar softmax-with-temperature, , with starting at 10 and decaying to 1 over the first 20 tokens. The high early temperature explores varied openings. Dropping back to 1 lets the model follow a memorized path once it finds one, instead of wandering off it.
- Internet: the model is prompted with 5–10 tokens from a separate Common Crawl scrape, so openings resemble real web text.
Step two: rank by surprise, and calibrate it
The baseline membership signal is perplexity:
The quantity inside the exponential is the average per-token cross-entropy, the same negative log-likelihood that training minimizes. Low perplexity means the model is rarely surprised. Models are more confident on training data, so low perplexity hints at membership.
Raw perplexity is fooled by text that is simply easy: the numbers 1 to 100, boilerplate, "I love you. I love you. I love you…". The fix is to compare against a reference that finds the same easy text easy but has not memorized the rare stuff. Writing for log-perplexity, the three main scores are:
and are perplexities under GPT-2 XL and under GPT-2 Small, which is less than a tenth of XL's size (there is also a Medium variant). is the number of bits zlib needs to compress , a crude, model-free measure of how predictable the text is. Compression length is a code length, and Shannon tells us code length is roughly of probability. is the lowercased text. Lowercasing wrecks the perplexity of memorized text that expects particular casing, such as headlines or error logs.
The low end of each score is suspicious. That means samples where XL is much more confident than the reference can explain. Readers who know hypothesis testing will recognize the spirit of a likelihood-ratio test, with one wrinkle: the textbook statistic is a difference of log-likelihoods, while the paper uses a quotient. The fourth score, Window, takes XL's minimum perplexity over any 50-token sliding window, which catches a memorized snippet buried in noise. With perplexity itself, that makes six scores.
A worked example
Take the UUID the paper extracted, 1e4bd2a8-e8c8-4a62-adcd-40a936480059. Treat it as 32 random hex characters. A model that understands only the format can do no better than nats per character, and its chance of producing the whole string is . No amount of generalization gets you there. If a model emits it, the model stored it. A small reference model that never stored it stays near 2.77 nats per character, so the ratio exposes the gap.
Now a toy sample, with numbers chosen for illustration. Say a fraction of the sample is a memorized stretch and the rest is ordinary prose. On the prose, assume XL averages 3.0 nats per token and Small 3.4. On the memorized stretch, XL averages 0.05. XL's log-perplexity is then , whether the stretch is a license or a UUID. Raw perplexity can't tell them apart. The ratio can:
- If Small also knows the stretch (0.1 nats), the score is , barely below ordinary prose at .
- If Small sees random hex (2.77 nats), the score is , which flags the sample.
Why a reference model beats raw perplexity
interactive- this sample's score
- ordinary unmemorized text
Hover or drag across the chart to read values.
- XL perplexity (identical at every x):
- 4.6
- score if Small sees random hex (2.77):
- 0.494
- score if Small knows it too (0.1):
- 0.871
Why bigger models memorize more
The same logic explains the size effect. A bigger model has more capacity to spend on rare strings, and its loss is pushed down on every training token. A small model trained on the same data can't afford to store a one-off UUID. The gap between the two models is exactly what the Small score measures.
What the results say
- Headline: 604 of 1,800 candidates (33.5%) were verbatim training data. The best setup, Internet prompts ranked by the zlib score, hit 67%.
- Calibration is the trick: ranking by raw perplexity found memorized text in only 9% of top- candidates and 3% of temperature-sampled ones. Every comparison score did far better. Internet prompts found the most unique memorized samples (273, against 191 for top-).
- Size matters: in a natural experiment with Reddit URLs that appear in a single Pastebin document, XL emitted every URL repeated 33 or more times. Medium emitted about half of them, and Small emitted none.
- Length: by seeding the model with memorized samples and continuing with a beam-search-like decoder, the authors extended one code snippet to 1,450 lines of verbatim source.
Limitations. "Appears in one document" is not "seen once." The high-entropy strings in Table 3 occur 10 to 311 times inside their single document. Per the GPT-2 authors, the model also trained for roughly 12 epochs. The paper studies GPT-2 only, a 2020 model with no instruction tuning and no chat interface. Its counts are a deliberately loose lower bound, since only 1,800 candidates were inspected by hand. And the largest category of memorized content was news (109 samples), mostly widely repeated text rather than rare text.
Connecting the dots
Wikimedia's line rests on a technical picture: training turns text into "something new and different," while retrieval hands the original back. This paper is the standard test of the first half of that picture, and it finds an exception. With no retrieval at all, a model's weights can work as storage for some strings and reproduce them verbatim on request. In those cases, "training" and "regurgitation" are the same mechanism.
The connection has limits. The brief doesn't cite this paper, the paper studies GPT-2 and not ChatGPT, and the brief's objection is specifically to RAG. RAG copies at answer time, deliberately and often at length. Memorization is rarer, accidental and concentrated in heavily duplicated text. Wikimedia could reasonably reply that its line still works for the typical case. The paper also says nothing about fair use, which is a legal judgment, not a measurement.
One finding fits Wikimedia's attribution concern closely. When GPT-2 regurgitated code, the authors were "almost always unable to recover the original authorship notices or terms of use." Memorized output can strip off exactly the credit that CC BY-SA requires.
Why it matters
For practitioners, the paper's mitigations still apply: de-duplicate within documents as well as across them, be careful with sources (Pastebin was GPT-2's 12th most common domain), and audit models with extraction attacks before release. Differential privacy is the principled defense, but the authors note that it costs utility and is hard to apply to scraped web data, where nobody records which user wrote what.
For research and policy, the takeaway is that memorization is a tail phenomenon. Average train–test loss gaps tell you little about the worst-case examples, which are the ones courts and privacy regulators care about. The follow-up by Nasr et al. reports extraction from ChatGPT itself and concludes that alignment "does not eliminate memorization."
Things to watch: whether the court grants Wikimedia leave to file, and whether any ruling in the consolidated case treats verbatim output from the weights differently from output that retrieval copies in.
Go deeper
- Extracting Training Data from Large Language Models (PDF) by Carlini et al., with the attack code
- Wikimedia Foundation's announcement of the amicus brief, with the motion and brief on Commons
- The Verge's coverage of Wikipedia's brief
- Quantifying Memorization Across Neural Language Models (Carlini et al.), a follow-up linking memorization to model size, duplication and prompt length
- Scalable Extraction of Training Data from (Production) Language Models (Nasr et al.), which carries the attack to open models and ChatGPT