A chatbot's "no" can live in a single vector

TL;DR

• Tech Against Terrorism tested 130+ AI models. Three in five failed, and models with their refusals stripped out ("abliterated") failed every test.
• Arditi et al. found that refusal in 13 chat models depends on one activation direction. Projecting that direction out of the weights turns refusal off.
• That geometry is why abliteration is cheap. Safety fine-tuning puts a thin, removable "no" on top of capabilities that stay intact.

What happened

CBS News reported on 9 October that Tech Against Terrorism, a UK nonprofit, scored more than 130 models on hundreds of requests of the kind a terrorist planning an attack might send. A model failed if it gave "one complete, specific answer about a mass-casualty subject, or a score below 90." Three in five models failed, and open-weight and closed models scored about the same. Models put through abliteration failed every test. Meta's Llama 3.1 8B scored 97 out of 100 on the group's benchmark, and its abliterated version scored about 3. Asked about a vehicle-as-weapon attack, the original model declined, while the abliterated one replied with an 18-point list. The group also counted more than 29,000 Hugging Face repositories, as of late September, that advertised models as uncensored or without safeguards (Anadolu reports the same figures).

The National gives more detail: 134 models, 627 requests, and 13 of 13 abliterated models failing. It describes the counts differently, though, saying all but 2 models gave dangerous information. It also quotes founder Adam Hadley: "A model that refuses a stated terrorist and answers a stated researcher has not been made safe. It has been made polite." All of these figures are Tech Against Terrorism's own findings as relayed by the press. I haven't seen the underlying report, and nobody has independently replicated it. Wired covered the study on 14 October, but I couldn't access that article, so nothing here depends on it.

The paper in brief

Chat models are fine-tuned to help with benign requests and refuse harmful ones, but how refusal works inside the model was poorly understood. Arditi and colleagues ran 13 open chat models (Qwen, Yi, Gemma, Llama-2 and Llama-3, from 1.8B to 72B parameters) on harmful and harmless instructions. They subtracted the average internal activation on harmless prompts from the average on harmful ones and got a single vector. Erasing that direction everywhere in the network stops the model from refusing harmful requests. Adding it makes the model refuse harmless ones. The erasure can also be written permanently into the weights with a rank-one edit that leaves benchmark scores nearly unchanged. The paper showed that interpretability can produce a working attack: no gradients, no harmful training completions, and, by the authors' estimate, under $5 of compute for a 70B model.

The math

Setup

A decoder-only transformer keeps a residual stream for each token position $i$: a vector $\mathbf{x}_i^{(l)} \in \mathbb{R}^{d_{\text{model}}}$ entering layer $l$. Each attention and MLP block reads from the stream and adds its output back in. The authors look at the post-instruction tokens, the chat-template tokens after the user's message (such as the tag that starts the assistant's turn). That is where the model settles how it will respond. The inputs are 128 harmful instructions and 128 harmless ones from Alpaca. The output is one direction, which they then test causally.

Step 1: a direction from two averages

$$\mathbf{r} = \boldsymbol{\mu} - \boldsymbol{\nu} = \frac{1}{|\mathcal{D}_{\text{harmful}}|}\sum_{\mathbf{t}\in\mathcal{D}_{\text{harmful}}} \mathbf{x}(\mathbf{t}) - \frac{1}{|\mathcal{D}_{\text{harmless}}|}\sum_{\mathbf{t}\in\mathcal{D}_{\text{harmless}}} \mathbf{x}(\mathbf{t})$$

Here $\mathbf{x}(\mathbf{t})$ is the activation at a fixed layer $l$ and position $i$ (I've dropped those indices) when the model reads prompt $\mathbf{t}$. $\boldsymbol{\mu}$ and $\boldsymbol{\nu}$ are the class means, and $\mathbf{r}$ is the vector between them. If you remember Fisher's linear discriminant, its direction is $\Sigma^{-1}(\boldsymbol{\mu}-\boldsymbol{\nu})$. This is the same idea without the covariance term. Nothing is trained.

Repeating this for every layer and post-instruction position gives $|I| \times L$ candidates. The authors pick one using 32 validation prompts per class. The winner is the candidate whose removal most lowers a refusal log-odds score on harmful prompts, subject to three checks:

• Adding it must actually cause refusals.
• Removing it must change the next-token distribution on harmless prompts by a KL divergence below 0.1.
• It must come from the first 80% of layers, so it isn't just "don't output the token 'I'".

Write $\hat{\mathbf{r}} = \mathbf{r}/\lVert\mathbf{r}\rVert$ for the unit vector.

Step 2: directional ablation

$$\mathbf{x}' = \mathbf{x} - \hat{\mathbf{r}}\hat{\mathbf{r}}^{\top}\mathbf{x}$$

The scalar $\hat{\mathbf{r}}^{\top}\mathbf{x}$ measures how much of the refusal direction $\mathbf{x}$ contains. The edit subtracts exactly that much, which is one step of Gram–Schmidt. In matrix form this is the projection $P = I - \hat{\mathbf{r}}\hat{\mathbf{r}}^{\top}$. Its eigenvalue is 0 along $\hat{\mathbf{r}}$ and 1 in every one of the $d_{\text{model}}-1$ orthogonal directions. One axis gets flattened and all the others are left alone. The authors apply it to every activation $\mathbf{x}_i^{(l)}$ and $\tilde{\mathbf{x}}_i^{(l)}$, at every layer and every position.

Worked example. Suppose $\mathbf{r} = (3, 4, 0)$, so $\lVert\mathbf{r}\rVert = 5$ and $\hat{\mathbf{r}} = (0.6, 0.8, 0)$. Take an activation $\mathbf{x} = (2, 1, 7)$. Its projection is $\hat{\mathbf{r}}^{\top}\mathbf{x} = 1.2 + 0.8 = 2$, so $\mathbf{x}' = (2,1,7) - 2(0.6, 0.8, 0) = (0.8, -0.6, 7)$. Check: $\hat{\mathbf{r}}^{\top}\mathbf{x}' = 0.48 - 0.48 = 0$. The squared norm drops from 54 to 50, exactly the $2^2$ that sat along $\hat{\mathbf{r}}$. The third coordinate, standing in for everything else the model knows, is unchanged.

Ablating the refusal direction in a one-dimensional toy

interactive

• harmful prompts
• harmless prompts
• harmless prompts after subtracting r

Hover or drag across the chart to read values.

ablation strength α0harmful–harmless gap ‖r‖ (in σ)4

refusal rate, harmful prompts:

97.7%

refusal rate, harmless prompts:

2.28%

The paper always uses full ablation. The slider $\alpha$ interpolates $\mathbf{x} - \alpha\hat{\mathbf{r}}\hat{\mathbf{r}}^{\top}\mathbf{x}$ only to make the geometry visible, and the Gaussian threshold model is my simplification, not the authors'.

Step 3: activation addition

$$\mathbf{x}^{(l^*)\prime} = \mathbf{x}^{(l^*)} + \mathbf{r}$$

This adds the full, unnormalized $\mathbf{r}$, magnitude included, at the single layer $l^*$ it came from and at every position. A harmless prompt's activations get pushed toward the harmful mean, and the model refuses. Ablation shows the direction is necessary for refusal, and addition shows it is sufficient. Together they make a two-sided causal test.

Subtracting $\mathbf{r}$ also stops refusal, but, as the dashed curve suggests, it shoves harmless activations off-distribution. On Llama-3 8B's own completions, cross-entropy loss is 0.195 at baseline, 0.213 after ablation and 0.441 after subtraction.

Step 4: baking it into the weights

$$W_{\text{out}}' = W_{\text{out}} - \hat{\mathbf{r}}\hat{\mathbf{r}}^{\top} W_{\text{out}}$$

$W_{\text{out}} \in \mathbb{R}^{d_{\text{model}} \times d_{\text{input}}}$ is any matrix that writes into the residual stream: the token and positional embeddings, attention output matrices and MLP output matrices, plus their biases. The residual stream is just the sum of these writes. If none of them can write along $\hat{\mathbf{r}}$, the stream never contains that direction, and Appendix E of the paper shows this is exactly equivalent to Step 2. The correction $\hat{\mathbf{r}}(\hat{\mathbf{r}}^{\top}W_{\text{out}})$ is an outer product, so each edit is a rank-one update. The result is an ordinary checkpoint that anyone can upload.

Why one dimension is enough: later components appear to read refusal off something like $\hat{\mathbf{r}}^{\top}\mathbf{x}$. Set that coordinate to zero and the check never fires, while the other $d_{\text{model}}-1$ dimensions carry on as before.

What the results say

• Refusal collapses. On 100 JailbreakBench prompts with Llama-3 8B, ablation lowers the refusal score (share of outputs containing phrases like "I'm sorry") from 0.95 to 0.01. The safety score (share that Llama Guard 2 judges safe) falls from 0.97 to 0.15. LoRA fine-tuning on 243 harmful examples gets 0.00 and 0.08.
• It competes with optimized attacks. On HarmBench, orthogonalized Qwen 7B reaches a 79.2% attack success rate, versus 7.0% with no jailbreak and 79.5% for GCG, which optimizes a separate suffix for each prompt. Llama-2 7B gets only 22.6% with its default safety system prompt, but 79.9% without it, and anyone holding the weights can drop the system prompt.
• Capabilities mostly survive. Llama-3 70B scores 79.8 vs 79.9 on MMLU and 90.8 vs 91.2 on GSM8K after the edit. TruthfulQA drops for every model, and Qwen 7B and Yi 34B show statistically significant drops on some other benchmarks.
• The direction predates safety training. In the four base models the authors checked, the same direction already separates harmful from harmless prompts. That suggests fine-tuning repurposes an existing feature rather than building a new one.

Limitations. The authors note that they tested open models of 72B parameters or less from 2024, so the results may not hold for larger or proprietary models. Their extraction relies on heuristics and is "an existence proof". They also can't say what the direction means: it might encode "harm" rather than "refusal". The adversarial-suffix analysis covers one model and one suffix. A careful reader should add that "unsafe" here is a Llama Guard 2 verdict, not a human judgment of real-world usefulness.

Connecting the dots

Abliteration is the technique this paper describes. CBS describes it as finding safety patterns in the weights and cancelling them. A September 2026 defense paper defines it as attacks that "identify and project out a linear refusal direction from the residual stream", which is Steps 2 and 4 above. What I can't confirm is which tools Tech Against Terrorism or the uploaders of the 29,000 repositories actually used. So the accurate claim is that this paper describes the technique underneath abliteration, not that it describes the method used in the study.

Two cautions. First, Llama 3.1 8B isn't in the paper; the authors tested Llama-3 8B. The "97" that appears in both places is a coincidence of different scales. Tech Against Terrorism's 97 → 3 is on its own 100-point benchmark, while the paper's 0.97 → 0.15 is a Llama Guard pass rate on JailbreakBench. Second, the news is broader than the paper. Most of the study's failures were unmodified models, including closed ones, and the finding that "researcher" framing raises compliance is a prompt-level weakness the paper doesn't address.

The two also disagree on what follows. The authors argued that their method "does not substantially alter the risk profile" of open-sourcing, since fine-tuning could already strip safeguards. Tech Against Terrorism wants abliterated models kept off public repositories.

Why it matters

For anyone releasing open weights, the practical lesson is to treat the abliterated version as the realistic worst case and evaluate it, as this study did. Refusal training changes what a model is willing to do, not what it knows, and the paper's base-model result suggests the relevant feature was there before safety training. For research, the question is whether safeguards can survive full weight access. Decoy Direction Optimization plants a misleading signal so that difference-in-means attacks remove the wrong direction, and its authors report attack success rates under 10% against standard refusal-direction ablation.

Things to watch for next: whether decoy-style defenses hold up against attackers who know to look for decoys, and whether Hugging Face or governments act on Tech Against Terrorism's call to keep abliterated models off public repositories.

Go deeper

• The paper on arXiv and the authors' code
• News coverage: CBS News, Anadolu, The National, and Wired (which I couldn't access)
• Tech Against Terrorism's July 2026 CT-AI Benchmark release, the earlier 27-model study this one builds on
• Decoy Direction Optimization: A Post-Hoc Defense Against LLM Abliteration (Muhamed, Diab, Smith, 2026), a follow-up on defending against abliteration