The refusal direction in LLMs

Recent interpretability work found that whether a chat model refuses is governed largely by one linear direction in its activation space. Push activations along it and the model refuses; remove it and the model stops.

One direction, many refusals

Across many kinds of "unsafe" request, the internal signal that precedes a refusal points the same way. That means refusal is not scattered through the network but concentrated in a direction you can measure by contrasting harmful and harmless prompts, then averaging the activation gap.

What you can do with it

Add the direction to a compliant model and it starts refusing harmless things; subtract it and a cautious model complies. Doing the subtraction permanently in the weights is abliteration. The finding is also useful defensively: a classifier can watch that direction to detect when a model is about to refuse or has been steered.

Where it breaks down

Linear is an approximation The refusal direction explains a large share of refusal behaviour, not all of it. Some refusals survive removal, and heavy edits can bleed into unrelated behaviour. It is a strong effect, not a clean switch.

Related

AbliterationAbliteration removes the direction in a model’s activations that produces refusals, so it stops declining, without retraining the weights from scratch. Uncensored vs abliteratedUncensored usually means a loosely aligned finetune that still refuses sometimes. Abliterated means the refusal direction was removed. They are different claims. Red-teaming LLMsRed-teaming an LLM means attacking your own model on purpose: jailbreaks, prompt injection and abuse generation, to find failures before an attacker does.

All concepts · Integration guides · Browse models