Refusal vector, refusal rate and KL divergence: a practical glossary

Terms you will see in model reports

Refusal direction

A direction in an activation space associated with refusal-related behavior under a particular method and model.

Refusal rate

The number of responses labeled as refusals divided by the number of valid evaluated responses, multiplied by 100.

Over-refusal

A refusal on a request that the evaluation rubric considers answerable.

KL divergence

A directional measure of how one probability distribution differs from a reference distribution.

Abliteration

An intervention intended to reduce refusal-related behavior by editing representations or weights.

Why a refusal rate needs a denominator

A statement such as zero refusals is incomplete without the number and kind of evaluated requests. A set of easy benign prompts and a deliberately difficult set can produce different rates for the same model. Also state whether ambiguous replies, partial answers and failed requests are included. An API error should be counted in service reliability rather than silently added to the refusal denominator. Use the refusal-label taxonomy to keep partial answers and service errors distinct.

What can KL tell you?

For discrete distributions, KL(P‖Q) = Σ P(x) log(P(x)/Q(x)). The direction matters: reversing P and Q generally changes the value. In model comparisons, specify the token positions, dataset, aggregation and reference. A small value on one corpus does not prove that reasoning, coding or long-context performance is unchanged. The capability evaluation guide explains why distribution similarity is not a task score.

Generation speed is another measurement

Tokens per second can refer to output generation, combined processing or a provider-specific display. Time to first token measures the wait before an answer starts. End-to-end latency includes the whole request. Keep all three labels distinct when describing a service; a single speed number cannot describe every aspect of responsiveness.

Read the nouns around the number

Before citing a metric, look for its scope: checkpoint, prompt set, language, precision and date. Missing context does not make the number automatically false, but it limits what a reader can conclude. A good report preserves those conditions every time the number appears in a comparison. Our model-card reading checklist turns these terms into documentation checks.