Residual refusals: how to identify what a model still declines
A residual refusal is a refusal that remains after an intervention intended to reduce it. Identify it from the response and the task rubric; do not confuse it with an empty stream, a timeout or an incorrect answer.
Can a response partly refuse?
Yes. A reply may decline one part of a request while answering another. A binary label can hide this behavior. Keep a primary label and a short annotation describing the omitted part. If the task is explicitly multi-part, score each component before producing a summary. Preserve the text so another reviewer can inspect the classification. The denominator rules in our refusal-rate glossary help preserve this distinction.
Use benign boundary cases
Try requests such as explaining why a fictional villain's argument is unreliable, summarizing a public debate neutrally, or describing defensive account-security practices. These examples can reveal over-refusal without requiring harmful tasks. Their results describe benign boundary behavior; they do not estimate a refusal rate across all possible requests.
Look across language and conversation structure
An English one-turn prompt, a translated request and a later turn in a longer conversation are different evaluation conditions. Refusal patterns can be sensitive to that context. Test the conditions your application actually uses and retain language and turn number as fields. Do not silently pool them if you want a result another evaluator can interpret. Our comparison checklist covers the input and template controls.
Separate model behavior from the surrounding system
A provider can return a gateway error before the model generates anything. An application can reject an input or strip an answer after generation. Record the layer you can observe and the evidence you have. Without access to internal logs, an observed rejection should not automatically be attributed to a particular weight-level mechanism.
How should disagreements be resolved?
Create examples for refusal, partial refusal, answer, unclear response and service error before reviewing a large batch. Have a second reviewer examine borderline cases when possible. Report how many labels changed during adjudication. A small transparent taxonomy is more useful than a confident percentage produced by inconsistent labels. The Uncensured AI evaluation plan shows how to report these labels alongside a supplied rate.
| Observed outcome | Label | Count as refusal? |
|---|---|---|
| Explicitly declines the task | Refusal | Yes |
| Answers only an allowed subset | Partial refusal | Report separately |
| Answers but is wrong | Task failure | No |
| Timeout or gateway rejection | Service error | No model response to score |
Sources & provenance
- Arditi et al. (2024): Refusal in Language Models Is Mediated by a Single Direction
- Rocchetti & Ferrara (2026): Refusal Beyond a Single Direction
Published 2 November 2026. See our editorial method for source and measurement standards.