Safety is not one test

RED-EVAL and RED-INSTRUCT: attack, collect, align

RED-EVAL places a harmful question inside a scripted Chain-of-Utterances exchange between a red-team agent and an unsafe-helpful assistant. The evaluated model completes the assistant’s next turn, including an “internal thought” prefix that encourages it to continue the role-play. On 200 DangerousQA questions, GPT-4 judged each completion as harmful or safe; on a 400-response audit, those labels agreed with three expert annotators more than 98% of the time. The attack succeeded on 65% of GPT-4 queries and 73% of ChatGPT queries, compared with near-zero success for direct and chain-of-thought prompts in those tests.

HarmfulQA was built around 1,960 generated questions spanning ten topics and 98 usable subtopics. ChatGPT produced 9,536 safe “blue” conversations with 65,925 turns, then RED-EVAL generated 7,356 paired harmful “red” conversations with 52,875 turns. Safe-Align fine-tuned Vicuna-7B on 7,356 blue conversations, 13,434 ordinary helpful question-answer pairs and 20,790 ShareGPT conversations. A second variant first pushed against the paired red responses for 200 steps before continuing with the blue mixture.

The blue-trained Starling improved the paper’s aggregate red-team robustness by 5.2 points over Vicuna-7B while its TruthfulQA/MMLU/BBH average remained 40.08 versus 40.11. The red-data variant exposed a failure in the training recipe: on the paper’s averaged red-team comparison it was slightly more susceptible than Vicuna, and extending the red-response penalty beyond 200 steps caused generation collapse and a large MMLU drop. The benchmark is also tied to 2023 model versions, the tested CoU prompt variants and a GPT-4 judge; HarmfulQA is dual-use by construction.

RESTA: restore a safety direction after fine-tuning

RESTA starts with Llama-2-7B-Chat and an intentionally less aligned copy. Their parameter difference defines a safety vector. After LoRA or full fine-tuning for Chinese, Hindi or English instruction following, code, or mathematics, RESTA adds half of that vector to the task model; the DARE variant first drops 30% of the task delta and rescales what remains. No further gradient training is used for the repair.

Safety is measured with GPT-4 labels on four datasets: the paper’s CatQA, HarmfulQA, 200 sampled AdversarialQA instructions and 200 DangerousQA questions. CatQA contains 550 prompts—ten questions for each of five subcategories under 11 prohibited-use categories—and human-corrected Chinese and Vietnamese translations. Task retention is checked on XCOPA, XNLI, COPA, HumanEval and GSM8K.

Across the reported safety sets and tasks, RESTA reduced mean harmful-response rates from 18.6% to 5.1% after parameter-efficient fine-tuning and from 9.2% to 1.5% after full fine-tuning. The corresponding mean task-score reductions were 2.41 and 0.47 points. These results cover one 7B model family; transfer of a safety vector between model families, larger models such as Llama-2-70B and broad searches over the vector scale were not tested. The safety metric also omits responses that the GPT-4 judge refused to label.

WalledEval: run the model and the judge through the same test

WalledEval is a Python library that turns a dataset, a model and a judge into three explicit flows: score model responses to unsafe prompts, score a safety judge against labelled text, or evaluate multiple-choice safety knowledge. The published release exposes more than 35 datasets through a common interface and supports both Hugging Face models and hosted APIs. Its paper evaluates harmful-response behaviour on HarmBench, AdvBench and CatQA, refusal behaviour on XSTest and SGXSTest, and multilingual safety on Aya Red-Teaming.

The mutation study applies nine surface transformations—including tense changes, paraphrases, misspellings and translation—to make 1,800 HarmBench and 3,600 XSTest variants. Average refusal-test scores fell by more than five points on mutated XSTest and by about four points on the Singapore-specific SGXSTest. The release also supplies SGXSTest’s 100 manually curated safe/unsafe pairs, HIXSTest’s 25 pairs, and a 494-million-parameter WalledGuard moderator that can itself be benchmarked as a judge.

The first release is text-only and library-only, does not batch Hugging Face inference, and contains only nine mostly surface-level mutators. The authors also report that several prompt templates were inconsistent and not well tested across generators. A common runner makes comparisons repeatable, but it does not remove dataset blind spots or errors from the chosen judge.

OffTopicEval: test the deployed role boundary

OffTopicEval instantiates 21 purpose-specific agents, each with an explicit role, allowed and forbidden behaviours, injection defences, examples and a fallback response. For each model, it tests 3,150 in-domain questions, 10,053 direct out-of-domain questions drawn from factual MMLU subjects, and 211,113 agent-specific adaptive questions in English, Chinese and Hindi. The adaptive set is produced by asking Llama-70B to rewrite a direct out-of-domain question so that it resembles the agent’s domain without changing the requested information; a Mistral-24B check found 98.55% content agreement and 98.97% answer consistency. The multilingual study covers 20 open-weight models; six hosted models are also evaluated in English.

The operational-safety score is the harmonic mean of in-domain acceptance and the average refusal rate on direct and adaptive out-of-domain queries. Across 20 open-weight models, 12.24% of direct out-of-domain queries were accepted, rising to 70.72% after laundering. For example, a Llama-3.3-70B agent accepted 99.62% of valid English requests and refused 69.73% of direct out-of-domain requests, but refused only 4.21% of their adaptive versions; its operational-safety score was 53.93.

Appending a short policy reminder raised Llama-3.3-70B’s English score from 53.92 to 94.99 and Qwen-3-30B’s from 65.10 to 91.86. It was not a uniform fix: the same intervention reduced Gemma-3-27B from 63.92 to 51.19 by rejecting too many valid requests. OffTopicEval measures adherence to specified role descriptions and factoid out-of-domain queries; it does not test whether those policies are correct, whether a refusal is useful, or whether the prompt suffix remains effective in a longer tool-using workflow.