CoU, RESTA, WalledEval + OffTopicEval

Learning from conversational attacks

RED-EVAL places a harmful question inside a scripted Chain-of-Utterances exchange between a red-team agent and an unsafe-helpful assistant. The evaluated model completes the assistant’s next turn, including an “internal thought” prefix that encourages it to continue the role-play. On 200 DangerousQA questions, GPT-4 judged each completion as harmful or safe; on a 400-response audit, those labels agreed with three expert annotators more than 98% of the time. The attack succeeded on 65% of GPT-4 queries and 73% of ChatGPT queries, compared with near-zero success for direct and chain-of-thought prompts in those tests.

HarmfulQA was built around 1,960 generated questions spanning ten topics and 98 usable subtopics. ChatGPT produced 9,536 safe “blue” conversations with 65,925 turns, then RED-EVAL generated 7,356 paired harmful “red” conversations with 52,875 turns. Safe-Align fine-tuned Vicuna-7B on 7,356 blue conversations, 13,434 ordinary helpful question-answer pairs and 20,790 ShareGPT conversations. A second variant first pushed against the paired red responses for 200 steps before continuing with the blue mixture.

The blue-trained Starling improved the paper’s aggregate red-team robustness by 5.2 points over Vicuna-7B while its TruthfulQA/MMLU/BBH average remained 40.08 versus 40.11. The red-data variant exposed a failure in the training recipe: on the paper’s averaged red-team comparison it was slightly more susceptible than Vicuna, and extending the red-response penalty beyond 200 steps caused generation collapse and a large MMLU drop. The benchmark is also tied to 2023 model versions, the tested CoU prompt variants and a GPT-4 judge; HarmfulQA is dual-use by construction.

RESTA: restore a safety direction after fine-tuning

Safety training can be undone when a model is later adapted to another task. RESTA studies whether the lost behaviour can be recovered directly in parameter space. It starts with Llama-2-7B-Chat and an intentionally less aligned copy, using their parameter difference as a safety vector. After LoRA or full fine-tuning for Chinese, Hindi or English instruction following, code, or mathematics, RESTA adds half of that vector to the task model; the DARE variant first drops 30% of the task delta and rescales what remains. The repair requires no further gradient training.

Safety is measured with GPT-4 labels on four datasets: the paper’s CatQA, HarmfulQA, 200 sampled AdversarialQA instructions and 200 DangerousQA questions. CatQA contains 550 prompts—ten questions for each of five subcategories under 11 prohibited-use categories—and human-corrected Chinese and Vietnamese translations. Task retention is checked on XCOPA, XNLI, COPA, HumanEval and GSM8K.

Across the reported safety sets and tasks, RESTA reduced mean harmful-response rates from 18.6% to 5.1% after parameter-efficient fine-tuning and from 9.2% to 1.5% after full fine-tuning. The corresponding mean task-score reductions were 2.41 and 0.47 points. These results cover one 7B model family; transfer of a safety vector between model families, larger models such as Llama-2-70B and broad searches over the vector scale were not tested. The safety metric also omits responses that the GPT-4 judge refused to label.

WalledEval: run the model and the judge through the same test

WalledEval is a Python library that turns a dataset, a model and a judge into three explicit flows: score model responses to unsafe prompts, score a safety judge against labelled text, or evaluate multiple-choice safety knowledge. The published release exposes more than 35 datasets through a common interface and supports both Hugging Face models and hosted APIs. Its paper evaluates harmful-response behaviour on HarmBench, AdvBench and CatQA, refusal behaviour on XSTest and SGXSTest, and multilingual safety on Aya Red-Teaming.

The mutation study applies nine surface transformations—including tense changes, paraphrases, misspellings and translation—to make 1,800 HarmBench and 3,600 XSTest variants. Average refusal-test scores fell by more than five points on mutated XSTest and by about four points on the Singapore-specific SGXSTest. The release also supplies SGXSTest’s 100 manually curated safe/unsafe pairs, HIXSTest’s 25 pairs, and a 494-million-parameter WalledGuard moderator that can itself be benchmarked as a judge.

Running these checks through the same library makes it easier to compare systems, but the comparison still depends on the tests and the judge. The first release is limited to text, has no interface beyond the library, and does not batch Hugging Face inference. Its nine mutators mainly change surface wording, and the authors report several prompt templates that were inconsistent or insufficiently tested across generators. Those details matter when deciding whether a low harmful-response score reflects robust behaviour or a narrow test.

OffTopicEval: test the deployed role boundary

OffTopicEval instantiates 21 purpose-specific agents, each with an explicit role, allowed and forbidden behaviours, injection defences, examples and a fallback response. For each model, it tests 3,150 in-domain questions, 10,053 direct out-of-domain questions drawn from factual MMLU subjects, and 211,113 agent-specific adaptive questions in English, Chinese and Hindi. The adaptive set is produced by asking Llama-70B to rewrite a direct out-of-domain question so that it resembles the agent’s domain without changing the requested information; a Mistral-24B check found 98.55% content agreement and 98.97% answer consistency. The multilingual study covers 20 open-weight models; six hosted models are also evaluated in English.

The operational-safety score is the harmonic mean of in-domain acceptance and the average refusal rate on direct and adaptive out-of-domain queries. Across 20 open-weight models, 12.24% of direct out-of-domain queries were accepted, rising to 70.72% after laundering. For example, a Llama-3.3-70B agent accepted 99.62% of valid English requests and refused 69.73% of direct out-of-domain requests, but refused only 4.21% of their adaptive versions; its operational-safety score was 53.93.

Appending a short policy reminder raised Llama-3.3-70B’s English score from 53.92 to 94.99 and Qwen-3-30B’s from 65.10 to 91.86. For Gemma-3-27B, however, the same reminder reduced the score from 63.92 to 51.19 because it rejected too many valid requests. A stronger boundary can make an assistant less useful if it also blocks the work it was meant to do. OffTopicEval exposes that trade-off for specified roles and factoid queries; deciding whether the role policy is appropriate, or whether the reminder survives a longer tool-using conversation, requires further tests.