Finding evidence, repairing reasoning and refusing unsupported questions

Chain-of-Knowledge starts when the model is unsure

A medical question, a fact about a public figure and a calculation over a table should not all be sent to the same search interface. Chain-of-Knowledge (CoK) was designed for that mismatch. It lets a language model decide which domain it needs, choose a source inside that domain and speak the source’s query language.

The reported system uses gpt-3.5-turbo-0613 for the main reasoning model. It first samples five chain-of-thought answers at temperature 0.7. If fewer than half agree—or fewer than four of five for the three-way FEVER task—the answer is marked uncertain and sent through knowledge adaptation. High-consistency answers pass through unchanged.

For an uncertain answer, CoK runs five operations:

1. Generate candidate rationales and name the knowledge domains they require.
2. Select a source in each domain.
3. Convert a rationale into the source’s native query: SPARQL for Wikidata, SQL for a table, or a natural-language query for text.
4. Execute the query, use the returned evidence to correct the first rationale, then generate and correct the next rationale from the corrected history.
5. Consolidate the repaired chain into the final answer.

The source map is explicit. Factual questions can use Wikidata, Wikipedia or WikiTables. Medical questions use a flashcard collection and UpToDate. Physics uses the Physics subset of ScienceQA and Physics Classroom; biology uses the Biology subset of ScienceQA and CK-12. ChatGPT generates SQL and natural-language queries. Because SPARQL is less common in ordinary language-model training data, the adaptive query generator uses a LoRA-tuned LLaMA-2-7B trained on sentence–SPARQL pairs, including negative query examples.

Progressive repair changes the next search

The order of correction matters. A parallel editor retrieves evidence for every original rationale independently; a wrong first step can therefore contaminate the query for the second. CoK corrects the first step before asking what the second step needs. In the ablation, dynamic adaptation improved performance by 4.2 points over chain-of-thought, while the parallel version was worse.

The paper evaluates factual, medical and science questions with three- and six-example prompts. The metrics differ by dataset: accuracy for FEVER, MedMCQA and MMLU; exact match for HotpotQA; BLEU for FeTaQA.

The six-shot CoK scores were 58.5 on FEVER, 35.4 on HotpotQA, 26.0 on FeTaQA, 73.3 on MedMCQA, 47.0 on MMLU Physics and 84.4 on MMLU Biology. More demonstrations did not always help: the six-shot FEVER score was lower than the three-shot score. Adding biology sources to the medical configuration improved MedMCQA by 1.3 points; using both Flashcard and UpToDate improved it by 2.1 points over Flashcard alone.

The paper also asked whether a better rationale necessarily yields the right answer. Two annotators compared 100 balanced FEVER and HotpotQA outputs. Among CoK’s incorrect predictions, they judged its rationale more factually consistent in 44% of cases; in 73% of that subset, they believed the improved rationale should have supported a better answer. Agreement was moderate (Cohen’s κ = 0.43). Retrieval can fix a premise while the final reasoning step still fails.

Trust-Align treats the retrieved documents as the boundary

Trust-Align begins after retrieval. Each example contains a question and a fixed set of documents. A response is correct only when its claims can be supported by those documents. If the documents are insufficient, refusal is the target behaviour even when the model may know the answer from pretraining. This is deliberately stricter than ordinary open-book question answering.

The paper separates five errors that a single accuracy score mixes together:

• Inaccurate answer: a generated claim is wrong.
• Over-responsiveness: the model answers although the documents are insufficient.
• Excessive refusal: it refuses an answerable question.
• Over-citation: it attaches citations that are not needed for a claim.
• Improper citation: the cited passage does not support the claim.

Trust-Score averages three F1 measures. Answer-correctness F1 is calibrated over what can be answered from the supplied documents. Grounded-refusal F1 measures whether the model answers answerable cases and refuses unanswerable ones. Grounded-citation F1 is the harmonic mean of citation recall—whether claims carry supporting citations—and citation precision—whether each cited passage entails the claim.

Construct both answerable and unanswerable training cases

Trust-Align starts with questions from ASQA, QAMPARI and ELI5. Questions are clustered and scored for quality, yielding roughly 10,000 seeds. For each question, the pipeline retrieves 100 documents from Wikipedia or Common Crawl and keeps five oracle documents with the same answer recall as the larger set. It then removes and recombines evidence to create roughly 70,000 answerable and unanswerable question–document pairs, shuffling document order to avoid position shortcuts.

GPT-4 composes preferred cited responses from the gold claims; unanswerable cases receive a refusal. A fine-tuned LLaMA-2-7B generates candidate negative responses. Of about 70,000 candidates, 40,985 contained at least one of the five target errors. The examples are ranked by error severity and the top half of both answerable and unanswerable cases form approximately 19,000 preference pairs for DPO.

Refusal and citation improve; answer accuracy is mixed

Evaluation uses five retrieved documents per question on ASQA, QAMPARI and ELI5. The released study covers nine open-weight model families and sizes—LLaMA from 1B to 8B, Qwen2.5 from 0.5B to 7B and Phi-3.5 at 3.8B—for 27 model–dataset configurations. Trust-Align beat the strongest compared baseline on overall Trust-Score in 26 of 27 configurations.

For LLaMA-3-8B, the gain over FRONT was 12.56 points on ASQA, 36.04 on QAMPARI and 17.69 on ELI5. Grounded-refusal F1 improved by 23.87, 47.95 and 45.77 points in the paper’s summary results; grounded-citation F1 improved by 22.12, 38.35 and 5.55. Answer correctness did not move uniformly: all nine configurations improved on QAMPARI, but only five of nine improved on ELI5 and two of nine on ASQA.

The data ablation is equally useful. Selecting only the hardest 25% of augmented pairs produced the best reported Trust-Scores on QAMPARI (57.73) and ELI5 (47.55). Adding more lower-severity pairs was not monotonically helpful because many were redundant; the authors also note possible overfitting at 50%.

CoK and Trust-Align address different points of failure. CoK changes what the model retrieves and repairs the reasoning that led there. Trust-Align holds retrieval fixed and tests whether the model stays inside the supplied evidence. CoK can still issue a bad query or reason incorrectly after a factual repair. Trust-Align can learn an appropriate refusal yet lose answer recall, and its citation judgements depend on automatic entailment models. Neither result is a general measure of truthfulness: both are bounded by the sources, tasks and evaluators used in the experiment.