What did the model actually see?

The problem: accuracy does not say where reasoning failed

A model can choose the wrong option because it missed a triangle, inferred the wrong rule or applied the right rule incorrectly. A single accuracy score merges those failures. These three projects make the evidence path progressively longer while keeping some part of it inspectable.

PuzzleVQA separates perception, induction and deduction

PuzzleVQA contains 2,000 generated multiple-choice questions: 20 templates with 100 instances each. Four single-concept categories test colour, number, size or shape; six dual-concept categories combine them. Every instance stores the visible objects as a caption, the induced pattern as an explanation and the final deduction separately. Models were evaluated zero-shot from the image, question and options with a chain-of-thought prompt.

GPT-4V scored 46.4% on the single-concept set, against a 27.1% random baseline, and 45.5% on the dual-concept set, against 26.4% random. On the same 40-question subset, 23 university students averaged 91.6% while GPT-4V scored 47.5%. Supplying the ground-truth perception and induction records allowed GPT-4V and Claude 3 Opus to solve almost all cases, locating much of their error before the final deduction. Qualitative examples show both kinds of failure: missing or inventing polygons in the image, and proposing a pattern not supported by the observed objects.

AlgoPuzzleVQA makes the rule executable

AlgoPuzzleVQA contains 1,800 evaluation-only instances, 100 for each of 18 puzzle types. Human-written generators draw each image and a corresponding solver computes the exact answer for tasks such as N-Queens, map colouring, maze search, Tower of Hanoi and water jugs. Seventeen tasks use four choices; Board Tiling is yes/no. This gives an overall random baseline of 26.4%.

The paper tested GPT-4V, Gemini Pro, Claude 3 Opus, InstructBLIP and LLaVA-1.5. The best configuration—GPT-4V prompted to describe the image before reasoning—reached 31.7% average accuracy, 5.3 points above random. No model-setting pair exceeded 60% on any puzzle; the highest cell was 57% for GPT-4V on Calendar. Supplying a written description of the image raised GPT-4V from 29% to 74% on Clock and from 25% to 61% on Move Box, but lowered Calendar from 51% to 43%. Perception was sometimes the bottleneck, but gold visual descriptions did not remove the algorithmic failures.

M-LongDoc hides the evidence in 210 pages

M-LongDoc uses 180 documents published in 2024 or later: 60 computer-science papers, 60 product manuals and 60 financial reports. Its 851 human-validated questions comprise 271 text, 283 figure and 297 table questions. Documents average 210.8 pages and 120,988 extracted text tokens. Each question has one target evidence page but requires an open explanation rather than a copied span.

At evaluation time, ColPali retrieves the top five pages. Retrieval-aware tuning uses 10,070 automatically generated examples from 300 separate documents and mixes the gold page with irrelevant retrieved pages. On a 1-to-5 correctness scale, Qwen2-VL-7B rose from 3.84 to 4.02 overall, the reported 4.6% relative gain. Its untuned scores were 4.08 for text questions, 3.83 for figures and 3.62 for tables. The evaluator is a committee of three multimodal judge models sampled five times each; on a 100-answer check, its aggregate score correlated with one human annotator at Pearson 0.889. These are judge scores, not percentage accuracy.

What these benchmarks do not establish

PuzzleVQA and AlgoPuzzleVQA are synthetic, template-generated and mostly multiple-choice; success can still exploit repeated layouts or answer construction. M-LongDoc uses recent real documents and open answers, but each question is designed around one evidence page and the final metric depends on model judges. Together they reveal whether failure begins in perception, rule induction, retrieval or answer grounding; they do not reduce all visual reasoning to one leaderboard number.