PuzzleVQA, AlgoPuzzleVQA + M-LongDoc

The problem: accuracy does not say where reasoning failed

A wrong puzzle answer does not tell us whether the model overlooked a triangle, inferred the wrong pattern or made a mistake applying it. PuzzleVQA records these steps separately so we can test what changes when a model is given the correct visual description or rule. AlgoPuzzleVQA asks a related question with algorithmic problems whose answers can be checked by a solver. M-LongDoc moves to a less controlled setting, where the model must first find the relevant page in a long document before explaining the evidence on it.

PuzzleVQA separates perception, induction and deduction

PuzzleVQA contains 2,000 generated multiple-choice questions: 20 templates with 100 instances each. Four single-concept categories test colour, number, size or shape; six dual-concept categories combine them. Every instance stores the visible objects as a caption, the induced pattern as an explanation and the final deduction separately. Models were evaluated zero-shot from the image, question and options with a chain-of-thought prompt.

GPT-4V scored 46.4% on the single-concept set, against a 27.1% random baseline, and 45.5% on the dual-concept set, against 26.4% random. On the same 40-question subset, 23 university students averaged 91.6% while GPT-4V scored 47.5%. Supplying the ground-truth perception and induction records allowed GPT-4V and Claude 3 Opus to solve almost all cases, locating much of their error before the final deduction. Qualitative examples show both kinds of failure: missing or inventing polygons in the image, and proposing a pattern not supported by the observed objects.

AlgoPuzzleVQA makes the rule executable

AlgoPuzzleVQA contains 1,800 evaluation-only instances, 100 for each of 18 puzzle types. Human-written generators draw each image and a corresponding solver computes the exact answer for tasks such as N-Queens, map colouring, maze search, Tower of Hanoi and water jugs. Seventeen tasks use four choices; Board Tiling is yes/no. This gives an overall random baseline of 26.4%.

The paper tested GPT-4V, Gemini Pro, Claude 3 Opus, InstructBLIP and LLaVA-1.5. The best configuration—GPT-4V prompted to describe the image before reasoning—reached 31.7% average accuracy, 5.3 points above random. No model-setting pair exceeded 60% on any puzzle; the highest cell was 57% for GPT-4V on Calendar. Supplying a written description of the image raised GPT-4V from 29% to 74% on Clock and from 25% to 61% on Move Box, but lowered Calendar from 51% to 43%. Perception was sometimes the bottleneck, but gold visual descriptions did not remove the algorithmic failures.

M-LongDoc: finding evidence in long documents

M-LongDoc uses 180 documents published in 2024 or later: 60 computer-science papers, 60 product manuals and 60 financial reports. Its 851 human-validated questions comprise 271 text, 283 figure and 297 table questions. Documents average 210.8 pages and 120,988 extracted text tokens. Each question has one target evidence page but requires an open explanation rather than a copied span.

At evaluation time, ColPali retrieves the top five pages. Retrieval-aware tuning uses 10,070 automatically generated examples from 300 separate documents and mixes the gold page with irrelevant retrieved pages. On a 1-to-5 correctness scale, Qwen2-VL-7B rose from 3.84 to 4.02 overall, the reported 4.6% relative gain. Its untuned scores were 4.08 for text questions, 3.83 for figures and 3.62 for tables. The evaluator is a committee of three multimodal judge models sampled five times each; on a 100-answer check, its aggregate score correlated with one human annotator at Pearson 0.889. These are judge scores, not percentage accuracy.

When a better description is still not enough

Giving a model the correct visual description sometimes produces a large improvement, as the Clock and Move Box results show, but it does not consistently solve the reasoning problem. That is why the intermediate checks matter alongside final accuracy. PuzzleVQA and AlgoPuzzleVQA make those checks possible with generated templates and mostly multiple-choice answers, at the cost of repeated layouts that a model might exploit. M-LongDoc uses real documents and open explanations, but still gives each question one target evidence page and relies on model judges to score the answer. These choices make particular failures easier to study while leaving harder cases, such as combining evidence spread across several pages, outside the reported test.