Generating and evaluating research ideas

Problem

A model can write a fluent research idea without settling whether its link to earlier work holds up, whether someone has already tried it, or whether the proposed experiment would answer an interesting question. The researcher still has to decide what to do with it.

MOOSE-Chem asks models to recover chemistry hypotheses from earlier literature. RQ-Bench checks whether model judges rate research questions the way scientists do. IDEAgent uses feedback to improve a set of candidate ideas while keeping them distinct. The three projects share a problem but were evaluated separately.

MOOSE-Chem

A chemistry hypothesis may connect a known research problem with results from several other papers. MOOSE-Chem makes that connection the object of search: retrieve possible inspirations, combine them into candidate hypotheses, then rank the candidates. Its search adds inspirations over multiple steps and uses mutation and recombination to explore alternative combinations.

The benchmark contains 51 chemistry papers first made public in 2024. Chemistry PhD students separate each paper into its background, inspirations and hypothesis. The system receives the background and a literature corpus containing the inspiration papers, but not the target hypothesis; the tested models have training cutoffs in 2023. It is a rediscovery test; no new wet-lab results are claimed. MOOSE-Chem, ICLR 2025.

In the 300-paper-corpus experiment, experts examined four candidates per background, preselected using GPT-4o’s similarity scores. The best candidate received a Matched Score of 3 or 4 in 21 of 51 cases; none received 5. The score measures overlap with the published hypothesis, so a low score can still be a valid alternative. Expert results and scoring rubric.

The implementation accepts a question, a background survey and a corpus of paper titles and abstracts. It returns ranked hypotheses, so researchers can inspect both the proposed connection and the literature behind it. Code and inputs.

RQ-Bench

Soumitra Sinhahajari, Navonil Majumder and Soujanya Poria test whether model judges agree with scientists in On the Limits of LLM-as-Judge for Scientific Novelty Assessment. They use research questions instead of full proposals, so the judges do not also have to assess an implementation or an experiment.

RQ-Bench contains 1,434 questions reconstructed from 746 computer science papers. Each record links the question to influential cited work, the gaps identified in that work and the target paper’s contributions. These author-anchored questions are references, not the only legitimate questions that could follow from the background. The released data includes the cited papers’ section text, making those connections inspectable. RQ-Bench dataset.

The main comparison is a blinded expert evaluation of 50 instances from computational linguistics and machine learning. Each pairs an author-anchored question with the best of five GPT-5.5 alternatives. On non-obviousness, the two experts preferred the reference in 78% and 56% of cases. Gemini-3.1-Pro and DeepSeek-V4-Pro instead preferred the generated alternative in 82% and 52% of cases. Expert–model agreement fell as low as 22%; expert–expert agreement was 60%.

Experts found many generated questions too narrow or tied too closely to their source papers. Explicitly evaluating that narrowness brought model assessments closer to expert judgments. The finding applies to these questions, judges and rubrics. Study and expert comparison.

IDEAgent

Feedback from a model judge can turn a rough proposal into a better one, but repeated revision tends to produce several polished versions of the same idea. IDEAgent, led by Varun Gumma, scores the whole set of ideas as well as each one, so improving one candidate does not crowd out the others.

IDEAgent generates candidates sequentially, retaining compact descriptions of earlier ideas. It compares new proposals with accepted ideas, rejected proposals and earlier versions. Repair addresses candidates that narrowly miss a threshold; refinement improves candidates that already qualify. Revised versions keep the same lineage, so a rewritten proposal does not count as a new discovery. Method and implementation.

Its Yield metric counts the largest mutually distinct subset that passes quality thresholds. The evaluation covers 32 topics, with two external model judges. With diversity and soundness thresholds of 7 and clarity of 6, the number of topics with at least one qualifying idea is:

These are rubric scores from model judges; no idea was tested experimentally. The two judges’ scores are averaged, and an idea qualifies only if it passes every threshold. Evaluation and Table 1.

Limitations

IDEAgent’s gains are measured by model judges, and RQ-Bench shows that such judges can rate a question highly when scientists find it adds little to its source papers. Using different models for generation and evaluation helps, but agreement with experts still has to be checked.

MOOSE-Chem’s targets come from published chemistry, so recovering them shows that a search over the literature can find useful connections. It does not show that a new candidate will work in the lab. Whether these systems help researchers choose better experiments can only be settled by expert review and real experiments.