Generating and evaluating research ideas

Finding an idea is only part of the problem

When a model proposes a research idea, the difficult decision is often what to do next. Is the connection to earlier work convincing? Has someone already tried it? Would the proposed experiment settle an interesting question? A fluent paragraph gives the researcher something to consider, but it can leave all of these questions unanswered.

MOOSE-Chem gave us a way to study the first part of this process by asking models to recover chemistry hypotheses from earlier literature. Once a system can produce a list of plausible candidates, however, somebody has to choose between them. RQ-Bench examines whether model judges can make that choice in agreement with scientists. IDEAgent explores how feedback can improve the candidates themselves, while keeping enough variety to give a researcher a real choice. The connection is a shared research problem; the three methods were evaluated separately.

MOOSE-Chem: recover a hypothesis from its background

A chemistry hypothesis may connect a known research problem with results from several other papers. MOOSE-Chem makes that connection the object of search: retrieve possible inspirations, combine them into candidate hypotheses, then rank the candidates. Its search adds inspirations over multiple steps and uses mutation and recombination to explore alternative combinations.

The benchmark contains 51 chemistry papers first made public in 2024. Chemistry PhD students separate each paper into its background, inspirations and hypothesis. The system receives the background and a literature corpus containing the inspiration papers, but not the target hypothesis; the tested models have training cutoffs in 2023. This is a retrospective rediscovery test, not a report of new wet-lab discoveries. MOOSE-Chem, ICLR 2025.

In the 300-paper-corpus experiment, experts examined four candidates per background, preselected using GPT-4o’s similarity scores. The best candidate received a Matched Score of 3 or 4 in 21 of 51 cases; none received 5. These scores measure overlap with the reference hypothesis, not experimental success. A low match can also miss a valid alternative direction. Expert results and scoring rubric.

The implementation accepts a question, a background survey and a corpus of paper titles and abstracts. It returns ranked hypotheses, so researchers can inspect both the proposed connection and the literature behind it. Code and inputs.

Ranking creates the next problem: if the same kind of model proposes an idea and assesses its promise, how much confidence should we place in that assessment?

RQ-Bench: ask whether the judges agree with scientists

Soumitra Sinhahajari, Navonil Majumder and Soujanya Poria study this question in On the Limits of LLM-as-Judge for Scientific Novelty Assessment. They focus on research questions rather than complete proposals, so the comparison does not also require judging an implementation or an experiment.

RQ-Bench contains 1,434 questions reconstructed from 746 computer science papers. Each record links the question to influential cited work, the gaps identified in that work and the target paper’s contributions. These author-anchored questions are references, not the only legitimate questions that could follow from the background. The released data includes the cited papers’ section text, making those connections inspectable. RQ-Bench dataset.

The telling comparison is a blinded expert evaluation of 50 instances from computational linguistics and machine learning. Each pairs an author-anchored question with the best of five GPT-5.5 alternatives. On non-obviousness, the two experts preferred the reference in 78% and 56% of cases. Gemini-3.1-Pro and DeepSeek-V4-Pro instead preferred the generated alternative in 82% and 52% of cases. Expert–model agreement fell as low as 22%; expert–expert agreement was 60%.

Experts found many generated questions too narrow or tied too closely to their source papers. Explicitly evaluating that narrowness brought model assessments closer to expert judgments. This is evidence about these questions, evaluators and rubrics—not proof that every automated scientific judgment is wrong. Study and expert comparison.

IDEAgent: improve the set, not just one proposal

Even with an imperfect evaluator, feedback may help a researcher develop a rough proposal into something worth examining. The danger is that repeated revision produces several polished versions of essentially the same idea. IDEAgent, led by Varun Gumma, therefore evaluates the collection as well as each proposal: improving one candidate should not come at the expense of losing the alternatives.

IDEAgent generates candidates sequentially, retaining compact descriptions of earlier ideas. It compares new proposals with accepted ideas, rejected proposals and earlier versions. Repair addresses candidates that narrowly miss a threshold; refinement improves candidates that already qualify. Revised versions keep the same lineage, so a rewritten proposal does not count as a new discovery. Method and implementation.

Its Yield metric counts the largest mutually distinct subset that passes quality thresholds. The evaluation covers 32 topics, with two external model judges. With diversity and soundness thresholds of 7 and clarity of 6, the number of topics with at least one qualifying idea is:

These are rubric-based outcomes, not counts of experimentally validated findings. The judges’ scores are averaged, and the criteria must be read together; a high novelty score alone does not qualify an idea. Evaluation and Table 1.

Choosing which idea to pursue

IDEAgent makes the tension in this work especially clear. Repair and refinement produce more proposals that pass the evaluation criteria, but those improvements are measured by model judges. Soumitra’s study shows why that distinction matters: scientists can read the same highly rated question and find that it adds little to the papers it came from. Using separate models for generation and evaluation helps keep the roles apart, but agreement with experts still has to be checked.

MOOSE-Chem offers a different kind of reference because the target hypotheses were pursued in published chemistry research. Recovering their main ideas shows that useful connections can emerge from a search through the literature, although it leaves open whether a new candidate will work in the lab. The question that follows is whether these systems help researchers choose better experiments. Answering it will require following proposals beyond their generated text and scores, through expert review and into the experiments themselves.