Papers Explained 406: Answer Matching | by Ritvik Rastogi | Jul, 2025

This paper argues that multiple-choice benchmarks, historically used for evaluating language fashions, endure from a essential flaw: they permit fashions to use discriminative shortcuts and reply questions with out really understanding or producing the right response. The authors suggest “reply matching” as a superior various for evaluating the generative capabilities of language fashions.

The challenge is obtainable at GitHub.

Evaluating generative fashions entails figuring out if a generated response (R) is a member of the set of right solutions (AQ) for a given query (Q). That is troublesome when there are numerous potential right responses (|AQ| > 1).

• If there’s just one right response (|AQ| = 1), analysis might be executed through string matching (e.g., in NLP benchmarks like SQuaD).
• In arithmetic, even with infinitely many equal expressions, rule-based symbolic equivalence can typically suffice.
• In pure language, many paraphrases can convey the identical that means, making direct string matching inadequate.

A number of alternative makes an attempt to bypass the |AQ| > 1 downside by offering the mannequin with a query (Q), a single right reply (a), and a number of other incorrect selections (distractors, wi). The mannequin’s response is marked right provided that it matches ‘a’.

This reduces the set of right solutions to a singleton (‘a’), simplifying automated grading.

Nonetheless, this strategy basically adjustments the duty. As an alternative of requiring the mannequin to generate an accurate response (a generative downside), it shifts to requiring the mannequin to discriminate between right and incorrect selections (a discriminative downside).

To reveal the extent to which multiple-choice benchmarks might be solved discriminately, a language mannequin (Qwen3–4B) is finetuned to foretell the right reply given solely the alternatives with out the query. For finetuning, the devoted prepare cut up of the dataset is used each time out there; in any other case, the check set is randomly cut up 50–50, coaching on the primary half and evaluating on the second half (held-out).

• Strikingly excessive accuracies might be achieved throughout fashionable datasets utilizing choice-only shortcuts.
• Accuracy past likelihood raises considerations about whether or not the dataset really displays generative query answering, because the mannequin doesn’t even know the query.

A easy option to forestall discriminative shortcuts is by not offering the mannequin with selections within the enter. The mannequin is solely tasked with offering a free-form response R, after which, one other mannequin checks whether or not the response R matches with a offered reference reply a, termed as Reply Matching.

Alignment is measured utilizing Scott’s π, an inter-annotator settlement metric beneficial in latest LLM-as-a-Choose literature.

Whereas this strategy has been sometimes thought of within the LLM-as-a-Choose literature, the excellence is essential. In conventional LLM-as-a-Choose duties, a choose mannequin J should confirm the correctness of a response R to a query Q with out entry to a reference reply, main to varied points. In distinction, utilizing a language mannequin for reply matching entails checking if the mannequin response is semantically or functionally equal to the reference reply within the context of the query, which is intuitively simpler than verifying the correctness of an arbitrary response.

def get_judge_prompt_with_gt(
query: str,
goal: str,
response: str,
incorrect_options: str | None = None,
cot: bool = True,
) -> str:
"""
Generate a immediate for the choose with floor fact.
Args:
query: The query being requested.
goal: The bottom‑fact reply.
response: The response to guage.
incorrect_options: Non-compulsory string containing incorrect choices.
cot: Whether or not to incorporate a sequence‑of‑thought (COT) instruction.
Returns:
A formatted immediate string for the choose.
"""
# The response can comprise extra data than the bottom fact.
# It may be extra particular (e.g., “Labrador” vs. “canine”) or listing extra
# right solutions, nevertheless it should cowl every part within the floor fact.
# Paraphrasing is appropriate.
immediate = f"""Your process is to guage whether or not the given response to a query
matches a offered floor‑fact reply or not. You're given a query, a
floor‑fact reply, and the response you will need to choose.
For a response to “match”, it should embrace a minimum of as a lot data because the
floor‑fact reply.
The response could comprise extra data than the bottom fact. It may be extra
particular (for instance, “Labrador” is extra particular than “canine”) or listing
extra potential right solutions, nevertheless it should cowl every part talked about in
the bottom fact. Paraphrasing is appropriate.
For numeric solutions, the relative error—outlined as
|response − ground_truth| / imply(response, ground_truth)—have to be lower than 1 %.
Attainable judgments:
"0": The response doesn't match the bottom‑fact reply.
"1": The response matches the bottom‑fact reply.
Query: "{query}"
Floor fact: "{goal}"
"""
if incorrect_options:
immediate += f"n{incorrect_options}"
immediate += f"""
Response: "{response}"
Your job is to ONLY verify whether or not the given response matches the bottom‑fact
reply within the context of the query. You DO NOT must assess factual
correctness. That is a part of an automatic analysis course of, subsequently you
MUST OUTPUT your last reply as "0" or "1" in  tags.
"""
if cot:
immediate += (
'nThink step-by-step and finish your response with '
'0 OR 1 TAGS.'
)
else:
immediate += (
'nYOU SHOULD ALWAYS END YOUR RESPONSE WITH '
'0 OR 1 TAGS.'
)
return immediate

Alignment on MATH Questions

The MATH dataset is used to judge LLMs, with the MATH-Confirm library offering rule-based ground-truth evaluations. A parallel multiple-choice model can be out there.

• Reply Matching: Reply matching, even with a comparatively smaller mannequin (1.7B parameter Qwen3), achieves near-perfect alignment with the ground-truth (π = 0.97). Bigger fashions like DeepSeek v3 (671B parameters) additionally carry out nicely as matchers (π = 0.98).
• LLM-as-a-Choose: Utilizing LLMs as judges reveals solely modest settlement with the bottom fact (π = 0.72) even with very giant fashions.
• Customary A number of Alternative (MCQ): Customary MCQ analysis has low alignment (π = 0.26) as a consequence of false positives, as the duty is a neater discriminative downside.
• A number of Alternative Verification: This methodology presents every alternative individually and requires the mannequin to independently confirm its correctness. It estimates related accuracy to reply matching however has poorer alignment (π = 0.43) than reply matching however higher than commonplace MCQ.
• A number of Alternative Cloze: This methodology solely gives the query and measures completion likelihoods over all selections. It has the bottom alignment (π = 0.07), indicating outcomes virtually unbiased from the ground-truth. It’s a non-generative probability analysis, which can not go well with fashionable fashions that use chain-of-thought technology.

Alignment on A number of Alternative Knowledge in Pure Language

Variants of MMLU-Professional and GPQA-Diamond for generative analysis are created, offering solely the query to the mannequin and utilizing the right alternative as a reference reply. Questions are filtered to make sure they may very well be answered with out selections and had a singular right reply, addressing the problem that many questions depend on the alternatives to convey the supposed reply’s model and specificity.

800 mannequin responses are manually evaluated for correctness, and people additionally rated the specificity of questions and reference solutions. The examine targeted on a subset of 493 MMLU-Professional questions and 126 GPQA-Diamond questions that met the specificity standards.

The alignment of various automated evaluations (LLMs as judges and LM matchers) are in contrast with human judgments, discovering that LM matchers constantly achieved greater settlement (Scott’s π).

Error Evaluation: Error evaluation of LLM-as-a-judge revealed a excessive price of false positives, the place the choose incorrectly recognized responses as right.

Smaller fashions (Qwen3) confirmed near-human degree alignment, whereas bigger fashions (DeepSeek, Llama) had settlement throughout the vary of inter-annotator disagreement. This aligns with findings that smaller fashions with a reference reply carry out higher than bigger fashions with out one.

The implications of adopting reply matching throughout the benchmarking ecosystem are examined, specializing in its influence on mannequin rankings, analysis prices, replicability of benchmark outcomes, and future dataset improvement.

Influence on Mannequin Rankings

• Mannequin rankings change considerably when shifting from MCQ to answer-matching on generative responses.
• Chat-optimized proprietary fashions (e.g., GPT variants, Claude 3.5 Haiku) have a tendency to enhance their rating in generative analysis.
• Open-weight fashions optimized for multiple-choice benchmarks (e.g., R1-Distill Llama 70B, WizardLM 2) can expertise marked drops in rating.
• This highlights that benchmark conclusions and mannequin choice critically rely on the chosen analysis protocol.

Addressing Benchmark Saturation

• Benchmarks that seem saturated as a consequence of excessive cardinal values in MCQ format reveal substantial headroom when switched to generative analysis.
• For instance, a drop of over 20% in accuracy was noticed throughout fashions on GPQA Diamond when evaluated generatively, with finest fashions scoring 60%.
• Current datasets might be repurposed for free-form evaluations, persevering with to function significant indicators of progress.
• Human-verified free-form subsets of MMLU-Professional and GPQA-Diamond are publicly launched to facilitate this.

Price-Effectiveness of Reply Matching

• Evaluating fashions utilizing reply matching, even with frontier fashions like DeepSeek v3, is not any dearer than MCQ evaluations.
• Utilizing fashions with excessive human alignment, reminiscent of Llama-4-Scout, may even make reply matching cheaper than MCQ.
• Analysis prices are primarily pushed by the size of mannequin responses; fashions generate longer responses for MCQs as they typically try free-form options first.
• The extra price of reply matching (operating a language mannequin as a matcher) is marginal in comparison with the technology overhead, as matching is a neater process than fixing from scratch.

Reliability of Reply Matching

• Reproducibility: Issues about reproducibility with LM-as-grader evaluations are mitigated by the improved capabilities of open-weight fashions (e.g., DeepSeek-v3, Qwen3–4B) and by conducting evaluations at zero temperature.
• Robustness: Rankings stay extremely secure even when utilizing totally different fashions for reply matching (e.g., DeepSeek-v3, Llama-4-Scout, Qwen3–4B).
• No proof of self-preference bias was discovered, not like in conventional LLM Choose setups.
• Whereas adversarial setups weren’t examined, the textual content suggests reporting MCQ outcomes alongside LM-based reply matching to lift suspicion if efficiency is solely excessive on the latter.

Intrinsic Validity and Timing

• Whereas MCQ has good assemble validity for measuring multiple-choice check efficiency, it lacks validity for generative capabilities.
• Older fashions lacked the intrinsic validity required for reply matching, performing poorly on this process.
• Solely with the latest technology of fashions, which obtain near-human settlement ranges, has reply matching emerged as a clearly superior mode of analysis.

Changing A number of Alternative Benchmarks

• Current multiple-choice benchmarks might be reused for reply matching, however with a caveat: many MCQ questions aren’t particular sufficient on their very own and depend on selections for disambiguation.
• Filtering such questions can scale back dataset measurement by greater than half and skew class distribution in direction of STEM (the place distinctive solutions are extra widespread).
• This motivates the creation of recent questions which can be extra particular or present a listing of reference solutions for a number of prospects.
• New benchmarks (e.g., SimpleQA, BrowserComp) are already designing questions with single, indeniable, brief solutions, which is taken into account extra fruitful than creating higher-quality distractors for MCQs.

Reply Matching Outperforms A number of Alternative for Language Mannequin Analysis 2507.02856