ReFormBench: Benchmarking the Stability of Response-Conditioned Format Correction in Large Language Models
Large language models often satisfy formatting requirements by chance rather than genuine correction, and standard pass rates cannot tell the difference. ReFormBench addresses this by using each model's own unconstrained response as a behavioral baseline, then tracking whether feature-and-mode targets are corrected across instruction variants. It introduces TCPR for complete compliance and EMCR for stable correction, showing that the two metrics yield different rankings across ten models.
The paper makes a well-motivated distinction between instruction compliance and response-conditioned format correction, and the benchmark design is conceptually elegant. The empirical separation of TCPR and EMCR rankings is a meaningful contribution, but the evaluation relies on an untested independence assumption for joint probabilities and an adjudication protocol that overwhelmingly favors one automated judge. These issues temper the confidence one can place in the absolute correction rates, even though the relative comparisons remain informative.
The response-conditioned baseline is a clean solution to a genuine problem: without knowing a model's unprompted output, one cannot tell whether a constraint forces a real change or merely preserves an existing format. The EMCR metric is carefully designed, linking target occurrences across variants and requiring success everywhere a target appears. The authors also deserve credit for an unusually thorough limitations section that flags the single-sample baseline, the factorized independence assumption, and the restricted experimental scope.
The evaluation rests on a strong and untested factorization assumption. Gao et al. explain that 'Under a factorization assumption, we treat the constituent pass events as independent and approximate joint success by multiplying their estimated probabilities' (Section 3.5, Eq. 9), yet the paper explicitly notes that 'dependence among pass events is not estimated' in the Limitations. If constraints within a response are positively or negatively correlated, the reported TCPR and EMCR values could be materially biased.
A second concern is severe adjudication bias toward GPT-5.4 Mini. Across 6,654 reviewed disagreements, the adjudicated label agreed with GPT 93.90% of the time versus only 6.10% for Gemini (Appendix Table A3), meaning the calibration procedure largely rubber-stamps one judge. The composition and prompting experiments are also limited to single models (Qwen3.6-27B and Qwen3.5-9B), and the four additional models are only assessed for catalog coverage, not correction stability.
The claim that compliance and correction are distinct dimensions is well supported: 'Qwen3.6-27B scores slightly below DeepSeek-V4-Pro on TCPR (65.46% versus 66.39%) but above it on EMCR (47.84% versus 44.81%)' (Section 4.2), and four models attain the highest EMCR point estimates across seven tasks. However, the prompting experiment is too narrow to support general conclusions about meta-prompt effectiveness—it improves Qwen3.5-9B, but only one model is tested. Similarly, the composition analysis shows a 6.05 pp drop for joint versus separate responses, yet this is demonstrated only for Qwen3.6-27B. The catalog coverage claim is stronger, with 'Coverage exceeds 95% for each additional model and reaches 95.75% pooled' and Spearman $\rho = 0.804$ across four held-out models.
The authors state that 'the code and data are available at <https://anonymous.4open.science/r/ReFormBench/>' (Abstract), and model configurations are detailed in Appendix A.10. Reproduction is nevertheless hindered by a pipeline that depends on proprietary models—GPT-5.4 for feature extraction, instruction generation, and judging; and Gemini 3.1 Flash-Lite as a secondary judge—whose exact weights and behavior may not be stable or accessible. The judgment calibration also requires human adjudication of thousands of disagreements, which is labor-intensive and not fully documented for external replication. Full reproduction therefore requires not only the released dataset but also replicating the bespoke judge stack and manual review protocol.
Reliable use of large language models requires responses to satisfy formatting requirements across changing instructions. Standard compliance scores do not distinguish preserving a model's observed format from correcting it, or reveal whether that correction holds across variants. We introduce ReFormBench, a benchmark that uses each model's unconstrained response to the same question as its formatting baseline. It links targets by feature and require/forbid mode, evaluating their instantiated requirements across variants. ReFormBench comprises 5,000 difficulty-screened three-constraint test instances derived from 910 source questions and covering 66 formatting features. TCPR measures complete instruction compliance, while EMCR measures individual and joint correction of targets opposed to the annotated baseline state. Across ten models, EMCR ranges from 34.04% to 64.72%, and the highest three-target correction estimate is 50.02%. Compliance and correction scores yield different model rankings and task-level profiles. A prompting experiment on Qwen3.5-9B improves both metrics, and the catalog covers over 95% of extracted instances on four additional models. ReFormBench makes response-conditioned correction across instruction variants a measurable dimension of format following. The code and data are available at https://anonymous.4open.science/r/ReFormBench/.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.