OR for AI That Does OR: Routing LLMs up the Escalator inside the OSCAR Framework
LLM-generated optimization models often silently misrepresent constraints, producing solver-ready code that is feasible-looking but incorrect. OSCAR addresses this by introducing an offline Simulator certified against labeled decision examples to compare successive formulations, coupled with a Coder–Reviewer improvement loop and a theoretically grounded Escalator routing policy that allocates calls across models of differing cost and capability. On five benchmark problems, the framework lifts two small open-weight LLMs from roughly 29–48% single-shot accuracy to 95–100% while reportedly costing several times less than frontier coding agents.
The paper presents a compelling, OR-informed approach to LLM-based optimization modeling. The core idea—using a certified Simulator as a persistent external oracle and a cost-ordered escalation policy to route LLMs—is both practically motivated and theoretically supported. The authors report that "OSCAR achieves 95% to 100% accuracy at the reported settings using two small open-weight LLMs, each deployable locally on a single GPU," even though "Their single-attempt accuracies average 29% and 48%." Nevertheless, the empirical evidence is limited to five selected problems and a narrow slice of FrontierOR, so the breadth of the claimed savings remains unproven.
The Simulator-as-oracle design is a genuine advance over prior frameworks that stop at feasibility or rely on LLM self-verification. The authors show empirically that Simulator explanations materially improve diagnosis: "The Reviewer identified the true defect in 11 of the 24 cases without the report and in 21 with it," and the share of repairs ending with a correct formulation rose from "4% without the report to 18% with it." Theorem 4.7 provides a strong theoretical backbone, proving that the Escalator policy achieves a prior-free competitive ratio bounded by $CR(\pi^{\rm EP}) < 5.2 \kappa_q(\mathbf{s}^*)$. The cost accounting is also unusually transparent, with OSCAR averaging $0.193 per run versus $0.600 for Codex and $1.127 for Claude Code.
The evaluation rests on only five hand-selected problems, and the authors disclose that "Screening also revealed ambiguous descriptions, unstated checker input formats, and inconsistencies among problem descriptions, reference formulations and solutions, and feasibility checkers," which led them to exclude many FrontierOR tasks. This selection process weakens external validity. A deeper limitation is the focus on solution correctness for a single target instance rather than general formulation correctness: "We assess a formulation by solution correctness: whether its solution is feasible and optimal for that instance in the real system. By contrast, formulation correctness requires the model to be correct for every instance consistent with the description." The Simulator construction pipeline is also not fully unattended; the authors note that on a few outside-benchmark problems, human help was required mainly with data formatting, undercutting the fully automated vision.
The one-shot baseline comparisons are statistically credible with 100 runs per model, but the frontier-agent comparison is limited to five runs and the authors concede that these agents receive only description and instance data without OSCAR's labeled examples, so it is not an apples-to-apples test of raw capability. They also note that "updates by the providers to their closed models may change their performance, making results from more extensive testing difficult to reproduce later." The resampling ablation strengthens internal validity: on Routing, Packing and Scheduling, where one-shot success is rare, "resampling to match OSCAR's accuracy costs 1.6 to 3.3 times as much," supporting the claim that the Reviewer-and-revision loop adds value precisely when one-shot generation is unreliable.
The authors publish code, certified Simulators, and labeled examples on GitHub, which aids reproduction. However, independent replication is blocked by substantial manual preparation: the authors state that "All five problems required data processing before Simulator construction, mainly to map reported decisions to instance elements," and each problem needs a tailored set of labeled decision examples and decision syntax. In addition, the experimental hyperparameters—attempt counts per configuration and the restart interval $r=3$—appear to be tuned manually on the benchmark rather than derived from Algorithm 1, and the paper does not report sensitivity of accuracy or cost to these choices. Without automated procedures to set them on new problems, transferability is uncertain.
Large language models can translate business descriptions into optimization models, but executable code may misrepresent constraints or objectives. A solver can then return an optimal solution to the wrong problem. Even when the solution satisfies the intended operating rules, a better plan may exist. For organizations that repeatedly use optimization modeling, an LLM-based framework should produce accurate formulations at low cost and, ideally, run locally. We study how to verify improvements and allocate attempts across LLMs that differ in price and capability. We develop OSCAR (Optimization modeling by Simulator, Coder, And Reviewer), which uses an offline Simulator certified against labeled decision examples to compare candidates and continues searching beyond feasibility. We model the search for the next certified improvement as sequential decisions under unobserved difficulty: which LLMs to call and when to stop. In a simplified known-prior setting, we give conditions under which cost-ordered escalation is optimal. For general menus, we derive a prior-free competitive guarantee. On five benchmark problems, OSCAR achieves 95% to 100% accuracy at the reported settings using two small open-weight LLMs, each deployable locally on a single GPU. Their single-attempt accuracies average 29% and 48%. In five runs per problem, Codex and Claude Code incur average token costs 3.1 and 5.8 times OSCAR's, respectively. OSCAR supports open-weight models locally or in the cloud, depending on budget and confidentiality requirements. Firms should maintain labeled decision examples of feasible and infeasible decisions to clarify plain-language operating rules. OSCAR follows these labels when an LLM's interpretation conflicts with them. As LLM capabilities and prices change, OSCAR's simple operating rules and adjustable settings help firms adapt their model choices and benefit from these advances.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.