Code Owns the Simulation, Jev Owns the Evaluation
Judgment models that return option probabilities without reasoning text are attractive as fast agent policies, but their competence boundary is unclear. This paper argues that such models excel at evaluation (judging given options against described criteria) yet fail at simulation (predicting missing consequences like opponent moves or physical dynamics). By cleanly separating the two, the authors show that supplying simulated outcomes via code restores strong performance across matrix games, text adventures, and robot control. The result is a sharp diagnostic of where fast judgment models can be trusted and a principled architecture for combining them with external simulators.
The paper delivers a crisp, cross-domain diagnosis of a single judgment model—Jev—by distinguishing decisions that need only evaluation from those requiring simulation. Its central thesis, that code should own the simulation while Jev owns the evaluation, is well operationalized and largely supported by the data. However, the title risks overgeneralizing from one proprietary API version to a universal principle, and several experiments rely on small samples or privileged simulators that weaken the practical punch. The work is best read as a careful case study with suggestive architectural implications rather than a definitive proof of generality across all fast scorers.
The strongest contribution is the deconstruction of failure modes via the surface-shortcut framework and the separate-call probes. The authors do not merely show that Jev fails; they trace failures to a specific mechanism: the inability to perform simulation and evaluation within a single call. Table 2 shows that separating the two calls raises accuracy from 33% to 94% in misleading games, and similar gains appear in ALFWorld. The domain span—from Cognitive Reflection Tests to robot torque control—gives the argument surprising breadth.
The evidence is strictly limited to a single model, Jev (jev-1.13.0), and the paper explicitly notes it "cannot yet say whether the effect is specific to Jev" (Section 8). Yet the framing—"Code Owns the Simulation, Jev Owns the Evaluation"—reads as a targeted law rather than a hypothesis for fast scorers in general. In robot control, the generalization claim rests on only 40 episodes across five new instructions, and the comparison hand-written controller was built for the native task, not for these variants, making the 22 versus 3 success comparison lopsided. The CRT benchmark is rendered trivial by the multiple-choice format; the 99% correct rate is uninformative compared with open-ended human performance and does not meaningfully stress the evaluation capacity claimed. Finally, the ALFWorld remedy assumes an exact, forkable copy of the game environment, which is unavailable in most real-world settings, and the authors acknowledge that "the cost grows quickly with the number of commands" (Section 8).
The evidence is internally coherent: predictions (i)–(v) are laid out in Section 2 and largely confirmed across Sections 3–6. The behavioral modeling in Appendix C is thorough, showing Jev acts like a quantal level-1 player ($\lambda \approx 5.5$ over average payoffs). Yet comparisons to related work are thin. The authors distinguish their split from Tree of Thoughts and LLM-Modulo, but do not compare empirically to these methods or to standard model-based RL pipelines that similarly separate world models from policies. The absence of strong ALFWorld baselines—such as ReAct or BERT-based scorers—makes it hard to gauge whether "code owns the simulation" is novel or simply rediscovers existing planning-while-scoring architectures. As they rightly note, prior work "report[s] how often an agent succeeds," whereas their focus on individual decision types is a welcome methodological advance (Section 7).
Reproducibility is mixed. The authors promise release of code, raw API runs, and recomputation scripts, and Appendices provide meticulous prompts and model version details. However, the model itself is a commercial API whose outputs drift across versions: "Jev is accessed through a public API, so its outputs can change across versions. Repeating a single call changes its probabilities by 0.04 on average" (Reproducibility Statement). The ALFWorld fork-based lookahead and the custom inverse-kinematics skill library for robot control are complex engineering artifacts whose fidelity would need exhaustive open-source documentation to replicate. Cost is low ($4), but the primary barrier to reproduction is the proprietary, evolving target model.
Judgment models such as \jev{} return, in a single call and without reasoning text, a probability for each described option. This makes them attractive as an agent's action-selection layer, but it is unclear which decisions they can be trusted with. We test \jev{} on reflection tests, one-shot matrix games, the text game ALFWorld and robot control, and find a sharp boundary. \jev{} succeeds when the right option can be judged from what the input describes, which we call \emph{evaluation}. Specifically, it solves 99\% of the counterintuitive Cognitive Reflection Test questions. However, it fails when the right option depends on \emph{simulation} (i.e., predicting something not in the input), such as the opponent's action or the subgoal that must come first. In games, \jev{} plays suboptimally as if its rational opponent acted at random, because the opponent's action is not given. In ALFWorld, \jev{} favors commands that mention an object or place named in the task description. For example, given the task ``put a clean knife in the drawer'', \jev{} carries an unwashed knife straight to the drawer instead of first washing it at the sink. Surprisingly, many of these failures are not due to a lack of knowledge. Asked separately what the opponent will do, \jev{} usually answers correctly, and it responds well given the opponent's action. It fails when one call must both perform the simulation and evaluate based on it. This suggests letting code make the prediction or simulation. When code supplies it, such as a lookahead in ALFWorld and physics simulation in robot control, \jev{} becomes an expert controller through its general evaluation ability.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.