How Do LLMs Understand Legal Philosophy? Diagnosing Legal-Philosophical Reasoning in LLMs

The benchmark design is the paper's strongest asset. The fourteen jurisprudential dimensions are grounded in legal theory and were refined with a law professor; the three annotators all have formal legal training, and the 9,240 labels are treated with appropriate caution by retaining all three votes rather than forcing a single gold label. The disagreement taxonomy—Opposite, Uncertain, and Overconfident—adds useful diagnostic granularity, and the finding that models hesitate when annotators disagree suggests the task captures genuine interpretive ambiguity rather than random noise.

The central conceptual flaw is equating annotation agreement with philosophical understanding. The authors claim that because frontier LLMs recover tendencies 'about as consistently as law-trained human annotators do,' the models 'can understand legal philosophy as law-trained humans do.' This is an unsupported leap: high agreement with noisy human judgments proves alignment with a particular interpretive convention, not comprehension of jurisprudence. Given that Fleiss' $\kappa = 0.474$—moderate agreement—matching this bar principally shows the model replicates the same ambiguity.

Second, the JAJP task does not cleanly isolate philosophical reasoning. The authors admit that 'there is certain possibility that LLMs acquire knowledge of judicial decisions and recurring jurisprudential tendencies during pretraining,' and while withholding the judge's name mitigates explicit memorization, the 14-dimensional vector may still act as a fingerprint of a specific justice's ideology, especially since $Q2$ explicitly maps to Modern American Conservatism vs. Liberalism. The benchmark therefore cannot yet distinguish reasoning from recall.

The evidence supports the narrow claim that jurisprudential profiles improve judgment prediction above a $50\%$ No Profile baseline, and the ablation in Figure 6 shows most individual tendencies carry predictive signal. The comparison to related work is fair: the authors correctly distinguish prior judge-aware prediction methods—which use identity or learned embeddings—from their own explicit profile conditioning. Nonetheless, they do not fully rule out that the profile merely encodes judge-recognizable correlates; an uninformative perturbation would not raise Overall accuracy, but a judge-specific ideological signature would. Until the authors test with counterfactual profiles or unfamiliar judges, the evidence supports 'predictive utility' better than 'philosophical reasoning.'

Reproduction is blocked by data and model opacity. The authors state, 'We will publicly release LePhilBench upon acceptance,' so the dataset and code are not yet available. Prompts are meticulously documented in Appendix B, but critical generation hyperparameters—temperature, top-p, max tokens—are never reported. Model sizes are listed as 'unknown' in Table 4, and the paper uses unreleased or future model versions (e.g., GPT-5.6, Claude Opus 5), further limiting independent replication. The aggregation protocols (Mean, Exponential Decay, Random Walk) are formally specified, which helps, but without the data and without confirmed reproducible API settings, the results cannot be verified.