MAPLE: A Multi-Axis Parametric Knowledge Extraction Lens for LLM Evaluation

The multi-axis diagnostic protocol is the clearest advance. By separating $\text{Gain}$ and $\text{Loss}$ at the surface level, tracking $\text{PRS}$ on unrecovered failures, and benchmarking against temperature sampling with $\text{SGR}$ and $\text{DGR}$, the authors expose intervention dynamics that single scores cannot capture. The benchmark construction is also commendably rigorous: they filter for unambiguous single-answer triplets and verify verbatim presence in OLMo2’s publicly documented pretraining corpus, reducing the risk of mistaking lack of exposure for extraction failure. The empirical narrative is well-supported by extensive appendices detailing hyperparameters, prompts, and raw counts.

The benchmark scope is narrow: it covers only single-answer English Wikipedia triplets, omitting multi-answer facts, non-English knowledge, and more complex reasoning, which limits generalizability to the broader knowledge landscapes LLMs encounter. Cross-method comparisons are also uneven because prompting techniques inject additional context and computational cost, fundamentally altering the input distribution, whereas steering and decoding do not; the paper does not control for these differences.\n\nMoreover, explicit corpus verification is performed solely for OLMo2 via exact-match filtering, while for Qwen2.5 and Llama3 the authors rely solely on temporal precedence, introducing unquantified exposure uncertainty. The paper also offers no mechanistic explanation for why decoding and steering methods perform markedly better on Llama3 than on OLMo2, leaving the observed cross-family gaps as descriptive rather than diagnostic.

The experimental evidence largely supports the paper’s core claims. The $\text{Gain}/\text{Loss}$ decomposition in Table 2, the $\text{PRS}$ distribution in Figure 2, and the $\text{SGR}/\text{DGR}$ values in Table 3 all align with the narrative that prompting yields the strongest recovery and that ITI is largely inert. However, the comparison across KEE families is not fully fair: prompting methods add context that changes the effective input, while steering and decoding operate within the original prompt boundary. Hyperparameters for each method are adopted from prior work without tuning on MAPLE, which may advantage some families over others. The beyond-sampling analysis is carefully calibrated—using $K=20$ with Kendall correlations above $0.89$—but the exclusive use of greedy decoding for baselines and interventions may overstate effect stability relative to realistic sampled generation.

Reproducibility is mixed. The paper documents model versions, KEE configurations, and prompts extensively in Appendices B and C, and provides raw counts in Appendix D, which aids transparency. However, the authors do not mention a publicly released code repository, benchmark dataset, or evaluation scripts. Because the benchmark requires reconstructing a multi-stage pipeline involving REBEL extraction, Wikidata filtering, LLM-as-a-judge cardinality classification, and infini-gram exact-match verification, independent reproduction would be labor-intensive despite the detailed descriptions. Full reproducibility would require sharing the exact 1,303 triplets, generation outputs, and inference scripts.