What Do Foundation Models for Market Microstructure Learn? A Controlled Study with Synthetic Limit Order Books
Autoregressive transformers trained on limit order book streams are usually evaluated only by output realism, leaving their internal mechanics opaque. This paper inverts the problem by training GPT-style models on synthetic order-flow simulators with known latent parameters, enabling exact verification against Bayes-optimal predictors and oracle entropy floors. It finds that next-token pretraining implements amortized Bayesian inference and online filtering, that latent states are linearly decodable from frozen activations, and that regime diversity—not model scale or token count—drives generalization.
The paper presents a tightly controlled design that yields unusually precise conclusions. The match to the exact Bayes-optimal predictive on F1 ($-3.4\times 10^{-4}$ nats per token, bootstrap CI spanning zero) and to oracle floors on F2–F4 is compelling evidence that small transformers perform amortized Bayesian inference in this domain. The authors are appropriately modest in generalizing to real markets, noting that "the simulators are simpler than real markets" and that "Real flow plausibly also has higher excess entropy, which would extend the scaling regime beyond the saturation we observe." The extension of the in-context-learning-as-Bayes paradigm to time-varying latent states is explicitly framed as advancing on Xie et al.'s fixed-concept setting: whereas Xie et al. proved that "the LM must infer a latent document-level concept to generate coherent next tokens during pretraining," this work shows filtering of a drifting intensity state within an episode. Overall, the study is a model of how to use synthetic ground truth to audit foundation-model behavior, though its external validity is bounded by narrow real-data evaluation.
The strongest evidence is the exact Bayes comparison on F1 and the oracle comparisons on F2–F4. The paper establishes that "the trained model matches it to $3.4\times 10^{-4}$ nats per token at 512 messages of context" and that gaps to the oracles are "the unavoidable cost of inferring it, not model suboptimality." The linear-probe analysis is carefully falsified against untrained-network and summary-statistic baselines, showing that the trained representation's advantage is largest for dynamical parameters at short contexts. The scaling analysis is methodologically sound because the known entropy floor lets the authors reinterpret what Chinchilla-style fits actually measure: "the exponents $\alpha_N, \alpha_D$ are poorly identified ($R^2_{\log}=0.87$): the loss surface is a floor plus a fast transient rather than a broad power law."
Several limitations deserve emphasis. The real-data validation is extremely narrow—one 2012 trading day across five NASDAQ tickers—which the authors themselves classify as "a controlled analysis of transfer, not evidence about current markets." Second, the closed-loop generation experiments reveal a genuine failure mode: for the agent-based family F3, generated streams suffer "a clear closed-loop gap for F3 (4.9 ticks), where feedback between the model and the book compounds small errors."
Third, the paper acknowledges that "all results use one fixed tokenization" and that sensitivity to the token schema is "not studied systematically," which matters because discretization choices could shape what the model can represent. While the scaling fits show early saturation, the authors are careful to note that this property is specific to regimes with a known floor and does not refute standard scaling laws for natural data where the floor is unknown.
The evidence broadly supports the central claims. The comparison to prior work is generally fair: the authors correctly note that Xie et al. (2022) fixed latent parameters, while their F4 family requires online filtering. The citation of Hoffmann et al. (2022) for the scaling law $L(N,D) = E + A N^{-\alpha_N} + B D^{-\alpha_D}$ is accurate, and the novelty lies in comparing the fitted $E$ to a known rather than unknown floor. However, the comparison to large-scale LOB foundation models like TradeFM and MarS is limited to conceptual positioning; the paper does not replicate or benchmark against these systems on shared metrics, relying instead on the argument that "All of these evaluations compare model output to data whose generative process is unknown." This is appropriate for the paper's goals but means the practical relevance for those systems is asserted rather than demonstrated.
The paper provides substantial experimental detail: simulator specifications (Table 1), exact priors, structural rules, token bin edges, model architectures, and training hyperparameters are all reported. The total compute budget is modest ("The complete study comprises roughly 100 runs and 55 GPU-hours"), making independent reproduction feasible in principle. However, no code repository or data generator is mentioned, and the exact Bayes and oracle computations depend on custom inference code that would need to be shared for rigorous verification. Reproduction is therefore possible from the text alone for the neural training, but blocked for the exact reference comparisons without additional author support.
Autoregressive transformers trained on limit order book (LOB) message streams are increasingly proposed as foundation models for market microstructure. Because the generative process of real markets is unknown, these models can be judged only by the realism of their output, leaving open what they actually learn. We make the question answerable by inverting the setup: we train GPT-style models on order flow from simulators whose latent parameters are known, so that the Bayes-optimal predictor, the generating state, and the exact data entropy serve as references. Measured against them, next-token pretraining on order flow implements amortized Bayesian inference over market regimes. The learned predictive matches the exact Bayes optimum to within a millinat per token and approaches the oracle floor even on a nonstationary family whose latent state drifts within an episode, a regime that requires filtering. The latent regime is linearly decodable from frozen activations, well above baselines, so a frozen-model read-out calibrates simulator parameters more accurately than windowed maximum likelihood at a fraction of the cost. In our experiments these capabilities are not produced by scale: capacity and token volume saturate early, and it is the diversity of training regimes, not the token count, that determines generalization to unseen markets. On real NASDAQ data, zero-shot transfer is set by how well the simulator prior covers real flow: an enriched mixture nearly closes the gap to an in-domain baseline, and pretraining beats from-scratch training at every budget across held-out tickers. Known ground truth turns average next-token loss, which is uninformative near the entropy floor, into an exact comparison against the Bayes-optimal and oracle predictors.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.