Autoregressive Drillhole Modelling Under Distribution Shift
The paper tackles autoregressive modelling of mineral-exploration drillholes, where lithology is revealed sequentially from shallow to deep. It introduces DrillBench, a 49,671-hole benchmark across Western Australia with a graded transfer spectrum from local prediction to cross-province generalisation, evaluating next-layer prediction (macro-F1) and free-running generation (joint $\mathrm{KL}(\text{true}\|\text{generated})$). The core finding is a stark transfer boundary: spatial and geochemical conditioning boosts local accuracy but collapses under shift, while sequence-only autoregressive models degrade more gracefully. To address this, the authors propose a backbone-agnostic recipe combining large-scale pretraining on historical archives with spatial retrieval of neighbouring lithology, showing that retrieval aids local cover prediction and pretraining aids bedrock and cross-province transfer.
This is a well-motivated empirical contribution that clearly identifies and mitigates a domain-specific distribution-shift problem. The benchmark curation is careful, the transfer boundary is demonstrated with extensive baselines, and the pretraining-vs-retrieval decomposition is geologically interpretable. However, cross-province evaluation is limited to a single target province, and the promised code and checkpoints are not yet available. The paired bootstrap analysis also shows that some gains of the full recipe over its pretraining-only variant are not significant on the ood-regional and pilbara-test splits, weakening the claim of universal complementarity.
The graded split design and comprehensive baseline suite make the transfer boundary claim convincing. The paper shows that retrieval and pretraining play complementary, geologically grounded roles: nearby drillholes help most in weathered cover where local continuity is reliable, while pretraining supplies broader geological priors that matter more in bedrock and under cross-province shift.
First, cross-province transfer is evaluated on only one target province (Pilbara), so it remains unclear whether the findings generalise to other province pairs or whether the Pilbara result is idiosyncratic; the authors acknowledge that extending the benchmark requires reconciling each company's logging vocabulary and leave it to future releases. Second, although absolute coordinates are withheld from the model input, the retrieved neighbour votes themselves encode spatial structure, so the retrieval-augmented model is not fully spatially blind. Third, the zero-shot LLM experiments are hard to interpret because the authors note that strong out-of-distribution LLM performance may reflect exposure to the public DASC records during pretraining, which would constitute data contamination rather than true zero-shot geological reasoning.
The main claims are supported by systematic ablations across 37 methods and paired bootstrap analysis. The evidence shows that the full recipe significantly outperforms plain sequence models on all splits, but the direct comparison between GRU+PT+R and GRU+PT yields paired differences of −0.4 and −1.1 percentage points on ood-regional and pilbara-test with intervals that include zero, so the added value of retrieval under strong geological shift is not clearly established. Comparisons to related work are fair: the paper correctly distinguishes prior interpolation and masked-modelling approaches from the autoregressive next-stratum formulation, and shows that simply giving spatial baselines the same larger retrieval pool does not close the performance gap.
Implementation details are thorough, with full hyperparameters, curation pipelines, and evaluation protocols in the appendices. However, the data, code, and pretrained checkpoints are only promised upon acceptance, so independent reproduction is currently impossible. The experiments are modest in compute (~400 GPU-hours on V100/A6000/H100), and the fixed training schedule without checkpoint selection is a reproducible choice, though the lack of released seed lists in the main text is a minor obstacle.
Autoregressive modelling has achieved remarkable success in language and sequence tasks by learning to predict future states from previous observation. Mineral-exploration drillholes provide a natural but largely unexplored setting for this paradigm: as drilling proceeds, lithology is revealed sequentially from shallow to deep, making prediction of deeper strata inherently autoregressive. Existing drillhole modelling, however, is dominated by spatial interpolation and reconstruction, or largely rely on masked modelling, leaving strictly autoregressive prediction largely underexplored. We introduce DrillBench, a benchmark of 49,671 Western Australian drillholes for next-layer prediction and autoregressive stratigraphic generation across a graded transfer spectrum, from local prediction through spatial shift to cross geological province transfer. Benchmarking classical, geostatistical, and neural models reveals a clear \emph{transfer boundary}: spatial and geochemical conditioning provides large local gains but deteriorates sharply under stronger shift, whereas lithology-sequence autoregressive models transfer more robustly. Guided by this finding, we develop a backbone-agnostic recipe combining large-scale pretraining on historical drillholes with spatial retrieval of neighbouring lithology. Retrieval is most effective in weathered cover, when local spatial continuity remains informative, whereas pretraining contributes more strongly in bedrock and under broader geological shift. Together, they retain strong local performance while improving generalisation under spatial and cross-province shift, most markedly on the most distant splits. The benchmark and code are available at https://github.com/yihaoding/drillbench.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.