DeepOpen: Decision-Oriented Intent Classification on BANKING77 and CLINC150
This paper studies fine-grained intent classification on BANKING77 and CLINC150 by adapting a flexible decision-model interface (Laya) into fixed-label classification heads. For BANKING77 it applies sharpness-aware minimization (SAM) to a single encoder, while for CLINC150 it fuses probabilities from one Laya specialist and three DeBERTa-v3-large models. Rather than claiming a universal advance, the authors foreground statistical uncertainty, selection effects, and execution-path drift, making the work a transparent---if narrow---benchmark audit.
The paper reports observed test accuracies of 94.1234% on BANKING77 and 98.0222% on CLINC150, yet its central contribution is methodological caution rather than a new state of the art. The authors explicitly state that the SAM continuation does not yield a statistically reliable gain and that the four-encoder fusion is an expensive, task-specific configuration without a unified causal story.
The strongest aspect of the paper is its transparency. The authors openly report that 'Selecting seed 13 as the headline is a separate, post-test display choice' and subject the same historical checkpoint to multiple execution paths, finding that 'The audit reports matching data and weight hashes but unresolved exact prediction reproduction.' They also show prediction drift between BF16 and FP32 evaluation, a rarely documented form of benchmark fragility.
The two reported systems are tied to different datasets and interventions, so they do not demonstrate a general method. The paper admits that 'The two tasks use different architectures and objectives, so their scores do not establish a common causal improvement,' and there is no cross-dataset ablation of the fixed-label adaptation itself. On CLINC150, the four-encoder ensemble still trails the much smaller TOATOD-small (98.45% vs 98.0222%), and the fusion weights were chosen after repeated test exposure because 'Earlier rounds had already exposed benchmark test summaries, so the overall development history is not a fresh blind evaluation.' Restricting CLINC150 to in-scope classes also sidesteps the benchmark's out-of-scope purpose.
The statistical evidence undermines the headline claim. Under a matched evaluation path, the SAM mean shows a difference of only $+0.0433$ percentage points with a shared-row 95% interval of $[-0.2273, +0.3247]$, which the paper correctly notes does not establish a reliable gain. External leaderboard comparisons are also explicitly caveated: 'These contextual comparisons are not reruns under one shared data, selection, or compute protocol.' This honesty is admirable, but it also means the reported ranks carry little causal weight.
Reproducibility documentation is mixed. The positive side is that 'The continuation protocol includes hashes for the historical model and all six continuation checkpoints, training configuration, precision, and the bootstrap scope,' plus archived evaluation-explorer JSON records. The negative side is that no public training code repository is linked in the text, and 'Historical hardware provenance and a uniformly rerun cross-model benchmark remain incomplete.' Independent reproduction would likely be blocked by missing scripts and the opaque historical provenance of the starting Laya checkpoint.
We present DeepOpen, a decision-Oriented intent-classification framework with dataset- specific systems for BANKING77 and CLINC150. On BANKING77, a single English classi- fier continued with sharpness-aware minimization (SAM) achieves 94.1234% test accuracy (2,899/3,080). This is the best observed result among three fixed SAM seeds, selected for display after test evaluation; their mean is 94.0368%. On the 150 in-scope CLINC150 classes, probability fusion of one Laya specialist and three DeBERTa-v3-large classifiers achieves 98.0222% (4,411/4,500). We describe the two configurations, reproduce the reference com- parisons displayed in the accompanying evaluation explorer, and separate headline scores from evidence about improvement. Under a uniform BANKING77 evaluation path, the SAM mean exceeds the historical checkpoint by 0.0433 percentage points, with a shared-row 95% interval of [-0.2273, +0.3247]. Thus, the continuation does not establish a reliable accuracy gain. The report documents strong observed task-specific performance, selection effects, and unresolved historical execution-path differences, rather than a new backbone or a unified algorithmic advantage.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.