What Comes Next and Why: Interpretable Next-Event Prediction with Neuro-Symbolic Rules
Next-event prediction in sequential data is usually handled by opaque neural networks, while existing symbolic rule methods lack a principled way to form a unified predictive model. This paper introduces Rules, a neuro-symbolic network that learns Temporal Conjunctive Prediction Rules via differentiable temperature annealing, and Runes, which adds a residual transformer to correct rule-based logit predictions. It matters because it offers an explicit accuracy-interpretability trade-off, producing compact, inspectable symbolic rules for safety-sensitive domains while remaining competitive with black-box transformers.
The paper presents a well-motivated and technically clean neuro-symbolic architecture that meaningfully addresses the accuracy-interpretability trade-off in next-event prediction. The two-phase training strategy—annealing Rules to a symbolic state before freezing it and then adding a residual network—is principled and yields compact rule sets on both synthetic and real data. However, the work is held back by a restrictive rule class limited to conjunctions over simple event-time predicates, quadratic $O(L^2)$ scaling in sequence length, and a strong novelty claim that constricts prior art. As the authors acknowledge, temperature annealing and layer sizes introduce hyperparameter sensitivity, and the residual network reintroduces a black-box component whose corrections require auxiliary metrics like symmetric KL to interpret.
The formalization of event-time predicates and their differentiable learning via temperature annealing is a strong conceptual contribution, supported by rigorous convergence arguments in the appendices. The synthetic evaluation convincingly demonstrates that Rules recovers ground-truth antecedents across signal-to-noise regimes, which is rare in neuro-symbolic rule learning. The residual analysis is particularly insightful: the additive residual logits mostly correct errors rather than overturn correct predictions, supporting a simple and practical trust heuristic for interpretability.
The claim that Runes is the first neuro-symbolic approach to perform next-event prediction on categorical sequences overstates novelty; while the authors carefully distinguish methods that require predefined predicates or binary traces, differentiable temporal rule learners sit close to this boundary and the claim risks excluding adaptable prior work. The rule class is also strictly limited: antecedents are pure conjunctions of positive event-time predicates with no support for negation, disjunction, or alternative temporal operators, which substantially restricts expressiveness. Furthermore, the two-phase decoupled training has no ablation against joint end-to-end training, leaving unclear whether freezing Rules is essential for stability or merely a convenient heuristic that sacrifices joint optimality.
The synthetic experiments provide solid evidence for rule recovery using a custom asymmetric similarity metric, and real-world results show Runes ranking second overall against standard baselines. However, the baseline set is uneven: comparing against flattened XGBoost and CatBoost and against PPM-C—which the authors note fails on long-range dependencies—understates the challenge posed by modern neural architectures. The transformer is an appropriate benchmark, but a stronger neuro-symbolic or program-synthesis baseline is missing. On the positive side, comparisons to related tabular neuro-symbolic work are fair, and the qualitative case on Amazon provides a concrete example of temporal structure that Markov models miss.
The authors release code at a public GitHub repository and describe training procedures, temperature schedules, and layer initialization in appendices. Reproduction is feasible but nontrivial: the model depends on many hyperparameters (annealing schedules, layer sizes, lookback windows), the rule extraction and refinement pipeline involves pruning thresholds, and the quadratic complexity in sequence length requires careful implementation. Real-world dataset preprocessing details are relegated to Appendix G, and the paper explicitly notes that extracted rule sets can vary across runs, indicating that training stability and seed choice may block exact reproduction.
In real-world applications, being able to explain why a prediction is made is often as important - if not more so - than achieving high accuracy. However, existing methods for sequential event prediction typically prioritize accuracy at the expense of interpretability. Neuro-symbolic approaches offer a promising direction to improve interpretability by learning symbolic rules, but extending them to next-event prediction is challenging due to the presence of temporal dependencies. We introduce RULES, a neuro-symbolic model that learns sequential rules directly from data, and combine it with a residual network to form RUNES (Rule Network for Event Sequences). RULES accurately recovers ground truth rules from synthetic data and discovers compact, interpretable rule sets on real data. RUNES combines these rules additively with a residual network, achieving competitive accuracy while providing transparent access to active rules and how residuals adjust predictions.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.