A foundation for systematic analysis of transformers and RNNs for tractography
Iterative tractography must reconcile noisy local diffusion signals with globally plausible fiber pathways, yet standard deep learning models optimize only a local next-step loss. This paper tackles that gap by systematically evaluating RNN and Transformer architectures and introducing a generation-validation (GV) phase that supervises training at the streamline level via connectivity scores. It reports top scores on the ISMRM 2015 phantom and provides open-source models, while also arguing that current phantoms and in vivo evaluation tools are too limited for definitive benchmarking.
The paper is a valuable empirical engineering study: it rigorously trains and compares sequence models for tractography, proposes a practical GV training protocol, and honestly confronts the mismatch between local loss and global tractogram quality. Still, its headline quantitative claim rests on a single phantom subject used for training, validation, and testing with privileged ground-truth masks, undercutting assertions of generalization. In vivo validation is explicitly exploratory and qualitative, and the failure to recover unseen bundles suggests the models memorize training structure rather than learn flexible anatomical priors.
The generation-validation phase is a genuinely useful contribution: by re-tracking partial validation streamlines and scoring endpoint connectivity, the authors enable efficient epoch-level hyperparameter selection without costly whole-brain tracking at each step. Their open release of code (dwi_ml) and trained checkpoints meaningfully supports reproducibility, and the 177-model ablation yields clear, actionable takeaways—most notably that adding an end-of-sequence token and tuning step size matter more than many architectural details.
The ISMRM experiments rely on ground-truth white-matter and interface masks for tracking and seeding, a setup unavailable in real data that likely inflates scores relative to challenge participants who lacked such information. Because the same single phantom subject is used for train, validation, and test, the results show memorization and hyperparameter sensitivity rather than generalization. Experiment 2 further reveals that models fail to reconstruct bundles absent from training, indicating they do not learn robust anatomical rules but rather reproduce the training set. Finally, the in vivo analysis is purely qualitative, and the authors rightly note that current segmentation methods are too imprecise for meaningful quantitative comparison.
Internal evidence supports the claim that the GV connectivity false-positive rate tracks final tractogram quality better than the local loss does. However, external comparisons to prior work are confounded by asymmetric tracking parameters: the authors benefited from GT-derived masks that earlier teams did not have, and they acknowledge that tracking masks and seed counts matter as much as model architecture. The attention visualizations for Transformers are anecdotal and do not explain why Transformers underperform RNNs here, contradicting broader sequence-modeling trends. In vivo claims rest solely on visual inspection, which the authors admit is the only currently reliable method, meaning no quantitative evidence supports real-brain generalization.
Reproducibility is comparatively strong: the dwi_ml library, model checkpoints, and learned parameters are openly released on Zenodo, and the public ISMRM and Tractoinferno datasets are used. Exact final hyperparameters are tabulated, though the full 177-configuration search space is relegated to supplemental material. The main practical barriers are computational cost—training requires multi-day runs on V100 or H100 GPU clusters—and the reliance on non-standard GT-derived seeding and tracking masks for the ISMRM benchmark, which complicates exact independent replication under typical pipeline conditions.
Machine learning (ML) has emerged as a promising approach for improving diffusion MRI (dMRI) tractography, a task that remains limited by the intrinsic tension between local diffusion information and global anatomical plausibility. In this work, we systematically evaluate recurrent neural networks (RNNs) and Transformer models for iterative tractography, with particular attention to training strategies, input representations (including convolutional neural network (CNN)-based embeddings and end-of-sequence (EOS) tokens), and hyperparameter selection. We introduce a generation-validation phase enabling supervision at the streamline level during training, allowing supervision despite the mismatch between local loss functions and global streamline quality. Using the ISMRM2015 tractography challenge dataset, our models achieve the highest reported performance to date. Through controlled experiments, we quantify the impact of missing bundles, noisy or imperfect training streamlines, and invalid fibers in the training set. Finally, we demonstrate the applicability of our best-performing models for in vivo data from the Tractoinferno database. Overall, our results highlight both the potential and the limits of sequence-based deep learning models such as Transformers and RNNs for tractography, and emphasize the need for improved phantoms and evaluation methods for in vivo validation. We provide takeaways and recommendations for future researchers training and validating sequence-based supervised methods for tractography.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.