CellTrack-Bench: Microscopic Cell Tracking Benchmark for Vision-Language Models
Most cell-tracking links are trivial nearest-neighbor associations that bury rare but biologically critical errors in cell divisions and state changes. This paper introduces CellTrack-Bench, a hard-mined benchmark that isolates these difficult local associations from 15 datasets across four imaging modalities and evaluates whether vision-language models can act as cognitive correctors—by linking cells directly, repairing tracker graphs, or inspecting them for human review. It matters because it provides a controlled diagnostic setting to test whether frontier VLMs genuinely reason about biological dynamics or simply exploit spatial proximity.
CellTrack-Bench is a carefully designed diagnostic benchmark with strong experimental hygiene: sequence-disjoint splits, proximity-debiased metrics, and a frozen harness selected on validation data alone. The three-task decomposition is practical and the hard-mining protocol using three diverse trackers is methodologically sound. However, the small hard-test size (244 neighborhoods), a single-harness policy that likely handicaps some models, and severe token-budget truncation for several open-weight models weaken the generality of the leaderboard conclusions. As the authors conclude, "Taken together, these facts suggest that, for now, VLMs are better used as reasoning modules in the cell-tracking pipeline rather than as substitutes for specialized trackers."
The three-task decomposition—link, repair, and inspect—is elegant and directly motivated by human-in-the-loop curation workflows. The hard-mining protocol is empirically justified: the authors show that shared errors across three trackers occur at 0.28% of links, far above the 0.0022% expected under independence, supporting the claim that these are genuinely difficult cases. The metric design is similarly thoughtful; the NN-Residual and Division F1 metrics explicitly diagnose whether models "rely on proximity for matching" rather than performing deeper reasoning, and the Subgraph-LNK score preserves comparability with the Cell Tracking Challenge graph-edit framework.
The decision to freeze harness c1 for all models is a notable limitation. While c1 was optimal for GPT-5.5 on validation, the authors admit that "A post-hoc check carried out with claude-opus-4.8 confirms the advantage of tool-bearing coordinate interfaces over the evidence-supplied interface, even though it does not independently identify c1 as being better than c8," meaning the frozen protocol likely handicaps some frontier models. The benchmark's scope is also restricted to single adjacent-frame neighborhoods, so it cannot assess longer-range temporal reasoning or track persistence across gaps.
Additionally, the hard-test set comprises only 244 neighborhoods; with 22 models evaluated, the effective sample size per model is thin and confidence intervals are not reported on the main leaderboard. Many open-weight models suffered severe truncation—"Qwen3.5-35B-A3B responses are truncated in 87% of problems"—raising questions about whether the fixed 8,192-token budget unfairly penalizes less efficient architectures and distorts the open-weight rankings.
The evidence broadly supports the paper's central argument. Frontier closed-source models exceed the nearest-centroid baseline on graph-level $\mathrm{LNK}_{\mathrm{sub}}$ and NN-resistant metrics, while open-weight models largely fail: the authors note that "Gemma 4 31B... almost matching the specialized trackers, but its LNKsub is 0.484, and its NN-residual and division F1 scores are only 0.149 and 0.103, respectively." The correlation analysis across tasks—linking and repair are tightly coupled ($\rho = 0.85$) while inspection is more independent ($\rho = 0.51$ with linking, $\rho = 0.29$ with repair)—adds useful diagnostic nuance. Comparisons to specialized trackers are explicitly scoped to local association, which is fair, but the paper is careful to note that "low tracker scores do not indicate anything about their full-sequence performance, since the benchmark problems were deliberately chosen to be ones where these trackers fail."
Reproducibility is mixed. The authors release code, stored model outputs, and a pipeline that regenerates the benchmark from source datasets, but they cannot redistribute the benchmark items themselves due to original dataset licenses. The frozen c1 prompt, tool schema, parser, and token budget (8,192 output tokens, 2,048 reserved for the answer) are documented in Appendix A.4, enabling exact replication of the interface. However, replicating the full leaderboard requires API access to nine proprietary models, and the paper does not disclose inference costs or wall-clock times. The high truncation rates for some open-weight models under the shared budget suggest that independent reproduction may yield different rankings if budget allocation is adjusted per model.
Large language models have shown outstanding performance on text-first tasks such as coding, math, and writing. Recent efforts are shifting models' training to include scientific problems that often live in highly specialized domains beyond text, such as protein structure and biological images. However, in these scenarios, their general-purpose accuracy compared with specialized algorithms remains in question. One such domain is cell tracking, which requires spatiotemporal awareness of cell dynamics. To evaluate this, we created a new benchmark, CellTrack-Bench, which assesses Vision-Language Models' (VLMs) cell-tracking-related performance against specialized trackers. The full benchmark draws from 15 standard cell-tracking datasets across four imaging modalities and comprises 1,537 problems corresponding to 9,041 cell-linking decisions. We selected a hard subset requiring greater reasoning effort by identifying failures shared across distinct algorithmic trackers. With this proposed dataset, we: (i) evaluated different environment harnesses for VLMs; (ii) benchmarked 9 frontier closed-source models and 13 open-weight models; and (iii) provided a detailed analysis of their failure modes. Code, the stored model outputs, and a pipeline that regenerates the benchmark from the source datasets are available at https://github.com/royerlab/celltrack-bench.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.