Boundary-Aware Instance Segmentation in Microscopy Imaging

cs.CV Thomas Mendelson, Joshua Francois, Galit Lahav, Tammy Riklin-Raviv · Mar 22, 2026
Local to this browser
What it does
Instance segmentation in dense microscopy images requires separating tightly packed, touching cells—a task where binary masks and pixel-wise losses often merge adjacent instances. This paper proposes predicting continuous signed distance...
Why it matters
This paper proposes predicting continuous signed distance functions (SDFs) mapped to probabilistic segmentations via a learnable sigmoid, trained with a differentiable Modified Hausdorff Distance (MHD) loss. The approach eliminates the...
Main concern
The paper presents a technically sound approach that combines SDF prediction with a geometry-driven MHD loss, yielding competitive segmentation accuracy on public and private benchmarks. The ablation study validates the necessity of...
Community signal
0
0 up · 0 down
Sign in to vote with arrows
AI Review AI reviewed
Plain-language introduction

Instance segmentation in dense microscopy images requires separating tightly packed, touching cells—a task where binary masks and pixel-wise losses often merge adjacent instances. This paper proposes predicting continuous signed distance functions (SDFs) mapped to probabilistic segmentations via a learnable sigmoid, trained with a differentiable Modified Hausdorff Distance (MHD) loss. The approach eliminates the need for interactive prompting or watershed post-processing while aiming to improve boundary fidelity in high-throughput cellular imaging.

Critical review
Verdict
Bottom line

The paper presents a technically sound approach that combines SDF prediction with a geometry-driven MHD loss, yielding competitive segmentation accuracy on public and private benchmarks. The ablation study validates the necessity of bidirectional boundary constraints, and the method demonstrates practical utility for prompt-free microscopy analysis. However, the architectural contribution is incremental—relying on a standard U-Net backbone—and claims of universal superiority over foundation models rely on specific, arguably suboptimal evaluation protocols.

“Across all datasets, our approach was either the best or second-best performer when compared to Cellpose, SAM, and MicroSAM”
Mendelson et al. · Section 1, Contributions
What holds up

The SDF-based representation provides smooth contour modeling that naturally handles ambiguous boundaries better than binary classification. The ablation study in Table 1 demonstrates that both directional MHD terms are required for peak performance: using only $\mathcal{L}_{\text{LMHD}}$ yields 0.882 on GOWT1, while adding $\mathcal{L}_{\text{RMHD}}$ increases the score to 0.944. The formulation of the soft boundary map $B_{\text{PRED}}(\mathbf{x}) = \sigma_{\alpha,\beta}(\phi_{\text{PRED}}(\mathbf{x})) \, \sigma_{\alpha,\beta}(-\phi_{\text{PRED}}(\mathbf{x}))$ is differentiable and effectively localizes object contours without heuristic post-processing.

“using both MHD terms ($\mathcal{L}_{\text{LMHD}}$ and $\mathcal{L}_{\text{RMHD}}$) consistently yields the highest SEG scores, confirming their importance for enforcing accurate boundary alignment”
Mendelson et al. · Section 3.3
Main concerns

Three significant limitations undermine the generalizability claims. First, the comparison with SAM uses rigid grid prompting (32×32 and 64×64 densities) rather than interactive point or box prompts, which contradicts SAM's intended use case and likely underestimates its performance on dense scenes. Second, the MCF7 spheroid dataset is private and not publicly available, making the claimed "best or second-best" performance on this data unverifiable and non-reproducible. Third, the loss weights ($\lambda_{\text{LMHD}}=0.9$, $\lambda_{\text{RMHD}}=10^{-1}$, $\lambda_{\text{LSE}}=1$, $\lambda_{\text{CE}}=1$) appear arbitrary and lack sensitivity analysis or ablation, raising questions about tuning robustness across datasets.

“For SAM (ViT_B), p=32 and p=64 indicate prompt-grid densities of 32×32 and 64×64”
Mendelson et al. · Table 2 caption
“The private dataset was provided by the Lahav Lab and consists of 3D irradiated MCF7 human breast cancer spheroids”
Mendelson et al. · Section 3.1
Evidence and comparison

The evidence supports improved instance separation compared to vanilla U-Net and Cellpose, particularly on the dense MSC and MCF7 datasets where boundary ambiguity is highest. However, the SEG metric (mean Jaccard index for matched instances with IoU > 0.5) favors methods that avoid over-segmentation, which the MHD loss explicitly targets. Without statistical significance testing (e.g., paired t-tests or confidence intervals beyond standard deviations), the marginal differences in Table 2—such as 0.944 vs. 0.943 on GOWT1 or 0.892 vs. SAM's 0.917 on HeLa—cannot be confirmed as meaningful. The comparison to $\mu$SAM is further limited by the asterisk noting evaluation on only 20 test images due to "high inference cost."

“The SEG measure... quantifies segmentation accuracy as the mean Jaccard index between each ground-truth object and its best-matching predicted object where a prediction is considered a valid match... only if its intersection is higher than 0.5”
Mendelson et al. · Section 3.2
“$\mu$SAM results ($*$) are computed on 20 test images due to high inference cost”
Mendelson et al. · Table 2 footnote
Reproducibility

The authors provide source code and detailed implementation specifics including optimizer settings (AdamW, $10^{-4}$ learning rate), augmentation strategies, and inference times (0.77 seconds per image), which facilitates reproduction on the public CSB datasets. However, reproduction of the complete results is blocked by the private MCF7 dataset. Additionally, the initialization of learnable sigmoid parameters ($\alpha=4$, $\beta=0$) and the specific loss weighting scheme lack justification or ablation, leaving uncertainty about whether these hyperparameters transfer to new microscopy modalities. Training requires approximately 45 minutes on an NVIDIA V100 GPU, which is reasonable.

“Source code is available at https://github.com/ThomasMendelson/BAISeg.git”
Mendelson et al. · Abstract
“Training was performed on a single NVIDIA V100 GPU taking approximately 45 minutes per dataset including augmentation”
Mendelson et al. · Section 2, Implementation Details
Abstract

Accurate delineation of individual cells in microscopy videos is essential for studying cellular dynamics, yet separating touching or overlapping instances remains a persistent challenge. Although foundation-model for segmentation such as SAM have broadened the accessibility of image segmentation, they still struggle to separate nearby cell instances in dense microscopy scenes without extensive prompting. We propose a prompt-free, boundary-aware instance segmentation framework that predicts signed distance functions (SDFs) instead of binary masks, enabling smooth and geometry-consistent modeling of cell contours. A learned sigmoid mapping converts SDFs into probability maps, yielding sharp boundary localization and robust separation of adjacent instances. Training is guided by a unified Modified Hausdorff Distance (MHD) loss that integrates region- and boundary-based terms. Evaluations on both public and private high-throughput microscopy datasets demonstrate improved boundary accuracy and instance-level performance compared to recent SAM-based and foundation-model approaches. Source code is available at: https://github.com/ThomasMendelson/BAISeg.git

Challenge the Review

Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.

No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.