Automating Feature Flag Cleanup at Scale with Multi-Agent LLM Systems: An Industrial Experience at DoorDash

The architectural rationale is tightly coupled to real codebase constraints. The three-layer dependency-injected wrapper pattern is clearly explained as semantically opaque to AST-based tools, making the LLM approach necessary rather than merely trendy. The integration with live rollout data via MCP is a critical safety mechanism, and the failure analysis is admirably honest in identifying completeness—not correctness—as the reliability boundary.

The empirical basis is limited. The evaluation covers only 50 DVs with complexity labels assigned post hoc by the authors, raising risks of selection bias and subjective classification. The reported 90% success rate conflates first-shot merges (62%) with cases needing revision (28%), meaning a significant fraction of cleanups still consume engineering attention. Cost figures exclude infrastructure spend on CI compute and test execution, and generalizability is explicitly constrained to Kotlin repositories at DoorDash.

Moreover, the system retains a mandatory human checkpoint for target-value confirmation, so the end-to-end automation claim applies only to Phase 2. For ambiguous rollout states or abandoned experiments, engineer review remains unavoidable, limiting the true throughput gains.

The conceptual comparison to rule-based tools like Piranha is well-argued but empirically one-sided: the authors declare Piranha 'structurally inapplicable' without attempting—or being able to attempt—a quantitative baseline on their codebase. The differentiation from recent industry LLM deployments is more robust; the authors fairly observe that prior reports from Google, Duolingo, and others lack published success rates, cost metrics, or failure analysis. Still, the absence of an ablation study (e.g., orchestrator and removal agent using the same model) makes it difficult to isolate the value of model role specialization.

Independent reproduction is blocked by proprietary dependencies. The paper does not release source code, prompts, or datasets, and the system relies on DoorDash's internal experimentation platform, Jira integration, private repositories, and a custom MCP server implementation. The evaluation is a retrospective analysis of the 50 most recent cleanups rather than a pre-registered experiment, and the authors explicitly caution that LLM non-determinism means re-running could yield different outcomes. While hyperparameters such as model versions (Claude Sonnet/Opus 4.6) and validation thresholds (JaCoCo patch coverage $\geq 95\%$) are reported, the absence of artifacts and the single-company scope make external verification impossible.