Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training
Configuring distributed LLM training over hundreds of GPUs involves a combinatorial explosion over tensor, pipeline, expert, context, and data parallelism plus micro-batch size, recomputation, and sharding, where a single byte of activation accounting can separate a legal run from an OOM crash. This paper proposes Primus Projection, a hybrid engine that benchmarks compute kernels on as few as one GPU and analytically reconstructs out-of-scope communication and pipeline schedules at the target scale, coupled with an LLM-driven tuning agent that searches the space in minutes instead of days. The authors validate across dense and MoE models on two AMD GPU generations, reporting projection errors around $5\%$ at up to 1,024 GPUs and a $+27\%$ throughput gain discovered in under 30 minutes of single-node exploration.
The paper makes a strong practical contribution to a first-class engineering pain point. The hybrid benchmark-and-restore methodology is well-reasoned, the evaluation is unusually broad for a systems paper (spanning dense Llama, MoE Mixtral, DeepSeek V3, and Qwen3 on up to 1,024 AMD GPUs), and the rank-preservation argument rigorously justifies using a slightly biased oracle for configuration search. However, the work is empirically confined to the AMD/ROCm stack with no cross-vendor validation, the LLM agent lacks comparisons against mature constrained-optimization baselines, and the manuscript promises an open-source release without providing a repository URL, which weakens reproducibility.
The central insight—benchmarking what fits on a single node and restoring the rest analytically—directly addresses the gaps of pure roofline models and trace-driven simulators. The evaluation is refreshingly honest: signed errors are reported, the sub-node sweep in Figure 2 places all 16 cells within $\sim 10\%$ of measured throughput, and the large-scale Table 7 corroborates fidelity at 1,024 GPUs for a model never seen during development. The rank-preservation experiments (Spearman $\rho \ge 0.93$, $0\%$ selection regret) rigorously validate the key assumption that the tuning agent only needs correct relative ordering, not absolute throughput. The systematic ablation in Table 11 also isolates the benchmark oracle (+36\%) from the agent framework (+10\%).
The biggest empirical gap is the complete absence of cross-vendor validation. Every number in the paper comes from AMD Instinct MI300X/MI325X/MI355X GPUs running ROCm, and while the authors argue that nothing in the modeling is vendor-specific, they also concede that they have not yet empirically validated on NVIDIA or TPU. For a method whose central selling point is eliminating multi-node trial-and-error, this limits generalizability.
The LLM tuning agent is compared against a hand-rolled Galvatron-style exhaustive search and a raw Claude Code loop, but not against mature constrained-optimization baselines such as Bayesian Optimization with feasibility handling or standard evolutionary search; the paper dismisses BO because it assumes a stationary objective, yet never tests this claim. Moreover, while Appendix J says the framework is released under an open-source license, no repository link, artifact appendix, or hardware-profile YAML examples are provided in the manuscript, blocking independent reproduction. Finally, the pure simulation mode and unmodeled network contention are acknowledged limitations that remain largely unsubstantiated by quantitative data.
The evidence supports most core claims. The sub-node sweep in Figure 2 and Tables 1 and 3 shows all 16 cells within $\sim 10\%$ of measured throughput, and Table 7 corroborates fidelity at 256 and 1,024 GPUs. The head-to-head with Calculon (Table 24) is fair in that both tools see the same Llama-70B workload, but it is also unsurprising because Calculon is a pure hardware-spec model with no ROCm path or MoE support; its $28.9\%$ error on AMD hardware is expected rather than a decisive win. The Galvatron-style baseline is handicapped by using the simulate oracle, and the authors correctly note that the gap is the oracle, not the search, which weakens the claim that the LLM agent itself is necessary. The rank-preservation analysis is the strongest evidence, directly justifying the tuning use-case.
Reproducibility is the paper's Achilles' heel. The manuscript mentions bundled hardware profiles and an open-source license in Appendix J, but it never provides a code repository URL, an artifact appendix, or a concrete YAML example. The sub-node benchmark harness, the DSPy agent loop, and the exact ROCm/hipBLASLt versions are all omitted from the main text. The LLM agent relies on closed Anthropic models (Claude Sonnet/Opus 4.6) accessed through an unspecified LiteLLM gateway; while the prompts are reproduced verbatim in Appendix H, the exact DSPy signatures and tool bindings are not detailed enough to replicate without the code. Without publicly available artifacts, an independent group would be blocked from reproducing even the single-node Llama projection.
Configuring distributed LLM training across hundreds of GPUs requires jointly optimizing tensor, pipeline, expert, context, and data parallelism alongside micro-batch size, recomputation, sharding, and overlap, a combinatorial space where memory legality can hinge on a single byte of activation accounting. Today, teams navigate it through expensive multi-node trial and error, burning GPU-hours on crashes and suboptimal recipes. We present Primus Projection, which closes this loop with two coupled components. First, a hybrid projection engine that measures what it can and simulates what it cannot: GPU microbenchmarks on as few as a single device capture compute kernels and any in-scope communication; a sub-node downscale-measure-upscale workflow analytically restores out-of-scope collectives, FSDP overlap, and the full pipeline schedule at the multi-node target. A pure simulation mode runs entirely on CPU, enabling pre-silicon planning without any GPU. Second, the Primus Tuning Agent treats the projection engine as a fast scoring oracle (seconds per candidate instead of tens of minutes) and uses it to drive an intelligent search over the configuration space. A deterministic seed planner sweeps single axes; an LLM investigation loop proposes cross-axis combinations the planner cannot reach, with an analytic memory pre-filter that rejects infeasible candidates before any GPU time is spent. Validated on Llama 3.1 (dense) and Mixtral 8x22B (MoE) across two GPU generations, all projections stay within ~10% of measured multi-node throughput, and fidelity holds at production scale: 4.7% error on DeepSeek V3 671B at 1,024 GPUs and 1.9% on Qwen3-235B-A22B at 256 GPUs. In a Mixtral 8x22B case study, the Primus Tuning Agent discovers a configuration delivering +27% throughput over the published 4-node BF16 reference at the same scale, in under 30 minutes of single-node exploration.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.