JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs

The empirical quantification of switching taxes is sharp and anchors the design. At full host-to-device bandwidth, moving a 7B FP16 expert still idles the GPU for 0.63 s, while re-prefilling a 27.5K-token KV context after eviction costs 2.64 s versus only 0.05 s to reload the same state from host memory—an order-of-magnitude spread that justifies the dual-state tier. The ablation is decisive: removing partial residency raises SCT by 18.0%, removing the host KV tier by 53.8%, and removing adaptive control by 24.3%, and the crash-budget case shows these controls are structurally super-additive. The comparison against INT4 co-residency is also honest: it notes that quantization merely relocates the weight–KV contention and fails structurally when even compressed weights exceed the card budget.

Three issues limit the generality of the claims. First, intent extraction assumes the planner emits a visible, deterministic tool-call header during generation; this is standard for structured function calling but breaks for agents that emit free-form chain-of-thought before naming an expert or that use opaque, non-standard interfaces. Second, the theoretical $(1-\varepsilon)$ guarantee in Theorem A.1 depends on independent candidates with additive values, yet Section 3.4 and Appendix C.2 acknowledge that shared-prefix recovery and mutually exclusive layer-group states require complete legal plans per model or session, with cost scaling in the number of enumerated options; the end-to-end experiments do not validate this grouped DP at the scale implied by hundreds of optional objects. Third, the dramatic crash-regime result—a $46\times$ gap—compares JointServe against whole-model baselines truncated at 35,000 s, which inflates uncertainty because those baselines never actually finish; the true gap could be smaller or larger, but without completion data the comparison is noisy.

The baseline coverage is commendable and spans the design space: vLLM (naive engine restart per switch), SGLang (framework default with NVMe reload), ServerlessLLM (optimized loading stack, tested both officially and in an idealized host-resident variant), INT4-CoResident (compression without swapping), and INT4-Swap (quantized whole-model swap). Evaluation across SWE-bench, $\tau$-bench, OSWorld, and a production-shaped TraceLab replay supports external validity. The stall decomposition precisely attributes gains: weight-switch stall drops to 0.00 s for 7B/14B experts because prefetch moves bytes before the request arrives, while the dual-state KV tier replaces 1.03 s of GPU recompute with 0.05 s of host reload per switch-back. One caveat is that SGLang is reported as $8.3\times$ slower than JointServe even at full HBM capacity ($b=1.0$), which the paper attributes to SGLang's storage-backed reload path; this suggests part of the headline advantage stems from bypassing a slow baseline loader rather than from the residency policy alone.

Independent reproduction would be difficult. The prototype is described as roughly 1.5K lines of Python atop SGLang, yet the text contains no mention of code, data, or artifact availability. The custom shard loader bypasses the engine's stock checkpoint reader and writes weight buffers directly from a pinned host weight bank, implying a non-trivial patch to SGLang's memory management. Hyperparameters such as the startup shard size $k^{*}$, the $\kappa$-floor bandwidth share (15%), and the three rounding budgets $\varepsilon_C, \varepsilon_S, \varepsilon_K$ are reported, but no sensitivity analysis or tuning procedure is described. Testbed details are precise—A800-SXM4-80GB, PCIe 4.0, measured 24.3 GB/s effective host-to-device bandwidth—but without the implementation, a reader cannot reproduce the in-place overwrite hot-swap or the grouped residency DP.