Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
Choosing a parallelism configuration for distributed LLM inference is challenging because the prefill and decode phases impose conflicting demands on time-to-first-token (TTFT) and inter-token latency (ITL). This paper develops a systematic framework that defines parallelism operating regions via TTFT-ITL Pareto frontiers and formulates SLO-aware selection using goodput $G_p(w,T_s)=\lambda_p(w)F_p(T_s|w)$. The core finding is a workload-dependent transition: tensor-parallel (TP) configurations excel for decode-heavy workloads and relaxed latency requirements, while pipeline-parallel (PP) configurations dominate for prefill-heavy workloads, stringent TTFT targets, and long contexts. The work matters because it moves parallelism selection from a static deployment choice to a workload- and SLO-dependent operating decision.
The paper presents a clean conceptual model and supportive empirical analysis on modern hardware. Its decomposition of latency into computation, communication, and scheduling overhead ($T_q^{(p)} = \tau_q^{(p)} + \eta_q^{(p)} + \gamma_q^{(p)}$ in Eq. 3) and the fixed-total-sequence-length experimental design are methodologically sound. However, the conclusions are drawn from a single 14B model on eight GPUs and may not generalize to larger-scale deployments where data or context parallelism become essential. The framework is valuable as a foundational characterization, but readers should treat the absolute configuration recommendations as scale-dependent.
The fixed-$L_{\rm seq}$ design cleanly isolates prefill-decode composition from overall load, making the Pareto transition interpretable. The authors precisely define the goodput objective ($G_p(w,T_s) = \lambda_p(w) F_p(T_s|w)$ in Eq. 12) and provide a clear empirical demonstration that the optimal parallelism shifts from PP-dominant under tight TTFT SLOs to TP-dominant as the SLO relaxes. The operating-region abstraction is a useful conceptual tool for reasoning about dynamic reconfiguration.
The SLO-aware analysis only optimizes for TTFT despite acknowledging broader objectives: "There are various criteria to choose the SLO constraints, such as decoding latency or end-to-end latency. Here, we consider TTFT as a SLO constraint" (Section 3, Footnote 5). This is a significant limitation because PP-dominant configurations suffer worse ITL, so their advantage may vanish under joint TTFT-ITL service-level objectives.
The evaluation also relies on average ITL ($\text{ITL}^{\rm avg}$ in Eq. 5) rather than tail metrics, which are critical for interactive perceived quality. Additionally, the entire study uses a single 14B model on eight H200 GPUs without data or context parallelism, leaving open whether the TP-to-PP transition pattern holds at larger scales where CP is standard for long-context inference.
The paper positions itself as a characterization study rather than a system benchmark, and comparisons to related work are generally fair. However, it does not validate its quantitative observations against prior serving systems such as DistServe or Sarathi-Serve, relying instead on synthetic homogeneous workloads with fixed concurrency and uniform lengths. The qualitative directions align with established behavior for TP and PP, yet the long-context claim that PP-dominant strategies scale most favorably is undercut by the omission of context parallelism, which recent work specifically targets for long-sequence serving and would likely alter the Pareto landscape.
The authors report the model, precision, hardware, and software stack: "we employ Qwen2.5-14B-Instruct with BF16 tensor type on $8\times$ NVIDIA H200 (141GB). All experiments are performed using vLLM v0.15.0 as an inference engine" (Section 2.2). However, no code, benchmark scripts, or measurement data are indicated as available, and many vLLM-internal scheduling defaults are not exposed. Reproduction therefore requires identical high-end hardware and depends on unshared artifacts, making independent verification difficult.
Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-token (TTFT), and decode, which determines inter-token latency (ITL). The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length. Moreover, despite the widespread adoption of distributed inference, there is still limited understanding of how parallelism configurations should be selected under varying workload conditions and service-level objectives (SLO). In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability. We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy. We further formulate SLO-aware parallelism selection using goodput, showing that stringent TTFT requirements favor PP-dominated configurations, whereas relaxed latency constraints progressively shift the optimality toward TP-dominated configurations with higher serving capacity. Finally, we demonstrate that increasing context length fundamentally changes the relative advantages of parallelism configurations, with PP-dominant configurations exhibiting more favorable scalability and SLO attainment in the long-context regime.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.