Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving
Gandalf tackles SSD wear-out in LLM serving systems that offload ever-growing KV caches beyond GPU and host memory. The core idea is that hybrid attention architectures (full plus linear/sparse) generate KV states with predictably different reuse lifetimes, so preferring SSD placement for long-lived full-attention KV while keeping short-lived linear states in DRAM can slash write volume without sacrificing cache hits. The paper claims average 7.3x lifetime extensions on trace-driven simulation, which matters because production agentic workloads can exhaust enterprise SSDs in months.
The paper identifies a real operational pain point—KV cache offloading can wear out SSDs in under a year—and proposes a plausible, semantics-aware filter to reduce writes. The offline oracle (§3.2) and the sensitivity sweeps (§5.3–5.5) are methodical. However, the paper overstates its maturity: it claims to have moved to a real production system yet presents no production measurements, and the trace-driven simulator elides critical SSD behavior such as garbage-collection-induced write amplification. The result is a promising proof-of-concept rather than a fully validated system.
The key insight that attention type correlates with reuse lifetime is well motivated. Table 1 and Figure 6 show that linear-attention states in multi-round conversations are reused once or never, while full-attention KV entries are reused across all future rounds. The authors distill this into an oracle-based cost model (Equation 4) that weighs expected hits against write and replacement costs, then derive a practical online heap policy with $O(1)$ decision overhead (§3.4). The evaluation across DRAM/SSD capacity ratios and hybrid models of varying linear-to-full ratios (§5.3, §5.5) is a strength.
First, the simulator assumes WAF = 1, explicitly omitting GC, wear leveling, and block copying. The authors argue this is benign because lower write volume still helps, but relative comparisons could be skewed if Gandalf's sparser write pattern also reduces WAF disproportionately. Second, despite claiming a real production deployment, every number in the evaluation comes from trace-driven simulation; no end-to-end latency, throughput, or wear validation on real SSDs under Gandalf is shown. Third, the attention-type heuristic is rigid: it assumes linear states are always ephemeral and full-attention KV always long-lived. Architectural priors can change, and non-append-only workloads (e.g., branching agents) may violate this assumption, yet the policy has no recourse if linear states become hot.
The headline 7.3x average lifetime gain is realizable only under the specific hardware configuration tested (100 GB DRAM, 1 TB SSD, Kimi-K3). The improvement reaches 120x in some 4 TB total-capacity scenarios (§5.3), while on other traces baselines like Marconi last longer at the cost of 2–6x more misses (§5.2). The cost framework in §3.2 is elegant—$\lambda$ captures how many hits a written byte must earn to be worth writing—but the paper never reports the concrete dollar values of $C_w$ or $C_r$ that operators would need to instantiate it. Comparisons to LRU-K, HotPrefix, and Marconi are fair in that each baseline is adapted to the same two-tier hierarchy, yet several baselines target hit-rate optimization rather than endurance, making Gandalf's win on lifetime somewhat expected.
Reproduction would be difficult. No code, simulator, or artifact is released. The traces are public or semi-public (Claude Code WEKA, Aliyun Bailian, AIPerf), but the simulator itself—how it replays them, how chunking and eviction interact, and how the eBPF logger filters events—must be rebuilt from scratch. Crucial hyperparameters such as the GPU recomputation cost $C_r$, SSD wear cost $C_w$, and the target lifetime $T_{\mathrm{target}}$ are not given with units or calibration methodology. The admission of the WAF=1 simplification (§4) means real-device reproduction could yield substantially different absolute lifetimes, though the relative ordering may hold.
KV caching reduces the serving cost of large language models (LLMs), but the growing KV cache footprint is increasingly exceeds GPU and host-memory capacity. Therefore, people use SSD to expand the KV cache storage capacity with high I/O bandwidth. However, in production LLM workload, such as multi-round conversation and LLM agents, the total amount of KV cache writes significantly exceeds SSD capacity, shortening SSD lifetime from years to months. To improve SSD lifetime, a natural solution is to reduce the KV cache writes by gating the KV cache writes via historical access patterns. However, that significantly lowers the KV cache hit rate (and increase the LLM serving cost) since KV caches from different attention types might have identical historical access pattern but different future access patterns. Our insight is that, to reduce KV cache writes without hurting KV cache hit rate, one need to not only consider the access pattern, but also application-level information such as the attention type of KV cache. Leveraging this observation, we propose Gandalf , the first SSD KV cache offloading system with SSD life time awareness. Gandalf prioritize the admission of full-attention KV cache into SSD over the cache generated by other types of attention, which significantly increases SSD lifetime without sacrificing cache hit rate. Our evaluation shows that Gandalf extends the SSD lifetime by 7.3X on average, while having negligible impact on cache hit rate across multiple workload traces and wide range of hardware capacity. Gandalf design can be extended to other flash-based storage, such as High Bandwidth Flash (HBF).
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.