Evaluating the Applicability of LLM Inference Optimizations on Apple Silicon

The experimental design is a genuine strength. Apple devices spanning three GPU generations are compared against a matched RTX 4090 under aligned batch-one workloads, and the authors reject easy generalizations. The speculative-decoding analysis is particularly careful: every method is shown to lose end-to-end throughput in the principal benchmark, yet the authors also identify configuration-specific positive cases and explain why decode-side acceptance does not imply E2E gains. The Metal kernel taxonomy (TRANSFERS, HW-GAP, RUNTIME, N/A) transforms opaque porting intuition into actionable decision criteria. Quantization results are equally nuanced, showing that the best runtime and format depend on the specific chip rather than simply on bit-width.

Two limitations stand out. First, KV-cache pruning is evaluated without measuring downstream output quality; the authors explicitly exclude this as outside scope, which means the latency gains for SnapKV and ExpectedAttention come with an unquantified accuracy cost that is material to real deployment. Second, the megakernel case study is billed as exploratory, but its scope is so narrow—one LFM2.5-8B-A1B MoE block on one M4 Pro—that its broader interpretive weight should be treated cautiously. A third concern is that some cross-platform speculative-decoding comparisons are confounded by runtime differences (MLX vs PyTorch/vLLM) rather than isolating hardware; the authors acknowledge this, but it weakens the hardware-only attribution. Finally, FastKVzip is reported to slow execution on both platforms without a satisfying causal explanation beyond overhead dominance.

The evidence largely supports the claims. The paper is commendably fair to prior work: it notes that reported 2–6× speculative-decoding gains on server GPUs arise under different models, workloads, and runtimes, and it does not overstate hardware causation where runtime confounds exist. The Marlin characterization—that it targets batched serving on NVIDIA GPUs—is consistent with the original paper, which indeed emphasizes batchsizes up to 16–32. The FlashInfer and MoE porting analyses are backed by kernel-level and end-to-end measurements, and the split-KV reversal on older Apple GPUs is traced to a quantifiable dispatch-and-partials overhead. One weakness is that the positive cases for speculative decoding (Table 2) are drawn from supplementary, uncontrolled configurations, so they serve as existence proofs rather than a ranked ranking of methods.

The paper provides unusually detailed experimental metadata, including exact model checkpoints, quantization group sizes, draft budgets, prompt distributions, and warm-up protocols in Appendix A.1. Metal kernel source appears inline in listings, which helps readers reconstruct the ports. However, there is no explicit statement that the full benchmark harness or ported kernels will be released as open-source, and several comparisons are withheld because of implementation mismatches rather than reported. Hyperparameters for pruning methods (compression ratios, context lengths) are clearly tabulated, and the split-KV routing uses a measured per-architecture table rather than opaque heuristics. Reproducing the end-to-end numbers should be feasible for readers with access to the same MLX versions and hardware, though the lack of version-pinned software artifacts raises a barrier.