ixi-sLM: On-Device sLM for AI Call Agent Service
ixi-sLM tackles the problem of running an AI call agent entirely on a mobile device under constraints that cloud deployments never face: roughly 1 GiB of memory, in-call thermal limits, static-integer NPU execution, Korean STT input riddled with disfluencies, and user expectations set by cloud LLMs. The core idea is to take a 1.2B-parameter Korean sLM and apply two service-specific optimizations—domain-adaptive vocabulary trimming and tokenizer merging to shrink footprint and token counts, followed by task-aware mixed-precision quantization calibrated on real calls—to squeeze the model onto Qualcomm and Apple NPUs. It matters because the authors claim a live production deployment serving several million subscribers at 94% of an unnamed cloud LLM's quality, cutting per-call energy to 23% of a CPU baseline.
The paper is a credible systems contribution anchored in a real production deployment, but it is strongest on engineering narrative and weakest on externally verifiable evidence. Vocabulary trimming, domain-aware tokenizer merges, and activation-sensitivity-driven mixed quantization are sound, well-ablated design choices that directly address the stated memory and NPU constraints. However, headline quality claims rely on an undisclosed cloud baseline, a 100-call evaluation set, and LLM judges with potential self-preference bias. Moreover, the aggregate quality metric hides severe degradation in conditional sub-tasks such as task suggestion.
Domain-adaptive modeling is the clearest win. Trimming the vocabulary from 102,400 to 65,536 using real call-traffic frequency removes dead tokens and shrinks the tied embedding and LM head from 419 MiB to 268 MiB, yielding a 15.8% total model reduction; domain-aware merges compound this by cutting call-traffic token counts 18%. The real-call calibration ablation is convincing: under the MW4A16 mixed-precision setting, calibration on calls achieves 82.23 ROUGE-L versus 79.47 on WikiText. Shipping one model on both Qualcomm and Apple NPUs with a single quantization recipe is also a genuine engineering accomplishment.
First, the cloud baseline is an anonymous frontier commercial LLM, making the headline 94% quality claim impossible to verify or contextualize. Second, the aggregate task-accuracy metric masks severe conditional-sub-task degradation under quantization: Appendix E shows schedule suggestion drops from 99.55 to 87.05, location from 97.61 to 83.95, and contact from 0.00 to 33.33. The authors argue these are acceptable because the user confirms actions, but the summary tables do not reflect this caveat. Third, the evaluation is limited to 100 held-out calls judged by LLMs, one of which shares the baseline model family; Appendix J admits this risks inflating the baseline. Finally, the reported failure of generic frameworks is vague—ExecuTorch NPU outputs were unusable with no error analysis or reproducer.
Internal ablations are solid, but external comparisons are thin. Tokenizer and quantization gains are measured against the authors' own FP16 checkpoint rather than against other on-device Korean baselines, and the framework comparison mixes different model sizes and configurations. The most defensible evidence is the on-device efficiency profile: the NPU cuts latency to 22% and per-call energy to 23% versus the same model on CPU, with prefill energy dropping roughly 17x. The production-scale claim of several million subscribers is offered without A/B or user-satisfaction metrics, so it signals deployment feasibility rather than validated quality superiority.
Reproduction is essentially impossible for outsiders. The de-identified call corpora, the domain-trained checkpoint, and the in-house Opt.Studio toolchain are not released. Hyperparameters for continued pretraining are described as standard domain-adaptive practice with no learning rates, steps, or compute budget disclosed. Exact per-module bit-width assignments are plotted but not tabulated, and the calibration call set and evaluation transcripts are private. Backend compiler versions and full operator mappings are omitted, leaving independent researchers to reverse-engineer both the data distribution and the export pipeline.
Running an AI call agent on-device makes it private, network-independent, and free of per-query cloud cost, but shipping one as a real-world service imposes constraints a cloud deployment never faces: ~1GiB memory budget, in-call latency and thermal limits, static-integer NPU execution, Korean STT input far from sLM pretraining, and cloud-level quality expectations. We bring ixi-sLM, a 1.2B Korean sLM, to mobile NPUs through ON-DEVICE MODELING: domain-adaptive modeling trims the vocabulary by call frequency to shrink the embedding and LM head that take nearly half the model under INT4 transformer weights and refits the tokenizer to call speech (18% fewer tokens on call traffic), while task-aware quantization calibrates on real calls and assigns per-module mixed precision by activation sensitivity. A single domain-adapted model runs ixi-O's two production services, AI Call Summary and AI Call Answering, at 94% of a state-of-the-art cloud LLM's task-weighted quality; our on-device modeling carries that quality onto the NPU with only a marginal drop in overall score, in an 803MiB model (28% of FP16) at 23% of the CPU's per-call energy. The same recipe deploys across Qualcomm and Apple NPUs and serves several million ixi-O subscribers in production.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.