LLM-jp-4 Technical Report
LLM-jp-4 is a family of Japanese sovereign language models—an 8B dense Transformer and a 32B-A3B mixture-of-experts—trained from scratch on roughly 12 trillion tokens with an explicit emphasis on transparency, legal reproducibility, and practical deployment. The authors impose strict open-source selection criteria on data, external models, and software, and they document every stage from tokenizer design through pre-training, mid-training, and post-training. The work aims to provide a publicly verifiable reference for sovereign AI development in Japan.
The report is a thorough, principled account of training medium-scale sovereign LLMs with unusually rigorous attention to licensing and procedural transparency. Its main weakness is that the provided text is truncated before Section 7 (Evaluation), so the performance claims in the abstract—such as "substantial improvements over previous LLM-jp models and competitive performance against open-weight models of similar scale"—cannot be verified from the excerpt alone.
The development methodology is the paper's strongest asset. The authors establish five data-release levels (L1–L3 for public models) and commit to OSAID-compatible, verifiable resources: "Specifically, we limit these resources to those that are available to third parties or for which third parties can construct equivalent resources based on publicly available information about their sources, acquisition methods, processing procedures, and other relevant details" (Section 2). The tokenizer is trained from scratch as a single multilingual Unigram model rather than via staged expansion, which the authors note "results in a markedly simpler and more reproducible construction recipe while retaining broad multilingual coverage" (Section 3.1). Complete hyperparameter tables, public training scripts, and ablation studies (e.g., WSD versus cosine scheduling in Section 4.2.2) give independent researchers a clear map of the design space.
First, the empirical evaluation section is entirely absent from the provided text, leaving all performance claims unsupported here. Second, despite strict licensing criteria, the authors rely on gpt-oss-120b—an OpenAI model—for synthetic data generation at nearly every stage: "we reconstructed the MMWPM pipeline using gpt-oss-120b [OpenAI et al. (2025)] in place of Llama 3.1 70B Instruct" (Section 4.1.3), and "we used gpt-oss-120b... to synthesize reasoning processes and final answers" (Section 6.2.2). The paper explicitly lists "OpenAI's proprietary GPT models" as excluded examples in Section 2.2, yet never states gpt-oss-120b's license or justifies its compatibility, creating a conspicuous documentation gap. Third, corpus-mixture decisions rest on fractional factorial designs that alias main effects with interactions, and the authors concede they "assumed that interaction effects would be relatively small" (Section 4.1.4). Finally, mid-training caused temporary degradation on tasks such as GSM8K and XWinograd (Section 5.3), suggesting that the multi-stage curriculum is not uniformly stable.
Without Section 7, no benchmark tables or head-to-head comparisons are visible in the provided text, so the abstract's claim of "competitive performance against open-weight models of similar scale" remains unsubstantiated here. The authors do compare architectural choices to widely adopted families (LLaMA 2, Qwen3, DeepSeek) and cite standard recipes for initialization and routing loss, but the ablation studies are limited to a 400B-token proxy for the dense model. The paper does not systematically compare its final tokenizer against prior LLM-jp tokenizers on downstream tasks, relying instead on per-language compression metrics.
Reproducibility is exceptionally well documented in procedural terms: "The complete training configurations are publicly available for the dense model... and the MoE model" (Section 4.2.1). Nonetheless, practical reproduction faces major barriers: compute (millions of steps on H200 clusters), licensing dependencies on L2 and L3 datasets that cannot be redistributed by LLM-jp—"Redistribution denotes redistribution of the dataset itself by LLM-jp. A check mark indicates that the corresponding use is permitted" (Table 1)—and an implicit dependency on the availability and licensing terms of gpt-oss-120b for synthetic data generation. The release of SFT and DPO datasets under permissible licenses does mitigate some friction for post-training reproduction.
This paper presents LLM-jp-4, a family of Japanese sovereign large language models (LLMs) developed with an emphasis on transparency, reproducibility, and practical usability. LLM-jp-4 comprises an 8B dense model and a 32B-A3B mixture-of-experts model, both trained from scratch on approximately 12 trillion tokens. To support independent verification and reproduction, we establish explicit criteria for selecting training data, external models, and software, and make the development process publicly available. Evaluation results demonstrate substantial improvements over previous LLM-jp models and competitive performance against open-weight models of similar scale. By documenting the development principles, training procedures, and empirical findings of LLM-jp-4, we aim to provide a reference for future sovereign AI development.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.