PrismaCoder: Pushing the Boundaries of Open Multimodal Code Intelligence through Decomposed Rubric Synthesis
Multimodal code intelligence demands programs that are correct, well-structured, and visually faithful, yet existing training signals are either too coarse or confined to single tasks. This paper introduces PrismaForge, which synthesizes per-instruction evaluation rubrics that separately assess code-level and visual-level criteria, then uses them to filter training data and reward reinforcement learning. The resulting dataset, PrismaCoder-2M, and the trained 9B and 35B models advance the open state of the art and compete with proprietary systems on charts, webpages, and scientific graphics.
The paper makes a strong case that decomposed rubric synthesis meaningfully improves multimodal code data quality and model performance. PrismaForge is principled, the dataset coverage is broad, and the released models lead open baselines on several benchmarks. At the same time, the pipeline is expensive to reproduce, relies on multiple proprietary API endpoints, and shows lingering weaknesses in specialized formats.
The diagnostic value of splitting code and vision rubrics is clearly demonstrated: the vision rubric catches a misplaced footer that every code check and a holistic judge let through, scoring 0.18 against a holistic 78/100. The pipeline also generalizes across backbones—Gemma-4-31B and Qwen3-VL-30B improve on all twelve metrics after SFT and RL—and rubric-based filtering consistently outperforms direct-judge selection, with gains as large as 24 points on Vega-Lite.
The largest practical barrier is reproducibility cost: the pipeline relies on Claude-Opus-4.6 for curation, Gemini-3-Pro for rubric generation, and Gemini-3-Flash for judging, with HTML evaluation costing roughly $0.04 per sample, and the authors note that resource constraints prevented a full GRPO comparison against direct scoring. At 100 samples, the human agreement study is also quite small for a dataset of two million.
Despite strong aggregate results, the models struggle on specialized notation languages such as LilyPond, LaTeX, and Asymptote, where error rates remain high. Benchmark-specific anomalies—such as TreeBLEU lagging behind weaker open baselines despite superior visual metrics—suggest that surface code similarity and visual fidelity are still in tension.
The empirical comparison is broad and generally favorable. PrismaCoder leads open baselines on ChartMimic, WebCode2M, DesignBench, ArtifactsBench, and VisPlotBench, and the authors explicitly benchmark across judge families to mitigate evaluator bias. That said, proprietary systems still win on several tasks—e.g., DesignBench editing and many Gemini 3 metrics—so the claim of rivaling proprietary systems is only partial. The rubric-based curation does beat direct-judge filtering on every benchmark tested.
The authors release the dataset, model weights, and code, which is commendable. Reproduction is nonetheless blocked by significant practical barriers: 32 H200 GPUs for training, reliance on multiple proprietary API endpoints throughout curation and judging, and per-sample costs that scale to tens of thousands of dollars for full dataset construction. Hyperparameters are documented in Appendix F, but the full suite of rubric prompts and judge configurations contains many hand-tuned details, and the paper explicitly states that resource limits prevented completing the full direct-score versus rubric RL ablation.
Multimodal code intelligence connects the symbolic world of programs with the visual world their execution creates. Interest in this capability is burgeoning, as models are increasingly asked to produce charts, webpages, animations, and semi-structured documents. Recently, proprietary models have pushed far ahead, producing interactive artifacts even from a text description or a screenshot alone. Yet their data and training remain closed, and open progress is held back by the quantity and quality of available data and by the pipelines and methods for producing it. Quality is the harder of the two, since a sample must be correct as a program, well built, and above all visually right once rendered. No existing signal covers all three. Execution often returns just a pass or fail, code similarity does not map to the visual output, and task-specific verifiers do not generalize. To push open progress further, we start from the data side with PrismaForge, a hierarchical data pipeline centered on a synthesized rubric that decomposes these factors into checkable criteria. The same rubric serves as both filter and reward: it curates the supervised corpus, then drives reinforcement learning on visual-coding tasks. Leveraging PrismaForge for data synthesis and filtering, we build PrismaCoder-2M, the most diverse open dataset for multimodal code intelligence and the largest to date. On the model side, we train PrismaCoder-9B and PrismaCoder-35B on this data and release both. They lead the open state of the art by a wide margin and rival leading proprietary systems. Our analyses further show that the proposed rubric-based pipeline improves multimodal coding performance across model backbones. We release our code and data at https://qiushisun.github.io/PrismaCoder-Home/.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.