Jev-IDS: System One Models for Network Intrusion Detection
Jev-IDS explores whether a System One Model (SOM)—the typed, probabilistic decision model Jev—can serve as a lightweight few-shot classifier for network intrusion detection. Instead of generating free-form text, Jev answers bounded noul and choice questions that return structured attack probabilities and traffic categories, promising lower latency and cost than autoregressive LLMs under label scarcity. This matters because operational NIDS deployments are constrained by per-flow inference overhead, monetary cost, and false-alarm rates, and conventional supervised detectors fail when labeled attack examples are scarce or entirely unseen.
The paper presents a controlled comparison of Jev, Gemini 3.6 Flash, and a low-data Random Forest on NSL-KDD, but its credibility is undermined by severe internal contradictions. The abstract and conclusion claim that Jev achieves the best overall results, yet the results section explicitly notes that 'Gemini achieved higher overall F1 than Jev at every evaluated value of k' (Section 6.1). Likewise, the conclusion asserts that Jev obtained 'the best overall results among the evaluated approaches' (Section 8), directly contradicting Table 2 and the main text. These discrepancies, together with mismatched baseline names and divergent sample sizes between abstract and body, make the headline claims unreliable.
The matched-protocol design and operational measurements are genuine strengths. The authors hold the flow representation, labeled examples per category $k$, and decision threshold constant across detectors, and they separately track known- versus novel-attack recall using frozen splits. The latency and cost differentials are stark and well-documented: 'Across the evaluated Example budgets, Jev required approximately 0.31–0.34 s per successful request, compared with 1.99–2.65 s for Gemini' (Section 6.4). In addition, McNemar tests show Jev achieves statistically significantly higher novel-attack recall than Gemini at $k=1$ and $k=2$ (Section 6.3).
The paper suffers from multiple serious inconsistencies between its abstract, results, and conclusion. Most critically, the conclusion states that Jev obtained 'the best overall results among the evaluated approaches' (Section 8), directly contradicting Table 2 and Section 6.1, where Gemini records higher $F_1$ at every $k$. The abstract also claims $1.5\times$ higher novel-attack recall than the generative baseline, but Table 2 reports Jev novel recall of 0.747 versus Gemini's 0.713 at $k=1$—a negligible gap of roughly 5%. The abstract further invokes a '300-flow NSL-KDD pilot split' with 'GPT-5.6 Luna,' neither of which matches the 2,000-flow experiment or the Gemini 3.6 Flash baseline used throughout the body. Methodologically, the evaluation rests on a single outdated dataset (NSL-KDD) with only three random seeds, limiting both external validity and statistical power. The fixed 0.5 threshold is acknowledged as suboptimal yet still used for all detectors, and no calibration analysis is performed despite the model returning explicit probabilities.
The head-to-head experimental design is praiseworthy: flow encoding, supervision budget, and threshold are matched across detectors. However, the evidence does not support the paper's framing. Gemini outperforms Jev in overall $F_1$ and known-attack recall at every budget, while Jev's advantage is limited to precision and novel-attack recall at small $k$. The authors correctly report these numbers in Table 2 and Section 6.1, but then mischaracterize them in the abstract and conclusion. The Random Forest comparison is also overstated: at $k=1$ it produces many false alarms because it is trained on only five labeled examples total (one per class), making the '15$\times$ fewer false alarms' claim operationally trivial rather than diagnostically informative. As the authors note, 'The Random Forest illustrates why novel-attack recall should not be considered in isolation' (Section 6.2), yet they do not fully heed this caution in their own summary.
The authors release code, prompts, frozen splits, and per-flow predictions on GitHub (commit 286a310), which satisfies surface-level reproducibility. However, independent reproduction is blocked by the closed, hosted nature of the Jev API (version jev-1.13.0): future API changes, model updates, or provider pricing shifts will alter latency and cost measurements. The paper relies on list-price cost estimates that are already noted to be time-bound, and the small number of seeds (three) means that variance from few-shot sampling is poorly characterized. Reproducing the statistical comparisons also requires access to the exact same hosted model endpoint, which may not persist.
Machine-learning Network Intrusion Detection Systems (IDS) depend on substantial labeled datasets and task-specific training, whereas Large Language Models (LLMs) detection can analyze flow records directly but incurs higher inference cost and latency, with less constrained outputs. This paper presents JEV-IDS, an open experimental general NIDS based on the Jev System One Model (SOM) to detect zero day intrusions Under label scarcity. JEV-IDS serializes one flow per request and asks JEV two questions: a binary attack probability and a finite-choice traffic category. Our results show that, at k=1, JEV was 4.8 times faster and 3.8 times cheaper than GPT-5.6 Luna, with 1.5 times higher novel-attack recall; it also produced 15 times fewer false alarms than a low-data Random Forest. Across 5,400 decisions on a 300-flow NSL-KDD pilot split, JEV achieved F1-Score 0.859, precision 0.941, recall 0.790, and novel-attack recall 0.838. Increasing k to 2 reduced its F1-Score to 0.839.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.