ReDence: A Hybrid Agentic and Decision-Model Framework for Rapid, Evidence-Grounded Detection of Modern Social Engineering Scams
ReDence tackles the problem of detecting modern social engineering scams that go beyond credential phishing by performing adaptive, multimodal investigations of websites and their artifacts. Its core idea is a hybrid architecture that pairs a fine-tuned 4B-parameter agentic Investigator (Qwen3.5-4B) for reasoning and reporting with a specialized 4B-parameter Decider (Tev1) for fast binary and multiple-choice decisions, cutting latency while preserving depth. This matters because SE scams caused billions in losses in 2025 and existing detectors either lack interactive evidence gathering or are too slow to operate at scale.
ReDence delivers a well-motivated hybrid architecture that meaningfully improves over both static classifiers and slower monolithic agents on the authors’ curated SE-scam benchmark. The system achieves $96\%$ accuracy and an $F_1$ score of $0.97$, while cutting median investigation time from $194$ s to $43.8$ s and GPU memory from $33.8$ GB to $3.4$ GB. As the authors note, “ReDence achieves 96% accuracy in detecting scams across twelve SE scam categories, substantially outperforming existing detectors while being at least four times faster than comparable agentic models without compromising investigative depth.” They further report that “ReDence not only matches but significantly exceeds the detection performance of the Mentor model, achieving an F1 score of 0.97 compared to 0.82 for the Mentor.” However, the gains are demonstrated on a manually assembled BBB dataset of modest size, and the Mentor-generated training trajectories may embed the teacher’s own hesitation and redundancy.
The strongest element is the careful separation of responsibilities between the generative Investigator and the compact Decider. The authors show that “68% of the aggregate runtime was spent on decision-making tasks” in the Mentor model, and by offloading those choices to Tev1 they reduce decision overhead to just $14\%$ of ReDence’s runtime. This yields a fourfold speedup without sacrificing depth: ReDence preserves the Mentor’s structured evidence ledger and reporting format, achieving “a median Evidence coverage (EC) score of 0.87 and auditable evidence coverage (AEC) of 0.81,” even outperforming the Mentor on held-out categories. The comparative case study on wego-france.com convincingly illustrates how multimodal interaction and dynamic adaptation let ReDence surface scams that text-only, non-interactive agents miss.
A first concern is the dependence on Mentor-generated training data and the Mentor’s own flaws. The authors candidly note that “Among the 97 websites marked as having inconclusive evidence, we identify this phenomenon in 72 cases,” where the Mentor repeated unresolved questions and issued redundant tool calls. Distilling these trajectories into a smaller model risks propagating similar non-termination patterns, even if the Decider is meant to curb them. A second concern is evaluation reliability: human assessments of report quality currently rely on a single coder, with the authors noting that “We are currently repeating this analysis with two independent coders to measure intercoder agreement. The evaluation described below uses one coder.” Additionally, the dataset is small ($728$ malicious ground-truth sites) and heavily imbalanced, and the held-out test categories comprise only $308$ websites across five types, leaving limited statistical power for generalization claims.
The experimental evidence supports the core claims about speed and accuracy, but some comparisons are uneven. ReDence’s margin over URLNet ($F_1=0.25$) is expected because URLNet is a URL-only classifier, making it a weak baseline for content-rich SE scams; the authors acknowledge that “URLNet has the lowest memory consumption (1.5 GB) and shortest runtime (1.2 seconds), but also exhibits the poorest detection performance, with an F1 score of only 0.25.” The comparison with ScamFerret is more informative: the case study shows that “ScamFerret, in contrast, retrieves the website’s text, source code, and hyperlinks but fails to recognize the impersonated retailer” because it neither interprets images nor interacts with dynamic pages. This fairly highlights ScamFerret’s narrower modality. That said, ScamFerret is evaluated with GPT-4.1 and Llama-3.3-70B, not necessarily optimized or fine-tuned for the authors’ tools, so the gap may partly reflect prompt and tool mismatch rather than pure architectural superiority. The internal Mentor comparison is clean but raises an information-leakage issue because the same Qwen3.5 family generated the training trajectories.
Reproducibility is partially strong but has gaps. On the plus side, the authors document fine-tuning hyperparameters in detail: “We independently fine-tune both models using QLoRA with frozen 4-bit backbones and trainable LoRA adapters ... Investigator uses a maximum sequence length of 16,384 tokens, LoRA rank 16, α = 32, and three training epochs at a learning rate of 1×10−4. Decider uses 2,048 tokens, rank 8, α = 16, and one epoch at 5×10−5.” They also archive websites with Browsertrix and report “a median damage rating of 0.08 (IQR 0.03-0.15),” which supports future re-evaluation. On the minus side, although the abstract labels ReDence as open-source, no repository URL or checkpoint release appears anywhere in the text. Full reproduction therefore requires rebuilding the Mentor trajectories from scratch, which demands access to the same multimodal teacher (Qwen3.5-27B), a Playwright browser environment with a CAPTCHA solver, and the proprietary Tev1 model, none of which may be freely or easily available.
Detecting online social engineering (SE) scams often requires examining website workflows, verifying claims against external sources, and connecting evidence across multiple pages. Existing detectors provide limited support for these investigations, leaving analysts to manually gather and validate evidence, which inevitably delays detection and subsequent takedown of these attacks. To address this gap, we present ReDence, an open-source hybrid framework for adaptive, evidence-grounded detection across diverse SE scam categories. ReDence combines two locally hosted models with complementary responsibilities: a fine-tuned multimodal agentic Investigator (Qwen3.5-4B) that performs investigative planning, evidence interpretation, and reporting, and a specialized Decider (Tev1) that handles binary and multiple-choice tasks, including browser-action selection, evidence assessment, and decisions about whether to continue an investigation. By delegating discrete decisions to a specialized model, ReDence reduces reliance on lengthy language-model responses, while preserving the reasoning capabilities required for complex investigations. Our evaluation demonstrates that ReDence achieves 96\% accuracy in detecting scams across twelve SE scam categories, substantially outperforming existing detectors while being at least four times faster than comparable agentic models without compromising investigative depth. ReDence also produces structured, evidence-grounded reports that document investigative actions, supporting and contradictory evidence, and collected artifacts, which allow analysts to independently audit detection decisions and utilize the collected evidence to support subsequent mitigation and takedown efforts.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.