VERA: A Responsible AI Dashboard for Assessing LLM Deployments in Light of the EU AI Act
Regulated institutions must produce technical evidence for LLM deployments under the EU AI Act, yet existing benchmark suites are engineer-facing command-line tools while accountable reviewers are non-technical managers. VERA is an open-source, self-hosted dashboard that closes this workflow gap by letting reviewers select a model, run public benchmarks, and read requirement-level scores—with decomposable evidence and run-level energy consumption—in a browser. Its core contribution is wrapping raw measurement into a review and filing process designed for institutional approval.
VERA is a well-motivated systems paper that translates the COMPL-AI taxonomy into an institutional review workflow, but its empirical evaluation is too small and confounded to support strong causal claims. The dashboard architecture directly addresses three observed gaps—access, opacity, and traceability—and the authors are commendably transparent about the study's limits. Still, the within-subject comparison is fixed-order, imbalanced by attrition, and sampled almost exclusively from AI researchers rather than the compliance audience the tool targets, so the headline improvements in correctness ($p = .016$) and time ($p = .064$) must be read as bounded reading-performance signals, not isolated interface effects.
The strongest parts of the paper are its problem grounding and architectural choices. The authors derive six design requirements (R1–R6) from a year of release reviews at a major European bank, and the resulting system delivers genuinely useful features: versioned requirement-to-benchmark mappings that avoid code changes when weights are disputed, role-filtered views for legal versus technical reviewers, and run-tied exports that bind evidence to a content digest. The energy instrumentation via CodeCarbon is also a concrete extension, turning environmental impact from a written attestation into a recorded run-level quantity.
The user study suffers from three flaws that the authors acknowledge but do not fully offset. First, attrition left eight of ten completers on the same question-set arm, confounding set difficulty with condition; the fixed order (raw always first) further prevents separation of interface effects from residual learning. Second, the participant population is mismatched to the target audience: nine of ten are AI researchers, and the sole non-research completer showed no benefit, so there is no positive evidence for compliance or risk reviewers. Third, the unaided launch task—the only direct test of R1—failed for four of nine participants, with two giving up within seconds, a gap the authors note "no amount of reading improvement compensates for." Finally, some requirement scores rely on fallback heuristics (e.g., robustness at 1.000 flagged as fallback) that a reviewer could misread as native measurements, though the dashboard now propagates fallback flags.
The paper fairly positions VERA against existing suites in Table I, noting that none of the prior tools provide requirement-level views, role-based filtering, run-tied artifacts, or per-run energy. The authors also appropriately caveat the benchmark scores as within-tool comparisons, because the harness uses an internal pass/fail threshold at a fixed sample budget rather than reproducing published model-card accuracies. However, the study baseline—reading raw files in monochrome panes—is an intentionally impoverished control; it does not compare VERA against the closest prior UI work (PromptOps or COMPL-AI's own interface), leaving open whether the gains come from any structured dashboard or specifically from VERA's design. The exact Wilcoxon test for correctness sits at the design's resolution floor ($p = .016$ is the minimum achievable with seven concordant differences), and the Holm-adjusted per-question McNemar tests survive correction only for the band-counting pair.
Reproducibility is partially supported but institutionally constrained. The authors release the dashboard, study instrument, and analysis script on GitHub, and the evaluation runs specify seed 42 and a sampling budget of 150. Yet a full independent reproduction would require substantial infrastructure: a self-hosted model endpoint, the synthetic banking corpus for privacy/copyright/data-adequacy benchmarks, and the Celery/Redis backend. Moreover, the paper explicitly states that running all twelve measurable requirements at the stated budget "was beyond the compute available for this paper," so the published scores cover only five requirements on four models. The within-subject study is administered by the dashboard itself, which aids replication of the instrument, but the tiny, homogeneous, institution-specific participant pool means results may not transfer to other organizational contexts.
Institutions that deploy large language models in regulated settings often need technical evidence of responsible AI practices before approving a deployment. COMPL-AI provides a technical interpretation of the EU AI Act that organises this evidence under six principle groups and eighteen technical requirements, twelve of which current benchmarks can measure. The benchmark suites that produce this evidence are typically operated from a command line, while the staff who approve deployments are non-engineers. We present VERA, an open-source self-hosted dashboard deployed to facilitate responsible AI processes within a large European bank. A reviewer selects a model and runs public benchmarks against it. VERA scores the twelve requirements that current benchmarks can measure and instruments the energy consumed by each evaluation, adding run-level evidence for environmental impact. We run four served open-weight models through the requirement pipeline, and the dashboard displays a score for each measurable requirement. A within-subject study then compares how participants read these benchmark results: from the raw output files in one condition, and through the VERA dashboard in the other. The dashboard reduced the time needed to perform the responsible AI assessment, reducing the time required from 56 seconds using raw files to 30 seconds using our dashboard, on average. Our dashboard scored highly (4.36 out of 5) on usefulness in the survey conducted within the organization. The dashboard and the study material are released publicly to promote replicability and reproducibility in different organisational contexts.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.