LLM-Based Policy Compliance Verification of Loan Applications
Corporate loan applications must comply with dense internal lending policies, but manual review is knowledge-intensive and prone to missed exceptions. This internship report develops a two-step LLM pipeline that first extracts machine-readable rules from policy documents and then evaluates real loan applications against them. Tested on eleven non-anonymized corporate loans at Erste Bank, the system detects most known policy exceptions but shows significant limitations in rigorous validation and scalability.
The two-step architecture is conceptually sound, but the evaluation is too limited to support confident deployment claims. While rule extraction demonstrates encouraging stability, the verification stage is assessed only through a small, expert-curated sample of eleven applications with no systematic error quantification. The authors acknowledge that the system produces false positives and cannot replace human judgement, making it best suited as a rough screening aid rather than a reliable compliance arbiter.
The decoupled extraction-verification architecture is pragmatic and well-justified: it allows rule reuse, human review of extracted rules, and easier debugging. The extraction stage is the strongest empirical result, with thirty-three of thirty-seven distinct rules consistently recovered across all five runs, including every rule deemed critical by a domain expert. Data protection measures are also thoughtfully documented, leveraging EU-hosted Databricks infrastructure with contractual guarantees that inputs are not used for model training.
The verification evaluation is systematically weak. The authors explicitly avoided a full audit of true and false positives, substituting rigorous metrics with selected anecdotes reviewed by a single expert. This makes it impossible to verify the headline claim that the system can identify additional compliance issues that had previously been overlooked. The prototype also suffers from a conservative bias that tends to over-flag rather than under-flag, imposing manual follow-up costs without quantified benefit. At just eleven applications, scalability is barely probed; the authors note that overlapping rules produced redundant flags within single applications.
The evidence does not robustly support the central claims. There are no precision, recall, or F1 scores for the verification task, no ablation studies, and no comparison against non-LLM baselines such as keyword matching or smaller models. Comparisons to related work are loosely framed: Baldwin and Ghanavati (2026) is cited for extracting obligations from regulatory documents, yet their actual work concerns AI safety policy text and knowledge-graph question answering, not financial lending. Similarly, Bowen et al. (2025) addresses mortgage racial-disparity mitigation, which is only tangentially related to internal corporate policy verification.
Independent reproduction is impossible. The system processes eleven non-anonymized real loan applications and proprietary Erste Bank policy documents that cannot be shared. The LLM is the proprietary Claude Opus 4.8 accessed through Databricks, and while the agent framework (pydantic-ai v2.38.0) and custom document_processing wrapper are named, the system prompts, tool implementations, and hyperparameters are undisclosed. No code repository, data snapshot, or comprehensive experimental log is provided.
Banks rely on extensive internal policy frameworks to ensure that corporate lending decisions comply with regulatory requirements and internal risk standards. Reviewing loan applications against these policies is a largely manual and knowledge-intensive process, which can lead to overlooked policy violations or exceptions. This report presents a prototype of an LLM-based policy checker that automatically verifies loan applications against internal lending policies. The proposed approach follows a two-step architecture. First, compliance-relevant rules are automatically extracted from policy documents and converted into structured, machine-readable representations. Second, loan applications are evaluated against the extracted rules to identify policy exceptions, non-compliant findings, missing information, and cases requiring manual review. Evaluation results demonstrate that the rule extraction process is highly stable across multiple runs and consistently captures rules identified as critical by a domain expert. When evaluated on 11 real loan applications with known policy exceptions, the system successfully detected the majority of these exceptions and identified additional compliance issues that had previously been overlooked during manual reviews. While the approach produces false positives and cannot fully replace human judgement, the results indicate that LLM-based policy compliance verification can provide valuable support in the loan application verification process by highlighting potential compliance risks.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.