Can Loan Processors and Underwriters Spot AI-Edited Statements? An Empirical Study
AI coding agents can now edit native PDF financial documents while preserving issuer layouts, making identity-swapped forgeries harder to spot than image-level fakes. This paper tests whether experienced loan underwriters and processors detect such edits in a first-pass document-only review. The authors gave three hired reviewers 132 agent-edited forged PDFs derived from 47 genuine documents and checked whether their concerns actually matched the identity-altering edits. The core finding that only about one in ten judgements contained an edit-supported concern is a clear warning about relying on visual human review for financial fraud detection.
This exploratory pilot delivers a valuable and alarming directional signal, but its severe limitations prevent any population inference. The authors correctly frame the ~10.6% supported-concern rate as descriptive of this setting rather than a benchmark of underwriting skill. The threat model is real: prior work with the same team establishes that off-the-shelf coding agents can reliably alter real financial PDFs at cents per forgery. What matters here is that even experienced reviewers, given known inconsistencies to find, mostly failed to flag the source of the problem.
The paper is refreshingly transparent about its scope and avoids overclaiming. It directly tests the practically important setting of native-PDF identity swapping rather than image-model generation, and it carefully separates edit-supported concerns from generic suspicion by aligning every reviewer note with the genuine original. The use of real issuer-downloaded documents from twelve sources, not fabricated templates, grounds the study in an authentic lending workflow.
The study is drastically underpowered: three reviewers examined different packages, so reviewer effects are hopelessly confounded with package content, and the absence of genuine-file controls precludes any specificity or accuracy estimate. Reviewers were paid only US$25–30 for up to three hours of skilled financial review, an implausibly low rate that likely compresses effort and attention; one reviewer additionally submitted a post-hoc correction that flipped eighteen Suspicious labels to Clear, raising quality concerns. The assessment of whether a concern was truly 'edit-supported' was performed solely by the first author, and every author is an employee of Scam.ai, a document-fraud detection vendor with an obvious commercial interest in the result.
The evidence supports the narrow descriptive claim—roughly 10% of judgements cited an identifier inconsistency introduced by the forgery pipeline—but it does not show that reviewers are blind to all AI forgeries or that stronger workflows would fail. All successful detections relied on internal identifier mismatches that the authors knowingly retained, meaning a more polished pipeline that replaced identifiers consistently would have faced even weaker scrutiny. The authors fairly situate their human-review outcome against prior automated-detection and image-forgery benchmarks without overstating comparability. Notably, no reviewer caught any edit via a visual anomaly, underscoring the limitation of appearance-based review.
Independent reproduction is essentially blocked. The authors state that the original records, forged PDFs and raw assessment spreadsheets will not be shared, and they do not identify the specific AI agent, model, prompts or hyperparameters used in their in-house editing tool. Review time and software aids were unrecorded, and the post-hoc label correction is only partially auditable from the provided description. Without open data, code or a fully specified editing pipeline, the study serves as an anecdotal pilot rather than a reproducible experiment.
AI-assisted editing can produce loan documents that preserve familiar issuer layouts while changing the applicant's identity. We investigated whether experienced underwriters and loan processors identify such forgeries in first-pass, document-only review. Three practitioners recruited through Upwork each received one applicant package drawn from 134 forged PDFs derived from 47 genuine financial documents supplied with full consent. The edits preserved amounts, dates and balances; identifier inconsistencies introduced by the pipeline were knowingly retained before distribution. Reviewers were not told that files had been altered and returned 132 file-level judgements. Checking their explanations against the originals yielded 14 edit-supported concerns (10.6%): 12 Suspicious judgements and two Insufficient information judgements. All supported concerns involved identifier disagreements within or across documents, and no reported visual cue identified an actual edit. Thus, approximately one in ten judgements contained evidence of the forgery, despite known inconsistencies remaining in the files. These observations motivate evaluating consistency checks and independent source verification alongside human review. With three reviewer-package pairs and no genuine-file controls, the findings describe this exploratory setting rather than population-wide detection accuracy.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.