# Contract extraction benchmark on 102 contracts finds full-document processing beats RAG

> An implementation benchmarked contract extraction on 102 held-out contracts, comparing full-document processing with RAG and examining MCP tool design.

- Status: reported (One outlet, not independently verified yet.)
- Topic: AI in Practice
- Published: 2026-10-29T09:28:11+00:00
- Updated: 2026-10-29T09:28:11+00:00
- Web page: https://ai-news.ikangai.com/story/contract-extraction-benchmark-on-102-contracts-finds-full-document-processing-be/
- Seen in: Demetrios from the AAIF
- Change: research (Research)
- Applies to: Contract Understanding Atticus Dataset (CUAD)

## What changed

In a contract extraction benchmark, full-document processing beat retrieval-augmented generation, with retrieval cutting cost by about 11 percent while losing roughly 3.4 to 4.6 F1 points.

Sending whole contracts to a model beat retrieval-augmented generation (RAG) in a document extraction benchmark. Vijay Bhore describes it in a write-up published 26 October 2026 and in an accompanying GitHub repository. Retrieval cut cost by only about 11 percent while losing roughly 3.4 to 4.6 F1 points against full-context processing. The final configuration reached a micro-F1 of 84.0 (95% CI 82 to 86) across 41 fields on 102 held-out contracts. Its precision was 79.0 percent and its recall was 89.7 percent, against 54.0 for a keyword baseline. The contracts come from the Contract Understanding Atticus Dataset (CUAD), which has 510 commercial contracts. Bhore split them into 328 for training, 80 for development, a 30-contract experiment subset, and 102 for test. The repository says the test set was scored once.

The system extracts 33 legal clause categories, such as change of control, audit rights and liability caps. It also pulls eight typed fields covering parties, dates, durations, governing law and document name. Each answer carries a supporting quote, and the pipeline checks that quote against the source text before returning JSON through a REST API, analytics queries and an MCP interface. Bhore built the evaluation first for arithmetic reasons. The average clause appears in only about 21.9 percent of contracts, so a system that always answers 'absent' would score roughly 78 percent accuracy while extracting nothing. He therefore scored precision, recall and F1 per field. He also measured run-to-run variation of about 1.6 F1 points between identical configurations. As a result, he used paired evaluations and bootstrap confidence intervals before treating a gain as real.

The RAG comparison turns on a mismatch between retrieval and the task. A retriever picks passages that look relevant to a query. But extracting 41 fields can draw on definitions, schedules and appendices spread across the whole document. Bhore's example is a chunk that seems irrelevant to one field but contains a defined term needed to interpret another. He tested full-document context, retrieved sections, and retrieved sections plus document-specific definitions. Full context became the default, with retrieval kept as a fallback for documents too large for a single context window. The write-up's stated cost is that full context increases input tokens. It can become prohibitively expensive for extremely long documents, and it asks the model to reason over more irrelevant material.

Bhore also compared three ways of calling the model: one call for all 41 fields, grouped calls by category, and one call per field. The three delivered broadly comparable quality. The single call was about 3.2 times cheaper than grouped extraction and 18 times cheaper than per-field extraction, and it was the fastest. A prompt-caching detail complicated the cost picture. Changing the response schema affected cache reuse. Standardizing schemas still did not guarantee cache hits on parallel requests, so Bhore budgets with conservative uncached estimates.

Document format was the other major cost variable. Bhore tested pypdf, which reads the embedded text layer, Docling, a layout-aware parser with OCR, and a multimodal model that reads page images. On born-digital PDFs the F1 scores were 83.6, 82.9 and 83.3, which he treated as within experimental noise. On synthetic scans, pypdf scored 0.0 because it has no text layer to read. Docling scored 81.5 and the multimodal route scored 83.0. Docling's OCR also degraded date extraction by about 11 F1 points relative to its born-digital results, and dates matter operationally. The pipeline therefore checks each PDF's text coverage. It sends text-layer documents to pypdf and image-only documents to the multimodal route. Scanned contracts cost about $0.0149 on the final test set versus $0.0080 for born-digital ones. Bhore notes that synthetic scans are not a full substitute for real low-quality scans.

On the model side, Bhore compared gpt-5.6-luna, gpt-5.6-terra and gpt-5.6-sol under the same production-style setup. Luna scored 83.6 on the experiment set at $0.0074 per contract. Terra scored 82.7 at $0.0819, and sol scored 82.6 at $0.1597. The larger models cost roughly 11 and 22 times more without better overall accuracy. Sol was worse on legal clauses because it flagged passages that did not meet the dataset's definitions. The write-up adds a caveat. The prompt was optimized mainly for luna, so the comparison covers complete model-plus-prompt configurations and does not show that luna is universally better. Prompt work itself moved development-set F1 from 81.52 to 82.70, a gain of about 1.18 points with a paired 95% interval of +0.07 to +2.38, for roughly 25 percent higher inference cost.

Validation sits on top of the structured output because, as Bhore puts it, a syntactically valid JSON response is not necessarily correct. The checks cover schema conformance, value normalization for dates and jurisdictions, quote grounding, and domain consistency rules. When needed, a single targeted re-ask passes validation errors back to the model. Grounding matching moves from exact text to whitespace and punctuation normalization, fuzzy matching, and matching across page breaks and footers. The write-up reports that supporting quotes were found in the source 99.9 percent of the time on born-digital test PDFs. The repository reports zero hallucinated values among 640 false positives. The blog reports that none of 316 born-digital test false positives had fabricated evidence. The write-up does not reconcile the two counts. The cloud load test reported a p95 end-to-end latency of 63 seconds. The write-up puts the cost at about $0.008 per born-digital contract.

The MCP interface exposes four tool groups. search_records runs filtered queries with a quote per matched clause. get_record returns projected fields with evidence. run_analytics runs the fixed analytics queries. submit_extraction and get_extraction_status handle adding and polling documents. The repository includes a command that runs 25 gold-derived tasks through an agent to compare description and granularity variants at about $0.05 per variant.

## Why it matters

Companies extracting data from contracts face a cost and accuracy choice between full-document processing and RAG, and this benchmark frames that tradeoff.

## What you can do

Teams extracting data from contracts can run the same comparison on their own held-out documents before choosing an architecture.

## Original source

- [GitHub - vijaybhorems/document-intelligence-pipeline at lens.vijaybhore.dev · GitHub](https://github.com/vijaybhorems/document-intelligence-pipeline)
- [Building a Production-Grade Document Intelligence Platform: Architecture, AI Engineering, and the Tradeoffs Behind Every Decision](https://lens.vijaybhore.dev/building-a-production-grade-document-intelligence-platform-architecture-ai-engineering-and-the-tradeoffs-behind-every-decision/?nlb=OWTm5uLbWe)

---

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it.

Cite as: IKANGAI AI News, "Contract extraction benchmark on 102 contracts finds full-document processing beats RAG", 2026-10-29, https://ai-news.ikangai.com/story/contract-extraction-benchmark-on-102-contracts-finds-full-document-processing-be/
