An archived financial statement can be easy to read and difficult to migrate. A person sees headings, detail rows and a final payout. In their mind it all makes sense. But in order to make the financial statements legible for a database, the database needs explicit relationships between them: which amount belongs to which field, which month governs the statement, and which figures describe individual transactions. A PDF can preserve the appearance of those relationships without storing them as usable records. It’s perfectly readable to a human, yet impossible to immediately transfer into database records without an intermediary processing pipeline to change the binary shape of the same symbolic information.
We encountered this problem while migrating PDF financial statements archived in WebCRM into PostgreSQL for a commission reporting system. Inside the archival PDFs, the file could contain a mix of native text, tables and embedded screenshots. Time and data information inside the file could vary and disagree across the different fields inside the same file. OCR could read the PDF, but through binary conversion, more information could be revealed beyond what’s visually legible.
We chose an extraction harness to effectively solve this interpretation problem at scale. The harness, a Python workflow that assembled document input data and instructions, ran one agent instance per PDF task and saved structured results with source traceability. The model read and classified the input PDF. The surrounding code controlled what it received, checked the response structure and retained the result for review. Once a parallel pipeline of such agent instances completed, then separate programmatic processing of the data ran to convert the extraction worker outputs into database-ready records.
This approach solved two problems whose solved interpretation depended on business conventions and codified interpretation. An authoritative structured export most likely needs only parsing and mapping. But in our case, we first had to reconstruct the relationships expressed on the page while preserving historical errors and walk backwards from the data we had to cover missing information.
This guide breaks down how we made it work in three parts: build the evidence bundle, turn domain decisions into reusable instructions, then run the extraction harness at scale over the data corpus and parallelize review and mapping one document at a time, one harness per PDF. The goal with this guide is to present reusable ideas on how to extract structured data from legacy format documents. The following steps generalize, as a crucial part of the pipeline is the human in the extraction loop who is the teacher of how to interpret the records. The human’s knowledge and interpretation are a key part of making the system able to scale on data without programmatic structure.
See What OCR Does Not See
Start With Edge Cases
The first job is to bake in enough evidence inside the extraction harness to be able to succeed in doing a full extraction on a new document it has not yet been tested on. We began with problematic documents that expose different edge cases. Examples of such are layouts that required pre-existing knowledge, dense tables, implicit structure and meaning of headers. The ability to successfully parse this kind of document requires more than schema validation and mapping. To effectively produce an output that is a structured reflection of the reality of the document, the system requires hands-on analysis from someone who knows the correct interpretation and can supply the system with its correct meaning. Once this hands-on work is sufficiently iterated across a large enough sample, only then can the system be parallelized and run over the full corpus. Edge cases are a sound place to start, as they give you substantial problems that need explicit heuristics. Here, a solved edge case problem is worth a thousand words (tokens).
We iteratively worked through an initial sample of PDFs with Mistral OCR, pinned to mistral-ocr-4-1. We then sent the remaining 1,920 PDFs from the broader archive through sixteen asynchronous batches using Mistral’s Batch API. We store the raw responses in the archive, so that downstream processing can work independently from the OCR step, reducing the cost of OCR through the batch endpoint versus the standard endpoint.
The OCR batch stage prepared one interpretation format of the broader archive. The OCR outputs proved a great starting point from where we could apply further enhancements to the OCR output content.
Keep the Table Intact
The OCR configuration requested HTML tables, layout blocks, headers, footers and embedded images. Then we converted the tables into fixed rows and columns and gave each block a stable reference to its page, allowing the system and the iterations to be referred to by humans, agents and code. Blank cells and column positions survived alongside the text.
In the anonymized report crop below, the code 10 sits under the stacked heading T K. The neighboring O K cell is blank. The saved extraction preserves both positions, recording the blank as an empty string, "".
Put simply, reading these columns correctly is not something an agentic system or an OCR service will be able to perfectly execute. In fact, we faced several interpretation errors like this, where the value, here, “10”, would be incorrectly attributed +/- one column position, making the OCR reading incorrect.
These kinds of interpretation errors are probabilistic, meaning some of the OCR outputs were indeed correct, all depending on the quality of the screenshot, or other matters such as how many rows were present in the screenshot. This then becomes a perfect foundation for explicit instructions about taking the OCR output with a grain of salt and cross-referencing it with the visual copies of the same source table. Visual cross-references require a visually enabled LLM for the pipeline to work, but most open and closed models currently support visual/image inputs. This makes visual cross-references a key part of the extraction harness’s ability to cross-reference its work, minimizing the opportunity for incorrect extractions.
Visually Cross-Check the Data
Each format of the input PDFs provides the extraction harness with a different shape of the same information. A full-page image shows where a table belongs in the document hierarchy. A close-up makes its smallest characters readable and its structure verifiable. OCR supplies text from embedded screenshots, while programmatic native text extraction offers another reading of characters stored directly in the PDF binary. The harness synthesizes these formats so the model can effectively cross-compare the same information across them to then more effectively extract the correct structure and meaning.
Poppler‘s pdftotext -layout is a programmatic command-line tool that supplied native text extraction from the PDF file. I view it as an essential cross-check tool for signs, characters, identifiers and lines that images/OCR might have a low-quality interpretation of. A minus sign lost by OCR, or pixelated in an image, could change the meaning of an amount. Therefore, harness instructions required comparison of the same information between text and images. An additional requirement took shape as a reverse check of the PDF data: find visible content missing from the OCR, including small labels, blank columns and fragments at screenshot edges.
Poppler’s pdftoppm is another programmatic command-line tool that, when run, renders full PDF pages at a maximum dimension of 2,400 pixels, from which images can then be produced and stored. We produced two image forms of every page of the PDFs. One form, the full-page view, took priority over supplementary views such as table-focused views and edge-case example images. The symbiosis of these dual views allows for the extraction harness to effectively load the context on potential edge cases it might face in the content. Presenting the edge cases as images is a more effective form of communication than explanations. An image is worth a thousand words.
Visual copies of the PDFs are not the holy grail, as underlying text objects and image resources can contain material that the visual rendering of a page does not hydrate, so visibility also needed a cross-reference. This is why the image approach is a piece of a three-body package.
The following pseudocode shows the preparation order. The OCR input is a saved response, and the helpers represent the application’s document-processing code.
def prepare_evidence(pdf, saved_ocr, image_budget):
blocks = structure_with_source_refs(saved_ocr)
native_text = read_native_text(pdf, layout=True)
pages = render_all_pages(pdf, max_dimension=2400)
if len(pages) > image_budget:
raise NeedsPageGrouping(pdf)
images = list(pages)
closeups = render_visible_regions(pdf, dpi=300)
images += closeups[:image_budget - len(images)]
return blocks, native_text, images
The image budget sets how many images a single model request can receive. It keeps the visual input bounded and gives the harness an order of priority: full pages first, then close-ups, then teaching-example images if space remains. The guard above stops a request if the full pages alone exceed that budget. Those documents need a grouping strategy that preserves relationships between pages, rather than silently dropping a page. Teaching examples still contribute their OCR, reviewed output and decision notes even when their images do not fit.
Inspect and Reveal Visually Hidden Information
Comparing representations can also reveal information beyond the displayed page. During source review, we extracted the underlying images from PDFs whose tables ended at a crop boundary. Some bitmaps retained readable rows outside the visible crop. Others were already missing the relevant content. Inspecting both the bitmap and the rendered page distinguished recoverable information from content absent from what’s visible on the PDF page you can see.
Here, the page cuts through a source code that remains complete in the embedded image. The original extraction left the row’s cells unreadable, represented as null. Later source review recovered the code 10.
The extraction harness is able to review what is hidden through the use of programmatic extraction of the embedded source images, hidden inside of the PDF binary.
The conventional methods of PDF data extraction, usually one-format extractions, most likely relying on OCR alone, simply cannot be compared to the proposed solution in this guide. Through this approach, we are able to extract information that is not visible in the PDF itself if it were to be visually extracted alone.
The extraction evidence bundle combines structured OCR, native text, page views and source references. These records make a reading cross-referenceable, but deciding which possible interpretations are correct warrants a record of codified judgement and context to inform the extraction harness of the right directional interpretation.
Codify Domain Judgment and Make It Reusable at Scale
Decide Which Evidence Governs the Record
The conflicting interpretations of the meaning of the data illustrate the next task in making the extraction harness functional at corpus-wide scale. This is where human inspection over the corpus is an essential task. The knowledge then needs to become an explicit rule the harness agent can internalize as it completes its extraction instance.
I worked with GPT-6 Astra in Codex to investigate the sources and build the extraction method. My familiarity with the domain knowledge helped me recognize the implicit data structures. Astra could trace the implementation, compare representations and test an explanation against the documents. The agentic exchange made analysis of each uncertainty specific enough and verifiable, making the agent trace of the conversation a codified bundle of insights the extraction harness used as supporting instructions.
A few examples of the inconsistencies of the statements: period identification could vary from one period on the detailed insurance rows to another in an embedded report, and to a different period annotated beside the final total. For this commission history, the period on the property and casualty insurance details governed the destination month, and the annotation turned out to be an error that occurred once in 24 months. I decided that the detailed insurance rows were the reliable source of periodic truth, which cascaded onto the rest of the period data.
Consider an illustrative conflict: the detail rows say January and the final-total annotation says February. Given that this is an error, the correct extraction is to override the final-total annotation with the higher-value context from the detailed rows.
To generalize this example, establish the authoritative context from domain knowledge. Work through an actual conflict and record the evidence that documents the path to the final decision. Working with an agent allows you to speed up this process and allows you to produce a context bundle from the recorded agent trace, from the session transcript.
Preserve Errors and Unknowns
A historical statement’s summated amounts could disagree with a summation of the printed rows. Our extraction policy retained both, including the original row amounts and the summated amounts. Correcting the arithmetic during extraction would have replaced the report’s historical accuracy with a new calculation. This was a deliberate tradeoff the system had to make.
Missing meaning also needs an explicit representation. A pension (one of the present insurance types) report contained readable numbers without the headers to establish their roles. A value that resembled a member count (employee count, how many employees receive a pension) retained an unknown role because the source did not allow the system to identify it because of the lack of proof. In an illustrative reading of 12, the value would be known, but its role unresolved. Recording it as a count of twelve members would add a claim the document had not established.
This becomes especially important when the destination requires a field the source does not supply. A database requirement cannot force the appearance of a historical fact. Keep the source extraction incomplete where the evidence is incomplete. Any later derivation or estimate belongs in the mapping policy, with its assumptions recorded explicitly.
These cases gave us concrete rules: retain conflicting labels, preserve printed amounts and leave unsupported meanings unresolved. To make those rules usable on the next document, we paired them with examples showing the expected result.
Turn Decisions Into Worked Examples
A bit of the magic that we could achieve with this system was its ability to replicate the correct extraction results over a large corpus of thousands of documents. And in order to achieve this scalability of the extraction harness, we needed to turn decisions into worked examples: examples with a full trace of the reasoning.
We assembled three human-reviewed source examples, each pairing document evidence with the expected structured output and notes about the interpretation. The notes identified the evidence, the applicable rule and the resulting extraction decision.
A condensed lesson for the period conflict looks like this. This is an illustrative decision note, not a raw record from the archive.
evidence:
- Period on property and casualty detail rows
- Conflicting caption beside the final total
rule:
- Retain both labels with their source locations
- Detail-period evidence governs destination mapping
reason:
- >-
The period selects the applicable
commission history and rates
extraction:
- Identify each label and its role
- Preserve the conflict for review
mapping:
- Use the reviewed authoritative period
This example provides the extraction harness agent with a clear information hierarchy: what takes precedence and what information could be affected by the hierarchy.
To further power the agent, we supplied a small sample of historical database records to act as a classification reference. For each insurance type, we added five historical rows. Historical records allow the agent to get a clear example of what X insurance type looks like historically, essentially giving the agent the opportunity to recognize patterns as a part of its classification reasoning.
The integration of historical examples and human-guided decisions produced reusable instructions and examples. We then incorporated all of this into the environment of the extraction harness. We supplied it as context, without training a new model or passing excess context. Each extraction harness worker receives a complete assignment with the original PDF as input, reading instructions and checking the environment for direction and decisions, and using examples and classification references for verification and assumption checks.
Turning this into a scalable and parallelizable pipeline was essential to be able to work through the entire corpus of data. Alone, this would’ve taken me a week or more to complete with a more hands-on programmatic flow.
From One PDF to the Whole Archive
Give Each Extraction One Clear Job
The extraction harness enabled the problem to be solved within one pipeline, one command, to orchestrate the full extraction. The rough flow of the harness goes as follows: read one PDF and return a structured representation of its contents, and store each response and all payloads so that retries and responses could easily be done. We built the harness using PydanticAI, with Kimi K3 as the model of choice, at maximum reasoning effort through Baseten’s Model APIs. Our Baseten account’s rate limits allowed us to parallelize the task, reducing the total time the task took to conclude.
Within the harness, the model had no callable tools or database access, because we believe this kind of agent increases in quality when it’s allowed to focus on a lower number of things at once. Scoped to reading and classification tasks, the agent could prepare evidence more efficiently, and data reconciliation instead became a task done with Python. Finally, purpose-built Python mapping code applied the final data processing and calculations. In the period example, the model’s job was to identify both dates and their roles. Overwriting the month belonged to the mapper.
The code below shows how PydanticAI gives each PDF its own agent and a defined output format. ExtractedPDF describes the expected fields, while reading_contract supplies the reading instructions. PromptedOutput puts that format into the prompt and lets Python parse the returned JSON. The configuration allows up to two output retries within a total of three model requests, with no tool calls. This is the core of one extraction job, with the model connection, schema, instructions and document evidence prepared separately.
from pydantic_ai import Agent, PromptedOutput
from pydantic_ai.usage import UsageLimits
async def extract_one_pdf(model, evidence_prompt):
extractor = Agent(
model,
output_type=PromptedOutput(ExtractedPDF),
instructions=reading_contract,
retries={"tools": 0, "output": 2},
)
return await extractor.run(
evidence_prompt,
usage_limits=UsageLimits(
request_limit=3,
tool_calls_limit=0,
),
)
One of the key mental models that arose from this task is the importance of separating “the output has the right structure” from “the underlying claims are correct.” In the problem, there was roughly an 80/20 ratio of data processing needs. The 80% were rather straightforward. The remaining 20% required processing or reasoned interpretation in order for the correct shape to emerge.
An output can contain a valid numeric value, be funneled into a valid column, yet be an incorrect representation compared to the source data’s meaning.
Anthropic’s guide to agent evaluations distinguishes the harness that runs an agent from the evaluation harness that tests it. The object of evaluation is the model and its harness working together, where programmatic checks trigger a feedback loop to the agent, and a reviewer agent reviews the work of the preceding agent, acting as a validator and editor, coming in as a second party to review and approve, giving the system another chance of success.
How to Scale the Task
The most important problem to solve with scaling this kind of harness is knowledge of the data the harness is tasked to deal with. There are two paths that could occur: a) the harness has sufficient context and knowledge about the data and how to interpret it, along with familiarity with edge cases, or b) the harness is exposed to data it does not have sufficient context or knowledge about, making the agent an unreliable tool for such uncertainty.
Dealing with data through this kind of harness forces you to scope the harness inputs strictly. This means that if you have a diverse range of data formats, then it’s more suitable to instead opt to make multiple purpose-built harnesses per data type, and iterate the process of curating the context.
The second important problem to solve is how to chunk each body of work. How much work does one harness instance complete? The answer boils down to the kind of data you are working with and the complexity of it. Try to find natural chunks in the data, where each harness can conclude its work cleanly.
A complete PDF kept headings, detail rows and totals together across pages. Adding N2 reports would introduce data that the N1 report extraction did not need. Splitting every page into a separate job will remove the context that connects and explains a table summation. For this archive, PDFs ranging from one to eight pages gave us a useful boundary.
Work per instance. Choose the smallest unit that contains the evidence needed for a complete result. Here, each instance read one whole report. Longer document families may need sections grouped around tables, headings and their supporting pages.
The specifics of how you choose to parallelize and run the harness code should be built around the urgency of the task, data amounts, and resources available.
Concurrent jobs. Up to six independent document jobs ran at once. A shared scheduler paced the requests within our Model API’s token and request limits.
Carry the Evidence Into the Database
Producing the final migration payload was in our case partly a task of reverse engineering data from known commission rules, in order to backpropagate the required data for our database schema. Given the programmatically valid dataset produced by the extraction harness, these enrichments could be done entirely by using dedicated Python code.
One specific tradeoff we had to make was in retaining historical correctness of the extracted records, both sum parts and individual rows. This produces a paradox, as once the historical rows got recalculated, their sum does not match the original PDF sum. The tradeoff here is that a perfect extraction does not fix historical errors. But we found a solution through inserting an explicitly labeled row that would reconcile the difference through adding what made up the margin. This allowed us to retain both row-level accuracy and the final sum accuracy. The task was historical accuracy, not overriding the data with mathematical perfection.
With those distinctions preserved, the results can be read precisely by the code and by humans.
What Comes Next
The extraction harness worked by combining complementary views of the documents with domain knowledge made explicit. OCR, native text and images gave the agent ways to cross-check a reading. Reviewed examples taught it how to interpret the relationships that a database needs. Keeping each instance focused on one PDF made that method repeatable across the archive. For a similar migration, start with the difficult documents, turn the decisions into reusable context, then parallelize the work. Preserve what the historical records actually say, including their errors and unknowns.