A reference cuts the cost

Data can be an incredible asset to have when engineering an agentic workflow. The engineering question in this post is how to make the contextually right data available when the agent needs it, and what improvements this introduces.

Consider an agent doing a body of work consolidating commission books. It receives a request to investigate X correction in Y period. In this agentic body of work, there is no reference to what exactly the correction should look like. Here, the agent needs a reference data tool. A reference data tool provides context about specific things from a database. In this case, “corrections” is one of the reference tools. The agent can use it to access database records showing what a “correction” looks like in context.

Now consider a person doing the same work, opening a particular spreadsheet, PDF, or dashboard before making the next decision. Knowing where to look is part of knowing how to do the job.

I see reference data in an agentic system in the same way. While doing X task, the agent calls a tool to obtain Y information before making Z decision. The tool queries a source at runtime, so the reference can reflect fresh information available when the work is being done.

The word reference is a specific choice. In audio, a reference track gives an engineer a known basis for comparison. They can compare characteristics of the mix against a recording selected for that purpose. Reference monitoring, signals, and playback levels serve related roles in establishing known listening or measurement conditions. iZotope on reference tracks, Genelec on monitoring calibration

The useful part of that mental model is the relationship between the reference and the judgment. A familiar recording can help me judge a particular aspect of a mix. A rate schedule can help establish the applicable commission. An earlier correction can help me recognize a recurring problem. Each reference is useful because I know what I am consulting it for.

The reference data is the specific data you point your agent towards. Its usefulness depends on that specificity. Giving an agent vague access to all data does not tell it which records answer the question in front of it. A deliberately chosen query can make the relevant source and its meaning much more explicit.

There are three things I want the system to make clear: where to look, what to ask for, and what the result establishes. For the commission task, that means identifying the right person and period, retrieving the applicable rates and transactions, and understanding whether a correction has already been made and whether adding a correction is correct.

This does not require all reference data to be historical. The reference might be the latest status of a transaction, the agreement effective for a particular period, or a set of earlier cases. What matters is that the agent can consult it while doing the work.

The SQL query can remain the same while the data changes. A new correction becomes available through the same lookup. An amended agreement can be returned with the period to which it applies. The stable part is the meaning of the query. The result supplies the changing facts.

The specificity of the agentic work helps determine the reference data scope. When the task has a specific goal, such as making a correction correctly, we can identify the information retrieval needed to make that decision. Through iterative engineering, we can find what the reference dataset should be to make the task accurate, repeatable, and effective.

Engineering the reference query

A reference query is a deliberately designed request for the information a particular agent task needs. It defines the source and the parameters, and helps clarify the meaning of the returned result.

For a commission workflow, the available queries might cover agents (as in sales agents), agent details, agent rates, agent sales, and agent corrections. Broader queries can expose sales and corrections across the curated scope, with filters and bounded results. Here, agent refers to the salesperson in the business records. The AI agent consults those records through the client application.

Each query answers a different question. Which person does the request concern? Which rate applies to this period? Which sales contribute to the amount? What adjustments already exist? Have comparable cases been handled before?

The reference query can be exposed as a tool through the Model Context Protocol, or MCP. The server publishes the tool’s name, description, and input schema. The agent requests it through the client, the server authenticates and runs the query, and the result returns as context for the next step. MCP provides the interface and data, while the application determines the query. MCP tool specification

The engineering can be quite small:

• A client tool called get_agent_rates could accept a person identifier and a period.

• The server limits the records to those the caller is allowed to access.

• The server returns the relevant rate records immediately, without the model having to guess its way to discovering the right data.

Fundamentally, I view reference data as minimizing the amount of trial and error that the agent has to do to achieve its given goal.

A feature of engineered reference data tools is their ability to deal with inconsistencies in source identifiers, retrieve matching records across periods, and return a bounded result with summaries. I think built-in mechanisms for catching data errors in a SQL reference query do two things: a) stop data errors from reaching the agent, and b) help it immediately understand what ‘data error’ means in the given context when the query is exposed to it. That is recurring work reference queries can perform directly. The agent uses fewer resources and gets additional valuable, hard-to-define context.

Reference data allows us to instill judgment in the tool design.

We tested this by asking an agent to reconcile a commission table against a proprietary insurance commission dataset, kept private from publication. The task was specific: compare the submitted amounts for 50 agreements with the amounts recorded in the database, and return a complete comparison.

For each task, we selected 50 distinct agreement numbers from the sales table for one period. We used their recorded commission totals to construct a Power BI-style table, then artificially changed the submitted amounts for 12 or 13 agreements. Alternating between 12 and 13 gave us a 25% discrepancy rate overall. The agreement numbers and database records stayed unchanged. The agent was not told which amounts we had changed or how many.

We gave the agent two different ways to get the reference data. With get_committed_agreement_totals, it supplied the period and agreement numbers to a tool with prewritten SQL. The tool returned the recorded total for each agreement. With query_database, it wrote its own read-only SQL against the same sales table.

In both conditions, the agent received the same user message, the same plain SQL schema, and the same calculation instructions. The general SQL tool could retrieve all 50 totals in one call. It supported the filtering and aggregation needed for the task, so the agent could decide how to write the query without having to discover the database schema first.

One agreement could have several sales postings in the period, including negative and zero amounts. Separate postings with the same amount still had to be counted separately. The query had to sum all relevant postings before rounding the total to two decimal places. These were the rules we implemented in the reference query, and the rules the agent using general SQL had to implement itself.

• Identify the period and agreements. With reference tools, the agent supplies tool arguments. With general SQL, the agent expresses the scope in SQL.

• Include the right postings. A prewritten reference query or agent-written SQL.

• Aggregate and round. A prewritten reference query or agent-written SQL.

• Compare with the submitted table. The agent completes the same structured answer in both conditions.

• Judge the answer. The same independent deterministic grader in both conditions.

The reference tool received no submitted amounts and no expected answer. It simply returned the database totals. The agent still had to compare those totals with the user’s table and complete the response.

We specified that response with a Pydantic schema. Each row had to contain the agreement number, submitted amount, recorded amount, difference, and a match or mismatch status. The difference was the submitted amount minus the recorded amount, in NOK. All 50 agreements had to appear exactly once, including those whose amounts matched. Both toolsets also included the same optional tool showing a valid 50-row response with generic values. This kept the schema guidance the same on both sides.

We calculated the correct answer separately in Python, adding up the individual database postings with exact decimal arithmetic. A passing answer had to satisfy the schema and match the expected period, agreements, and values in all 50 rows. The score was fully correct tables divided by attempted tables. One wrong field or a missing answer meant a failed attempt.

We ran 20 held-out tasks, each with a different set of 50 agreements, and repeated each task five times per toolset. That gave us 100 attempts with reference tools and 100 with general SQL. We used GLM 5.3 Flash through Baseten with the same model settings in both conditions. Pydantic AI called the Python tools directly in this experiment. MCP transport was outside the comparison scope given that the tools ran locally.

What we observed

Both toolsets produced 99 fully correct tables out of 100 attempts. With reference tools, the agent used a mean of 2.03 model requests per attempt. With general SQL, it used 3.90. That is about 48% fewer requests with the reference tool. Mean elapsed time per attempt was 42.72 seconds and 86.33 seconds, respectively. A turn in the figure means one request to the model.

Explore the interactive figure · Saved data (JSON) · Sources and measurement

One reference attempt timed out. One general SQL attempt returned a table with one incorrect row. Both counted as failed tables. The timeout also left us without a token-usage receipt for its final request. We recorded $0.42710 in inference cost for reference tools and $0.92795 for general SQL, but the reference total is incomplete. Its exact cost remains unknown.

The turn counts include requests used to recover from rejected SQL calls. After the run, we also checked the result with the affected attempts and their reference counterparts removed. Reference tools still required fewer turns. The rejection counts and that comparison are in the measurement notes.

Both conditions had access to the same proprietary dataset. What we tested was the value of engineering a specific way to use it. The reference query put the filtering, aggregation, and rounding into maintained code, with a tool description and a result shaped for the task.

Keeping the reference current

The operational system already maintains the records. When a new sale or correction is recorded, the same reference query can make it available to the agent. We do not need to keep copying updated values into its instructions.

A read replica can provide this access, with some delay as changes arrive from the primary database. The agent still needs to call the tool again when it needs an update. PostgreSQL hot standby documentation

Current data also needs the right scope. A correction for last month requires the rate that applied last month. The query’s scope and calculation rules still need upkeep when the schema or business rules change.

Looking ahead

I am interested in how much of the reasoning behind earlier work we can make available as a reference. For the commission task, that could mean retrieving a reviewed correction together with the records, agreement passage, and explanation that supported it. The next agent could see why the correction was made and assess whether the same reasoning applies.

This gives the system a way to accumulate useful references through the work itself. A reviewed decision adds more than a final amount. It preserves what was known, what was decided, and why. The reference tool would need to distinguish reviewed decisions from proposals the agent produced but nobody accepted.

Other tasks could draw on different kinds of data. A reference query might retrieve a current document passage, an image from an inspection, or an audio observation. The same design question applies: which part of that source does the agent need, and what can it establish from it?

A reference could also come from a computation performed at runtime. Before recommending a change, the agent could ask a simulation to evaluate it against the current operating state. The response would need to carry the assumptions and conditions behind the result. A prediction describes what a model expects to happen, and the agent needs to preserve that distinction when comparing it with observed records.

The direction I want to explore is how much useful context a system can make available at the moment it is needed. For each body of work, that means choosing the references the decision depends on, making them available through specific queries, and checking that the agent has used them correctly.

Sources and measurement

We ran the study on 25 September 2026 using GLM 5.3 Flash through Baseten, with max reasoning effort and temperature 0.2. Each attempt had a limit of 12 model requests, 16,384 output tokens per request, and 240 seconds. The 200 scored attempts cover 20 held-out tasks, five repetitions, and two toolsets. Each pair used the same submitted table and expected answer, with randomized pair and condition order. Each attempt started a new conversation. The comparison evaluates the complete tool design, including the query, its description, and the response shape.

The task inputs were constructed reconciliation requests using the proprietary insurance commission dataset. We selected the latest valid reporting period with at least 500 distinct nonblank agreement numbers, then used a seeded selection of disjoint groups of 50 from the holdout pool. Development and holdout agreements were separated. The submitted amounts changed, while the source records and agreement numbers remained unchanged. The 25% discrepancy rate was an experimental choice, not an estimate of how often reporting errors occur.

The saved measurements and analysis configuration (JSON) contain aggregate task summaries only. The proprietary dataset remains private, with no source agreement numbers, commission amounts, prompts, or model answers published. Each point in the figure represents one task and toolset, averaged over five attempts. Cost is inference USD per attempt, turns are model requests per attempt, and score is the fraction of complete 50-row tables that passed the independent grader. The turns/score view includes all 40 points. Cost views omit the one reference task with incomplete usage receipts. All scored attempts remain in the condition summaries.

The uncertainty intervals resample whole task groups, keeping both toolsets and all five repetitions together. With 20,000 bootstrap draws and seed 1701, the reference-minus-SQL mean turn difference is −1.87, with a 95% percentile interval of −2.13 to −1.61. The score difference is zero percentage points, with an interval of −3 to +3 percentage points. Equal observed scores do not establish equivalence.

The general SQL tool rejected 13 calls across 12 paired attempts. In a post hoc check, removing both sides of those pairs left 88 attempts per toolset. Mean turns were 2.03 for reference tools and 3.68 for general SQL, about a 45% reduction. Both passed 87 of the remaining 88 tables. This shows how the result changes when those attempts are excluded. It does not tell us how the agent would perform with a different SQL guard. Query text was deliberately not retained, so we cannot reconstruct each rejected query.

Costs use received token-usage receipts and the rates recorded for this run: $0.15 per million uncached input tokens, $0.03 per million cached input tokens, and $0.50 per million output tokens. The reference total is incomplete because the scored timeout has no final usage receipt. An aggregate provider usage check found a zero-token request during the timeout’s idle window but could not link it to that attempt. We therefore left the missing cost unknown. Database compute, tool execution, engineering, and maintenance are outside the cost measure.

One unserved HTTP 429 request was excluded and rerun. Its record remains in the audit trail, separate from the scored timeout. The replacement is included in the 200 scored attempts. Costs reported here cover the scored attempts with available receipts, not the entire provider invoice.

The tools queried a read replica at runtime. Each uninterrupted part of the run used one fixed, read-only database snapshot so the two toolsets would work against the same evidence. Across resumptions, the saved commitments matched the selected source records, submitted prompts, and expected answers in all three snapshots. This checks that the task evidence matched across the interrupted run. It does not establish that the rest of the database stayed unchanged, and the study did not test how agents respond to records changing during a task.

The harness retained no source records, submitted tables, or model answers as a dataset. Its local ledger preserves scores, usage receipts, timings, error categories, and keyed commitments that can be checked without storing the underlying financial records. Those commitments cannot reconstruct an earlier database state or an unsaved answer. Applying a different grader to the original answers would require data we deliberately did not retain.

Inside the commission reconciliation platform we have built for a client, call them client X here, is an agentic feature that lets the user cross-verify their data, check corrections, and do whatever else is useful with it. One of the key problems we faced in delivering this service was that the cost of the model powering the agent quickly started to eat into the margins.

This is why it was difficult

The model we had chosen was a specific choice, and it gave the system the right capabilities at the right price at the time. The agentic feature was added in February, on Claude. By summer it ran on Claude Sonnet 5 with adaptive thinking and a one-hour prompt cache, and that is the configuration measured below. At the time it was the right choice, balancing cost, speed, data retention and developer velocity.

Sonnet was also the easiest choice, because we did not see the value in going for a less tested open model. Keep in mind that this was February. The GLM-5 report came out on 17 February, claiming state of the art on the major open benchmarks in its own abstract, and the open models that followed it through the spring were still to be proven on anything like our workload. Since then, the market of open models has grown greatly in count and quality.

So the agent for client X was ripe for a round of model choice optimization.

This is what we changed

The first requirement with this kind of system, which handles large amounts of sensitive data and needs a higher level of environmental control, is to consider which model API services meet the data retention requirements while also meeting the quality and cost targets that are the foundation of the work.

Through my experience with a wide range of providers, I do have a selected few. For the client X system I chose Baseten, which provides accounts with zero data retention out of the box. In addition to meeting the privacy requirement, Baseten brings up the latest models quickly and has run an inference service of the highest quality for years.

Beyond the provider comes the evaluation harness and the workload tasks to run each configuration against, to find the suitable choice. In the client X system we already had seven months of chat history in the database, about 400 user messages across 90 conversations, which we had access to and from which we could build the evaluations. This let us simulate the specific use cases accurately and find the exact tradeoff between cost, quality and intelligence.

The tests we created were mainly about the model using a SQL tool against a read replica of the customer database. The agent writes SQL, receives the result into a sandbox, and runs commands there to read and use the data. Other tasks were to process a pasted or uploaded file into the agent’s sandbox, from where it uses the same commands to compare the file against database data, to check for differences or to help the user understand what they have.

The beauty of this kind of agentic system is that its outputs can be validated and checked. What this fundamentally means is that when testing a wide range of models, I can programmatically verify whether each model’s output is correct. With that feedback loop in place, the model evaluations scale with much less work.

I designed the evaluations to work through the problems users had already used the system for, and to verify the agent’s output against the data in the customer’s database. The harness is 22 cases with deterministic gates: a figure resolved from the database at run time that must appear in the answer, a tool that must have been called first, a table that must reach the panel with the right row count, a write that must be proposed and never executed, Norwegian orthography that must survive. Six of the cases attach a file or a paste and run in the sandbox. Every case runs in-process against the same tools and the same prompt, and the eval runner records the tokens the provider reported and what they cost at list price.

Keep in mind that there are a lot of different model configurations. From the effort level (an approximation of the size of the reasoning budget), to which inference provider (and their model deployment configuration) is used.

In our comparison below we look at the previous model used for client X system, with the new model configuration, and comparing the costs before and after on the same cases.

In its production configuration: Sonnet 5 with adaptive thinking, and GLM-5.3-Flash on Baseten at max effort. GLM-5.3-Flash passed 62 of 66 runs across three first repetitions, succeeding once environment error was fixed.

Cost per turn is where the two separate. The shared prefix, the system prompt and the tool definitions, is about 17,000 tokens on Sonnet and 11,000 on GLM. So most of the bill is the input tokens that get cached prefix. With the prefix served from cache (which is what an iterative agent loop does), the turns after the initial message is billed at, $0.012 per turn on Sonnet 5 and $0.0016 on GLM-5.3-Flash. That is a saving of 86% per turn, on every one of the 22 cases, GLM is cheaper by between 4.6 and 12.3 times.

The comparison harness starts a fresh conversation when running each case, which means that we loose out on the cache read for the first step. Priced that way the run cost Sonnet $1.48 against GLM $0.11, a saving of 98%. Running a cold (not cached) test is the honest bill for a workload of mostly one-turn conversations, and the 86% figure is the honest bill for a session that continues (warm, cache hit).

Speed went the same way for most of the set. GLM-5.3-Flash was faster in 15 of the 22 cases, with a median of 5.8 seconds versus 7.5, and the difference on direct lookups was about a second and a half. GLM was slower on seven cases, and badly so on one: a formula question that Sonnet answered in 3.6 seconds took GLM 25 seconds at max effort, almost all of it spent reasoning. That is one downside of running GLM at max effort.

A practical solution to that would be to configure a timeout on each API call so that, if the model exceeds X seconds, a new API call immediately fires at a lower reasoning level. Given GLM’s lower cost, this is a pragmatic way to eliminate pathological over-reasoning. The obvious qualification is the task itself: if it genuinely requires the additional reasoning time, then permit it.

We have so far discussed only two models, Sonnet and GLM 5.3 Flash. This is after having iterated the evaluations over the models and configurations available. I converged on GLM-5.3-Flash as the suitable option for this workload.

The effort setting is another axis the evaluation measures. On direct lookups, GLM-5.3-Flash at low effort responds in a median of 2.2 seconds versus 5.0 at max, spends 14 reasoning tokens per turn versus 711, and costs $0.0012 per turn versus $0.0016. High effort sits between the two.

GLM-5.3 at max effort (the larger sibling of GLM-5.3-Flash, fallback behind Flash in production) passed 39 of 44 runs, took a median of 12.6 seconds, and exceeded the two-minute budget on both runs of the largest sandbox case. At $0.019 per turn, it cost 1.6× as much as Sonnet 5 on a cached prefix and 12× as much as Flash.

This exposes a distinction that headline API prices hide. Sonnet 5 costs $2 per million input tokens and $10 per million output tokens, compared with $1.40 and $4.40 for GLM-5.3. On paper, GLM is cheaper. But on the largest sandbox case, running the models side by side reverses that relationship: GLM-5.3 costs 1.6× more per turn than Sonnet 5.

An evaluation harness is what lets us measure the practical cost of a model rather than infer it from the price of its tokens.

This is why that intervention worked

This cost reduction is possible because of the accelerating capabilities of open models, and because the bounded data on the existing user tasks allows a programmatic feedback loop that makes the model’s output verifiable. That is different from a qualitative task, like the creation of a design or a website, which is much harder to quantify or validate because its output is judged by a person’s opinion of its quality.

I believe that spending time understanding the evaluation harness as fully as possible, and considering creative ways to build feedback loops that validate a model’s performance on a system, is very high value work. It is the precedent that allows every downstream evaluation of a model configuration.

When I have an evaluation that can be programmatically verified, I can scale the number of samples. More samples give higher resolution, because the distance between what was observed and what is inferred from it shrinks. Through such feedback loops I reach the scale at which I can make informed judgements about which model achieves the best outcome. On this set, three repetitions were enough to find that one failure was a provider defect that recurs on every run, and that another was a reasoning budget that occasionally overruns.

This is the measurable result

Measuring the v1 system against the optimized v2, the cost per simulated user task fell by 86% on a continuing session and by 98% on a fresh one, while the answers stayed accurate and reliable. Through the harness, the savings were low-hanging fruit, readily available to collect.

The greater picture is that open models are now cheap, fast and reliable enough that most of the workloads we run could be moved to them, for a more economically effective system overall.

One of the greater potentials here is the increase in margin, which can be used in two ways: retained, or reinvested in a more expensive open model. Since we have cut the cost of using a closed model, that capital is freed and could be spent on a more intelligent model at the same cost. That is an appealing option if users start doing more complex tasks and the cheaper model falls short. The margin can then be used to step up the model on those tasks, to increase the probability of success.

I enjoy this kind of discovery because it reveals that potential of a system needs to be modelled and discovered. The reward is greater operational efficiency.

Sources and measurement

Every number above comes from the platform’s own eval harness, run on 9 September 2026 against a branch of the customer database, and the per-case results are in the dataset. The answers, the tool arguments and the progress labels are not in it, because they can carry figures from the customer’s data. Nothing in the dataset names the client, a user or an agreement.

Prices are list prices read from the providers on the same day: Anthropic’s pricing page for Sonnet 5, at $2 per million input tokens, $4 for a one-hour cache write, $0.20 for a cache read and $10 per million output, and Baseten’s models endpoint for GLM-5.3-Flash, at $0.15 per million input, $0.03 cached and $0.50 output. Reasoning tokens are billed as output on both. The cost in the figures is the provider’s reported usage at those prices, with the prefix on Sonnet priced as a cache read. The cold figures price the cache write the harness actually paid.

Sonnet 5 ran once over the 22 cases. The remaining cells ran two or three times each, and a case’s figure is the median over its runs. Wall time is measured from the question to the last token and includes tool calls, database queries and the sandbox.

The figures are also available as images under /research/model-selection/, with SVG originals and a manifest recording the dataset hash.

This is the kind of work we do at Kylace: evaluate and improve AI workloads against the requirements that matter for the system. We test changes across models and the wider system on the actual work, then turn the results into a reusable evaluation and a recommendation your team can act on.

I have discussed running on-prem AI inference with a number of founders. Most consider it overkill to buy hardware for their own workloads. A few see it as a necessity. The disagreement becomes easier to understand once you separate the hope of a smaller bill from the need to control where the work happens.

For a new workload, I would usually start with an API. It lets you find out what the model can do before you take on the job of running it. Ownership becomes worth considering when you know what work you need done, which locally deployable model can do it, and what you gain by taking responsibility for the hardware.

The model question comes first. In the previous essay, models with the same benchmark score produced very different bills. A deployment comparison has to preserve the same discipline: measure the cost of work you can accept. Buying a machine does not make its cheaper tokens a substitute for a model that does the job better.

The cloud is a batching machine

A model API service gives you access to a serving operation whose job is to get as much useful work as possible from its hardware. Its customers supply the demand, and its software determines how efficiently that demand can be served. That combination is part of what you buy with an API.

Batching is one of the reasons this works. A serving system can process several requests together, sharing the cost of reading the model’s weights across them. With enough concurrent work, the same hardware produces more tokens overall, although each person may have to wait longer for theirs.

SemiAnalysis’s InferenceX measurements make the tradeoff concrete. Kimi K3 on B200 hardware delivers 1,007 total tokens per second per GPU when the target is 200 output tokens per second per user. Lower that target to 30 and throughput reaches 6,783. These are the best measured configurations at each speed, normalized per GPU, on InferenceX’s AgentX workload, which replays Claude Code sessions with a median of 142k input and 444 output tokens a request. That is the shape of an agent loop, and the shape of the work I run.

The provider gets more work from the hardware it already pays for. How much speed you can give up depends on the work. An overnight processing job and an interactive conversation do not need the same thing.

The model matters as much as the speed. On the same GPU at 100 tokens a second per user, GLM-5.3, a 744B parameter model, serves 10,968 tokens a second, five times Kimi K3’s 2,147. Kimi K3 has 2.8T parameters, and its measured configuration spans sixteen B200s, two servers rather than one. The figure carries both.

Caching gives the provider another way to avoid repeating work. If each request begins with the same instructions, tool definitions or document, the system can reuse the computation for that prefix. Anthropic’s prompt-caching documentation describes how this reuse becomes a lower input price. In the replayed sessions, 95 to 98 percent of input tokens are cache hits, so cached input dominates the API bill.

You can use batching and caching when you run models yourself. What you cannot assume is that your workload will keep the machines busy enough to reproduce a provider’s economics. Pooling demand from many customers gives an inference service more opportunities to fill its capacity. On your own hardware, the available work has to come from you.

Control over convenience

Cost is only part of the reason someone might choose to own the system. The nature of the business can make control worth paying for, even when an API would be cheaper.

The data cannot leave. A business may have data it will not send to an external model provider at all. Argonne National Laboratory described that requirement in its work on a local geospatial pipeline: control over model execution, no external API dependencies, and sensitive intelligence processed “without data leaving secure infrastructure”.

Other buyers can get the control they need through a service agreement. In July, Norsk Helsenett, which runs the Norwegian health sector’s network, announced a direct award to Anthropic for Claude Enterprise. Its stated reasons included a separate data processing agreement, control of where data is processed and better security mechanisms. For that buyer, more control was available through the contract.

Before buying hardware on privacy grounds, I would be precise about the boundary the data must stay within. Keeping it inside your building, inside a country and inside an agreed processing environment are different requirements. They need not lead to the same purchase.

The response cannot wait. Some work needs inference close to the device doing it. 1X, building the NEO humanoid, states the requirement directly: “Once models are trained, they need to run directly on the robot with minimal latency.” For that kind of system, the acceptable delay is part of the design. An attractive average response time is little comfort if the slow responses arrive too late.

The model needs to stay available. Once a workflow has been tested against a model, a forced change means work. Providers can retire models, as Anthropic’s deprecation notices document. Running open weights with a pinned runtime gives you control over when that configuration changes, whether the hardware is rented or owned. You take on maintenance, but you can plan the model change around your own system.

All of these are reasons to want more control. None makes a machine secure simply because you own it. A local network can restrict external connections, but access, patches, monitoring, backups and data transfers still need someone responsible for them. That responsibility is part of what you are buying.

Where ownership breaks even

Suppose the model can do the work and either deployment meets your requirements. Now the price comparison becomes useful. How busy would an owned machine have to be before it costs less than the service?

Start with the bill that arrives whether the machine is busy or not. An eight-B200 server starts at $400,328.50 at one reseller. Write it off over three years with no residual value and the hardware costs about $11,120 a month.

Electricity adds about $960. For this estimate I use Nvidia’s 14.3 kW maximum draw for a DGX B200 throughout a 730-hour month, at the June US industrial electricity rate. Together, hardware and electricity come to roughly $12,100 a month, or $2.07 per GPU hour.

This is a partial ownership cost. Colocation, staff, spares, networking and cooling overhead are left out, as is extra capacity to keep the service available through maintenance or failure. The calculation gives ownership the benefit of those omissions.

The amount of work that bill buys you depends on the model, the speed required, and how many servers the model needs.

GLM-5.3 runs on one. At 100 tokens a second per user, InferenceX measures 10,968 total tokens a second per B200. Across eight GPUs that is an assumed capacity of 7.6 billion tokens a day, and a token costs $0.05 at full use.

Kimi K3’s 2.8T parameters do not fit in a server’s 1,440 GB, so it takes two, $24,155 a month. At the same speed it serves 2,147 tokens a second per GPU. Across sixteen GPUs that is an assumed capacity of 3.0 billion tokens a day, and a token costs $0.27 at full use.

That is an extrapolation from the benchmark, not a measurement of the quoted servers. The benchmark’s request shape is a median of 142k input and 444 output tokens, with most of the input served from cache. Your agent will land at a different rate. Before making a purchase, I would want the intended configuration measured on the actual work.

For the crossover, I give the API its strongest case and price every input token as a cache hit. For the benchmark’s request shape, Z.ai’s GLM-5.3 prices blend to $0.27 per million total tokens. One server meets that at 19% of assumed capacity, about 1.45 billion tokens a day. Moonshot’s Kimi K3 prices blend to $0.35. Two servers meet that at 77%, about 2.3 billion tokens a day. The surveys below say almost nobody reaches that utilization.

Without caching the picture inverts, and that is the measure of what caching is worth. At the uncached prices, $1.40 per million input for GLM-5.3 and $3.00 for Kimi K3, the API bills come to $1.41 and $3.04 per million, and owning breaks even at 4% and 9%. This is the opposite bound. Agent loops reuse much of their prefix, and the AgentX sessions report 95 to 98 percent cache hits. The cache is the provider’s discount for work it does not have to redo, and it is most of the reason the API is hard to beat.

Renting the GPUs is a third option. At Lambda’s listed $6.69 per B200 hour, renting breaks even with owning at about 31% for either model, because a rented GPU and an owned one serve the same tokens at the same throughput. The proviso is that every rented hour has to stay productive, and an always-available service with uneven traffic would have idle hours to account for.

Here, utilization means the work delivered over the month as a share of that assumed token capacity. The more of it you use, the more tokens that fixed bill spreads over. For GLM-5.3 on one server, the owned token is $1.05 per million at 5% and $0.10 at 50%. Those are scenarios to compare with your demand, not predictions of what your business will achieve.

Two things follow. The smaller model is the one worth owning, not because it is better, but because the same hardware serves five times the tokens with it and its maker prices the cache hit at five times what the owned token costs. The flagship is the one to rent by the token. Moonshot’s cached price is $0.30, close to the $0.27 that serving Kimi K3 costs at full utilization on your own machines, before colocation, staff and spares.

The machine does not have to run at full capacity around the clock to cross the API price in this example. It does need enough work, at the required quality and speed, to pay for the capacity you committed to. If demand falls, the hardware still has to be paid for. If it outgrows the machine, you need more capacity or a longer queue.

That is why I would begin with the service and let the workload make the case for ownership. Measure what an accepted result costs, how much work arrives and how much of it can wait. If those measurements support the full cost of running the system, there is an economic case to examine. If the business requires local control regardless of price, the same measurements tell you what that control will cost.

Sources and measurement

Nothing here is a hardware benchmark run by Zero One Labs. Every input and calculation is in the dataset, generated by scripts/prepare-onprem-dataset.mjs.

• Throughput. SemiAnalysis InferenceX on B200, the AgentX agentic coding workload. Kimi K3: 13 configurations from 29 July to 28 August 2026, Dynamo vLLM in FP4, sixteen physical chips. GLM-5.3: 5 configurations on 2 September, SGLang in FP4, eight chips, in a sweep that stopped at 16 concurrent clients, so its 30, 50 and 75 rungs are unmeasured. The batching figure takes the best configuration at each per-user speed. The cost calculation takes the 100 tokens a second point, the slowest both models reach.

• Workload. AgentX v1.0, 393 replayed Claude Code sessions, median 142k input and 444 output tokens a request. Input is 98% cache hits in the full set and 95% in the 256k variant. Which variant each run replayed is not printed on the run page.

• Tokens. Blended prices and capacity count input and output together, cache hits included, on both sides of every comparison.

• Servers. One reseller’s listed starting price for an eight-GPU HGX B200 system, retrieved 8 September. 36 months of depreciation, no residual value. Kimi K3 is priced at two of them, the smallest footprint its measured configuration fits.

• Power. Nvidia’s stated maximum draw for a DGX B200, applied for all 730 hours of the month, at the EIA’s June 2026 US industrial average.

• API. Z.ai’s GLM-5.3 and Moonshot’s Kimi K3 list prices, retrieved 9 September. The crossover prices every input token as a cache hit; the prose also gives the fully uncached bound.

• Rental. Lambda’s saved hourly rate at the same assumed throughput, every rented hour productive. Startup time and idle or reserve capacity would raise it.

• Images. The batching and crossover figures are under /research/on-prem/, with SVG originals and a manifest recording the dataset hash.

Ultimately, what matters is how many total tokens the model spends, the speed it can output tokens, and how much each token costs. Completion comes first. Iterate until the model nails the task every time. Then the goal is faster, cheaper.

Available models on the market have a different consumption of tokens per task. They all behave varyingly, but most are able. And the intelligence benchmarks used to score the models are not always the right mark to gauge a model against. Because the workload you're applying the model to will be different than an agentic math test. In essence, desired model behavior and performance is something that is discovered and engineered rather than selected for.

Same score, three bills

The benchmark for DeepSWE shows the difference between two frontier models, and how they behave on "long-horizon engineering tasks". Datacurve wrote the 113 tasks from scratch across 91 open-source repositories, so no model has seen the solutions, and grades the committed code in a clean environment. When inspecting the leaderboard as of 3 September, we recognise a stark difference between Opus and Astra. On the same tests, Opus uses 3.9X the amount of tokens to complete the tasks (Opus spends 118k vs Astra 30k). When looking at the average cost of these two models specifically, Opus costs $11.84, compared to Astra at $6.52.

There is a third model tied at 74%. Gemini 3.8 Flash, released the day before the snapshot, spends 143k tokens per task, more than either, and costs $2.36. The most token-hungry model on the top edge is the cheapest by a factor of five.

• 1 GPT-6 Astra (xhigh). 74.1% pass@1, 30k output tokens, $6.52 per task, 29 steps.

• 2 Gemini 3.8 Flash (high). 73.8% pass@1, 143k output tokens, $2.36 per task, 166 steps.

• 3 Claude Opus 5 (max). 73.6% pass@1, 118k output tokens, $11.84 per task, 99 steps.

Clearly models behave very differently and have completely different cost profiles, even while completing the same tasks at identical scores.

• 1 GPT-6 Astra (xhigh). 74.1% pass@1, 30k output tokens, $6.52 per task, 29 steps.

• 2 Gemini 3.8 Flash (high). 73.8% pass@1, 143k output tokens, $2.36 per task, 166 steps.

• 3 Claude Opus 5 (max). 73.6% pass@1, 118k output tokens, $11.84 per task, 99 steps.

Why is that?

Token efficiency. Partly.

Output tokens are a minority of the bill. At list prices, Opus's 118k output tokens cost $2.95 of its $11.84. Astra's 30k cost $1.50 of $6.52. Gemini's 143k cost $0.54 of $2.36. Roughly three quarters of every bill is input, the context re-read on every step of the agent loop. Astra takes 29 steps to Opus's 99 and Gemini's 166. Steps times context is the lever. Those are estimates, because Datacurve has not published its cost method and its cache handling has been questioned, but the split holds at any plausible cache treatment.

The compute a provider chooses not to bill

The model API hides the reasoning chain from view, and bills it anyway. So the compute that can be hidden from the invoice is the compute the provider chooses not to bill. And there is a possibility worth considering here: What if a model provider deliberately chose to take on some of the cost of reasoning? Inference itself is very profitable, which makes it possible for the provider to subsidize some of the cost. When reviewing the larger model performance game, such a choice becomes rational.

SemiAnalysis wrote in December 2024 that "Gross margins on frontier models are north of 70%, but on trailing models with open source competition, margins are below 20%." In January 2026 they put frontier inference gross margins north of 60%, and in June they estimated Opus 4.8 at 85% or more. By August: "Serving inference tokens is unbelievably profitable for the frontier model companies." These are estimates of the margin on serving paid tokens. The reported company numbers are lower. The Information reported in January that Anthropic projected a 40% gross margin on its paid business for 2025 and OpenAI around 46% across all users, free included. Both can be true at once.

As inference gets cheaper, providers gain increasing freedom to hide computation cost without exposing that computation as the product's unit of value.

The reduced cost of inference tokens is driven by much higher tokens per second per GPU, better batching, caching, and newer accelerators. Epoch AI measured the price of reaching a fixed benchmark score falling between 9x and 900x per year. SemiAnalysis's InferenceX measures a rack of GB300s at up to 100x the realized throughput of a strong H100 setup.

That leaves considerable room between the cost of producing intelligence and the price charged for it. A provider does not necessarily have to return every efficiency gain to the user as lower prices. It could instead spend some of that margin on more inference.

Imagine an API call currently costs the provider $0.30 to serve and produces $1.00 in revenue. That is a 70% gross margin.

Now suppose the provider allows the model to reason longer, verify its work, consult an "advisor model" and retry internally. The underlying inference cost rises from $0.30 to $0.45, while the user-facing price remains $1.00.

The provider has deliberately reduced its gross margin from 70% to 55%.

But it has bought something with that margin: a better model.

If the resulting system completes more tasks, performs better on agent benchmarks, appears cheaper per successful task and attracts more developer workloads, sacrificing some margin on the individual inference may be economically rational.

This becomes a game-theoretic decision rather than a simple cost-plus pricing decision.

Game-theoretic means the payoff of a choice depends on what the other players choose. Cost-plus pricing looks inward: cost, plus a margin. A game-theoretic price looks across the table: what a rival will charge, what a developer will switch for, what a benchmark position is worth next quarter. Game theory is the study of how agents choose when each outcome depends on everyone's choices.

OpenAI, Anthropic and other frontier providers are not necessarily optimizing the profitability of a single token. They are competing for workloads, developers, benchmark position and ultimately the economic layer through which machine intelligence is purchased.

In that game, there is no reason to assume that every unit of computation performed internally must be reflected proportionally in the price visible to the user. A provider with sufficiently high inference margins could effectively subsidize reasoning.

There is no evidence here that OpenAI is currently doing this at material scale. It is a thought experiment. But the economics make the strategy possible.

The exit from tokens

This gets considerably more interesting as pricing moves away from tokens. OpenAI's CFO, Sarah Friar, wrote in January: "As intelligence moves into scientific research, drug discovery, energy systems, and financial modeling, new economic models will emerge. Licensing, IP-based agreements, and outcome-based pricing will share in the value created. That is how the internet evolved. Intelligence will follow the same path."

Forbes read the post as OpenAI calling time on the industry's experimental phase, and put the pricing logic plainly: "Think of it like a consulting firm that takes a percentage of the savings it generates, rather than billing hourly. If an AI agent helps a pharmaceutical company discover a new drug faster, OpenAI wants a cut of that value." In July the company's scorecard post said "the lowest price per token does not always produce the lowest cost per outcome" and proposed "Useful Intelligence per Dollar" as the number that matters. In the same post it advertises GPT-5.6 Sol using 54% fewer output tokens than a rival. Token efficiency as marketing, and outcome pricing as the exit from tokens.

If the provider is paid for completing the task rather than for every token generated, spending additional internal compute becomes worthwhile whenever the increase in success probability is worth more than the additional inference cost.

The optimization target shifts from:

How cheaply can we generate a token?

to:

How much compute should we spend to produce the most profitable successful outcome?

That creates a strange consequence for model evaluation. And sets us up for a future where two possible outcomes might happen: models increase in cost as the strategy to "get you hooked" (lower initial cost) flattens out, and the provider changes strategy to profit maximization. Or we get a completely different incentives model built around outcomes.

A model that appears to consume fewer tokens may genuinely be more efficient. But the number exposed by the API may also become an increasingly incomplete proxy for the amount of computation actually spent producing the result.

The leaderboard makes the point on its own. Its cost column moved four times in two months without a single new run: OpenAI cut prices on 30 July and again on 21 August, Google on 13 August, and a DeepSeek cut had been counted twice. Sol dropped from $8.39 to $6.46 per task on the same 60k tokens. Gemini 3.8 Flash's $2.36 doubles on 1 January.

As the economics of intelligence progresses we will see different strategies being tested and iterated. Currently, I look at total output tokens with a grain of salt, and the cost of the model as an impermanent fact.

Sources and measurement

The figures use the DeepSWE v1.1 leaderboard as generated by Datacurve on 3 September 2026 and retrieved on 7 September. Two records of it are saved: the public artifact and the 70 configuration rows embedded in the page. They share one generation time and disagree on cost for 19 configurations, because the page reprices models whose list price changed after their runs and the artifact does not. The chart dataset plots the page's cost, keeps the artifact's figure beside it wherever the two differ, and records both file hashes.

Every point is one model at one reasoning effort, with its mean pass@1 over four whole-benchmark runs, its mean cost per task, its mean output tokens per task and its mean agent steps. Nothing is excluded. The three highlighted configurations are picked by one rule: every configuration whose pass rate rounds to the same whole point as the leader's, which is the tie the leaderboard itself prints.

Only the GPT-6 Astra rows break the bill into cache reads, cache writes, reasoning tokens and compute units, and only they state a price basis. The dataset script checks that the stated rates applied to the stated token counts reproduce the reported cost to the cent, which they do only when reasoning tokens are counted inside output tokens. So for Astra the output-token column on the leaderboard includes billed reasoning. The basis DeepSWE states for Astra is above OpenAI's published rates and carries a compute-unit line the pricing page does not list. The dashed mark on each bar is the same tokens at the published rates without that line. Which basis Astra is actually billed at is not settled by anything I could find, so the essay quotes DeepSWE's number and shows both.

All three figures are also available as images: tokens, cost and bill, with an SVG original and a chart-only variant beside each, and a manifest recording the dataset hash.

Astra. Fable. Grok. The names sound expensive. The engineering question is whether the extra intelligence earns its bill.

I think of the labs as players in a repeated game, with two strategies worth watching. Crown chasers push for the highest capability they can demonstrate. Their flagship has to do something the competition cannot. Price breakers attack the cost of capable models. Their threat is a model good enough to make an expensive incumbent uncomfortable.

These are competitive strategies. One lab can play both. A flagship can chase the crown while its little brother attacks the bill.

When the little brother makes a task cheaper, the flagship has to defend its premium. A lab might cut the price, push capability further (like loosening guardrails to widen access to capabilities), or win on something the cheaper model cannot do. Or win on more reliable tool use or lower latency. Every move changes what the next player has to beat.

My bet is that this pressure is a powerful driver of progress. After reviewing the data on the Frontier class and the Pareto class, I can observe capability at the Pareto class that did not exist in March of this year. Pareto models are getting better per dollar.

Frontier models and Pareto models

Crown chasers and price breakers describe how the labs compete. Frontier and Pareto describe what we buy. Here is the vocabulary I use.

Frontier model. A model at the leading edge of demonstrated general-purpose capability, at a given date. It earns consideration for the difficult work that weaker models cannot yet do reliably.

Pareto model. The efficient model choice for a specified workload. Among the configurations evaluated, it meets the quality, reliability, and response-time requirements at the lowest total cost.

The frontier is a moving capability boundary. The Frontier Model Forum uses the term broadly for the current state of the art. I look for evidence across difficult reasoning, engineering, and tool use. A model can keep every capability it has today and lose its frontier position when a rival advances. The unlock with a frontier model is the degree to which it can challenge or improve a technical decision, or contribute an insight for the greater whole. Call it higher-order thinking: reasoning through the consequences of a decision and navigating the branches of possibility.

Our Pareto definition adds a selection rule to Pareto efficiency. Pareto efficiency identifies tradeoffs: improving one objective means giving up another. Our workload supplies the quality bar and the evaluation meter. Evaluations establish which configurations clear the bar. Among those, we choose the lowest total cost, and retries and human repair count in that bill.

I run such selections by starting at the frontier and testing downwards. The strongest model establishes what a good result looks like. Each cheaper model then tests how well I have specified the work and equipped the agent.

A failure at a cheaper model can mean two things. The model is too weak, or the workload is underspecified: vague instructions, missing context, a brittle environment. A frontier model tends to reason its way around those defects, at a cost, and hides them. A cheaper model exposes them. Fixing what it exposes improves the whole system, so testing downwards is a quality step as well as a cost step.

These labels can overlap. A frontier model may also be Pareto-efficient, and may be our cheapest reliable choice, all depending on the nature of the task. Model family, size, and open or closed weights do not settle that question. Neither does a low token price. The comparison belongs to a configuration doing a job. Anthropic's model-selection guide makes the same practical case for testing capability, speed, cost, and reasoning effort against actual work. Anthropic presents two recommended paths: efficiency-first, or capability-first. I believe that starting capability-first, then progressing downwards, is the more generally applicable choice. It lets you find the specific definition of the system more closely than starting bottom up.

The good news is that the number of models delivering near-frontier intelligence at Pareto costs is growing. Comparing early September with March shows how quickly the truly competent Pareto-class candidates have multiplied.

March: the challenger box is empty

Take the Artificial Analysis leaderboard captured on 6 March 2026. Among 138 configurations with a measured score and usable cost, Gemini 3.1 Pro Preview held the highest score.

The challenger box sets two conditions for taking on the leader:

• Score: at least 80% of the leader’s score.

• Cost: at most 20% of the leader’s bill for running the same index.

Meet both, and a configuration is an 80/20 challenger. Near the leading score. A fraction of the bill.

In March, nobody qualified.

• 1 Gemini 3.1 Pro Preview. Leading score. 57.2 index, $892.28 index cost.

• 2 Gemini 3 Flash Preview (Reasoning). 81.2% of the leading score, 31.19% of the leader’s cost. Cheapest efficient configuration at 80% of the leader.

• 3 MiMo-V2-Flash (Feb 2026). 72.5% of the leading score, 7.65% of the leader’s cost. Cheapest efficient configuration at 70% of the leader.

The staircase is the Pareto frontier: configurations for which no other option scores at least as well for less, or better for the same cost. March had efficient choices. None cleared our 80/20 threshold. A place on the staircase and a place in the challenger box are separate qualifications.

September: 29 challengers

In the saved 4 September snapshot, Claude Fable 5.1 at max effort held the highest score. Apply the same relative rule and 29 of 129 eligible configurations sit inside the challenger box.

Different reasoning efforts count as separate configurations. Each of these 29 challengers clears the same score floor and cost ceiling. The leader and the evaluation mix changed between the two snapshots, so this compares each market against its own leader. It does not measure the price decline for a fixed level of capability.

• 1 Claude Fable 5.1 (max). Leading score. 65.7 index, $8,523.16 index cost.

• 2 GLM-5.3-Flash. 87.5% of the leading score, 1.62% of the leader’s cost. Cheapest efficient configuration at 80% of the leader.

• 3 GPT-5.6 Luna (high). 71.5% of the leading score, 0.66% of the leader’s cost. Cheapest efficient configuration at 70% of the leader.

Three points on the staircase show the whole argument. Claude Fable 5.1 at max effort is the frontier. GLM-5.3-Flash is the challenger: 87.5% of the leading score for 1.6% of the leader’s index cost. GPT-5.6 Luna at high effort sits at the bottom of the staircase, the cheapest configuration that keeps at least 70% of the leading score, at 0.66% of the cost. All three are Pareto-efficient. Which one is the Pareto model depends on where the workload sets its bar.

GLM-5.3-Flash makes the challenger concrete, and the team behind it has been building toward this for years. Z.ai, formerly Zhipu AI, grew out of Tsinghua University. Nathan Lambert’s history of GLM at Interconnects traces the company’s founding in 2019, the first GLM research in 2021, and ChatGLM in 2023. His reading of the current generation is that it is a post-training story, not a distillation story: the same base model, with more environments and more compute spent training on them. This is a sustained research program showing up in the price breakers’ game.

GLM-5.3-Flash has 320 billion parameters, with 18 billion active per token, and combines sparse and linear attention. It takes text, images, and video as input. Those are specific architectural choices behind a model with an unusually small benchmark bill. They earn it a place in the challenger box. Whether it becomes my Pareto model depends on the work it can deliver.

The task sets the budget

Fable is the reliable engineer in my setup. When designing a new system, working through an unfamiliar codebase, or deciding what to build, I want to maximize the intelligence I can get. A better model might find the simpler architecture, catch a broken assumption, or save me a day on the wrong problem. That is specifically what the frontier premium gets me. The opportunity cost of building a worse system far outweighs the cost of the maximum intelligence that might prevent it.

Opportunity cost is the value of the best alternative we give up. In model selection, that might be a better design, an hour of engineering time, or budget we could put to work elsewhere.

On the contrary, if the model is part of a system, the calculus changes. Inside an engineered system, the model works with constraints and specific instructions. The model receives a contract to follow, and novelty is a defect. Inside such a system, evaluations are run on the model at scale to reach a probabilistic conclusion that the model is suitably instructed, with errors prevented through retry loops, for example. Once a cheaper model meets that contract reliably, the costly flagship has little to no justified presence.

My default is Frontier for discovery, Pareto for iterated delivery. Pay for judgement while the problem is open. Once the problem is defined and a cheaper model is proven, settle for the cheapest option that meets the bar.

What comes after the price breakers?

Over the next year, I expect frontier models to take on longer design and engineering jobs from an incomplete brief, and to become more able to correct their own course and explore decision branches. For Pareto models, I expect greater speed and a lower price per task.

These are roles in a system. The same lab, and sometimes the same model at different reasoning settings, may serve both.

Working forecast, September 2026. I expect frontier models to improve judgement and decision making, and Pareto models to expand the work we can afford to repeat.

Recent pricing changes show why neither class will have one simple price:

• Anthropic estimates that Fable 5.1 will cost 25% less than Fable 5 on typical token-billed workloads because of cheaper cache reads, with savings up to approximately 45% for highly agentic work. Fable 5.1 announcement.

• Claude Fable 5.1 through Anthropic's Message Batches API costs $5 per million input tokens and $25 per million output tokens. Exactly 50% of its standard $10 and $50 rates. Requests run asynchronously and can take up to 24 hours. Fable 5.1 pricing, Message Batches API.

• DeepSeek announced peak and off-peak billing effective 16 August 2026, with off-peak rates half the peak rates. DeepSeek pricing update.

• OpenAI lists Sol’s promotional pricing as available at least through 21 November 2026. Google Cloud lists introductory standard global Gemini 3.7 Flash rates of $0.75 input and $3.75 output per million tokens through December, with $1.50 and $7.50 scheduled for 1 January 2027. OpenAI pricing, Google Cloud pricing.

• GPT-6 Astra prices context length and urgency separately. Standard rates are $10 per million input tokens and $50 per million output tokens. A request with more than 272,000 input tokens is billed at $20 and $75 for the whole request. Batch and Flex cost half the applicable standard rates. Fast mode costs twice them. Astra pricing.

I expect model selection to become more tightly tied to the type of task the model is applied to. Reusable context, a flexible delivery window, or the end of a promotion can change the cost of a task. A stronger frontier model can earn a larger bill if it saves equivalent human work.

Model selection can thus be consolidated into creation versus iteration. A Frontier model is the creator, the Pareto model the iterator.

Sources and measurement

The figures use the 6 March leaderboard capture and the Artificial Analysis leaderboard retrieved on 4 September 2026. These are fixed observations, not a live feed.

The chart dataset, March observations, September observations, and pricing source notes are available as JSON. The chart dataset records source URLs, file hashes, and inclusion counts.

The definition and batch-pricing notes record the additional sources checked through Exa on 6 September 2026. The publication source notes cover the Fable 5 redeployment, the GLM history and architecture, and the Astra pricing. The workload definitions are our editorial choices. The batch rates are Anthropic's published prices.

Both figures are also available as images, with the three highlighted points labelled: March as PNG or SVG, September as PNG or SVG. The highlights follow one rule on both dates: the leader, then the cheapest efficient configuration keeping 80% of its score, then the cheapest keeping 70%.

Each point is a non-deprecated configuration with a non-estimated score and a positive recorded cost for running that snapshot's full index. We exclude configurations without those measurements. The reference is the highest-scoring eligible configuration, with lower cost breaking ties. Both plots use the same relative axes.

The score percentage is a fraction of a benchmark index, not a fraction of intelligence. Cost includes the evaluation's token usage at the recorded prices. It is not a quote for a production workload. See the Artificial Analysis methodology.

Two September costs were calculated from Artificial Analysis token counts and external prices: Muse Spark 1.3 (max) from Meta, and Nex-N2-Pro from OpenRouter. Their saved records identify that calculation. Motif 3 and K2 Horizon 375B remain excluded because the saved work did not establish a public price.

Entopic is the deep research layer for hiring, built by Zero One Labs LLC. It helps founders, recruiters, and hiring managers turn a hiring brief into a researched, evidence-backed shortlist—without reducing people to keywords or an opaque match score.

Entopic turns the brief into an explicit specification you can inspect and correct before research begins. It then maps the relevant talent market broadly, researches the strongest candidates deeply, and cites the source behind every material claim.

The process begins with the Researcher, a recruiting assistant tuned to ask the questions that make a hiring brief testable. Instead of acting on a low-context prompt, it helps you clarify what the role requires, what is flexible, and what—if anything—is non-negotiable. You approve the resulting specification before the research starts.

Once the specification is locked, Entopic maps the market before narrowing it. Candidates are assessed against the same stated requirements, and the strongest are researched in depth. The distinction is simple: a claim is supported by public evidence, merely plausible, or not established. Unknowns are shown rather than hidden.

Entopic does not make the hiring decision. You review the evidence, decide who belongs on your shortlist, and choose whom to contact. Entopic does the research, you make the call.

The research appears in Entopic’s talent map, where you can see how the market was explored and open each candidate’s cited profile. Every material claim sits beside its source, along with a dated career timeline and questions worth testing in the first conversation.

The Researcher also helps you work through the completed research. You can use it to compare priorities, surface specific expertise, and decide what to examine next. For example:

• Find people with a specific skill set within a large research.

• Build an informed list of the first people to contact.

• Discuss which candidates warrant a first conversation—and why.

You can try Entopic today, with a introductionary usage that allows you to run Research without any up front costs. Try it now at Entopic.ai

What if the app operated on your behalf instead?

For a long time, a simple idea has been forming in my mind. What if the application has its own email address, and you can just forward it files, messages, and instructions for whatever you need done? This is entirely possible with standard tools that are easy to set up and consistently reliable.

The simplest version of an agent is an email

The word “agent” has accumulated a lot of noise. Strip it back to first principles and an agent is just a system that receives intent, decides what to do, and acts. Email fits that definition perfectly. You express intent (the message), attach context (the files), and the system does the rest.

Give your application its own email address. Then users can forward files, attach instructions, and send requests to that address. The system receives, authenticates, processes, and completes the work in the background. The user never logs in.

I built this for a commission management platform that insurance firms use. Every month, firms receive new commission data files from their providers. These files carry inherent parsing complexity: new agents, changed rates, structural inconsistencies between providers. Historically, a user would log in, upload the file, wait for parsing, review errors, and manually resolve each one. The entire workflow required the user to sit inside the app.

Now they forward the file to an email address and move on with their day.

The system parses the files through production Python parsers, validates the data against existing records, flags anomalies, and stores everything. If the agent needs clarification (an ambiguous agent name, a rate that doesn’t match any known structure) it replies to the email thread and asks. The user responds when they have a moment. The loop continues until the work is complete.

The entire interaction model is: send email, receive confirmation or follow-up question, done. This is identical in shape to how you work with a human colleague. The difference is that this colleague is perpetually available, never forgets context, and processes files in minutes rather than hours.

The architecture is four ordinary components

The infrastructure behind this is deliberately boring. Nothing here is novel. That is the point: the value is in the design pattern, not the technology.

First, an email domain. This can be your app’s existing domain or a simple subdomain. It receives inbound messages and routes them to a worker for processing.

Second, a worker. This is the first thing that touches an inbound email. It authenticates the sender against an allowed-senders table (one row per authorized address, with metadata for audit trails). If the sender is not recognized, the email is dropped. No ambiguity. If authenticated, the worker extracts attachments, parses the message body for instructions, and routes everything to the correct storage and processing pipeline.

Third, a backend API. This runs deterministic processing: the production parsers that handle commission files, validation logic, data storage. No LLM in this layer. These are the same parsers that would run if the user uploaded the file through the app’s front end. The trigger is different; the processing is identical.

Fourth, a sandboxed agent environment. Once deterministic processing completes, the API triggers an LLM agent in a sandboxed environment. This agent handles anything requiring judgment: interpreting what changed in this month’s file compared to last month, configuring new rates for a new period, responding to freeform instructions the user included in their email. The sandbox is synchronous with the API call, meaning the output is streamed back and verified before anything is committed. The timeout is generous (thirteen minutes) because rigor matters more than speed here.

The agent can send follow-up questions back through the email thread. This creates a genuinely asynchronous collaboration loop. The user is not waiting on a loading spinner. They sent an email, and they will get a response when the work is done or when a question needs answering. Just like working with a person.

Reason from first principles about where your interface lives

Every app has one central function. One reason someone uses it, the core value proposition of your tool. The question worth asking before you build any interface: does that function require a custom screen, or could it travel through a channel the user already inhabits?

This is where most software gets it wrong by default. We build login pages, dashboards, navigation bars, settings screens. We build these because that is what apps look like. But “what apps look like” is convention, not principle. The presentation layer should be derived from the nature of the core function and the habits of the user, not from what other apps happen to do.

Two valid design paths exist.

The first: your tool is delivered through a bespoke interface that genuinely increases the quality of the experience. A design tool needs a canvas. A code editor needs syntax highlighting and a file tree. A mapping application needs a map. In these cases, the interface is inseparable from the value. The custom UI is not overhead; it is central to the product.

The second: you conform as closely as possible to the user’s existing workflows. If the core function reduces to “receive input, process, return output,” then the interface is a delivery mechanism, not a product. Here, the highest-leverage move is to meet the user where they already are.

Both are correct. The nature of your core function decides which path is natural. But most apps default to the first path without ever seriously considering the second. They build a full interface because that is what you do when you build an app. The result is that users must learn and operate a custom tool for a function that could have been an email.

The test is straightforward. Does your user prefer giving text instructions over clicking buttons? Do they already communicate the relevant information through email or messaging? Is the core value in the processing, not in the interaction with a screen? If the answer to any of these is yes, wrapping your core function behind an email address is not a limitation. It is a higher-leverage interface than anything you would build custom.

The real value is background processing

The deeper point is not about email as a protocol. It is about when work happens relative to the user’s attention.

In a traditional app, processing is synchronous with the user’s session. You upload, you wait, you review, you fix, you upload again. Your attention is locked to the app for the duration of the work. The app cannot do anything without you present.

With an email-triggered system (or any async system), the entire processing pipeline runs in the background before the user ever considers opening the app. For commission files with inherent parsing complexity, this matters enormously. The validation, anomaly detection, and initial resolution all happen between the moment the user hits send and the moment they next check their inbox. By the time they look, the work is either done or a specific question is waiting for them.

This is the difference between operating a tool and delegating to a teammate. You do not sit next to a colleague and watch them process a spreadsheet. You hand it to them, explain what you need, and check back later. The email-based agent design creates exactly this dynamic.

This generalizes beyond commission files

The pattern is simple enough to apply anywhere the core function is receive, process, return.

Consider a research assistant. You forward an email with a topic and constraints. The system runs deep research: searching, reading, synthesizing. Hours later, a finished report arrives in your inbox as a PDF or a link to a hosted page. You never opened an app. You sent an email and received a deliverable.

Consider an accounting tool. You forward receipts throughout the month, one email at a time or batched. At month end, you send a message: close the books for March. The system categorizes, reconciles, and produces a summary. The entire interaction happened inside your inbox.

Consider a document review system. You forward a contract. The agent reads it against your standard terms, flags deviations, and replies with a structured summary of what needs attention. You respond with approvals or questions. The negotiation prep happens through a channel you check thirty times a day anyway.

In each case, the same structure holds. The email is the trigger. The backend does the work. The inbox is the interface. The user never has to learn a new tool, navigate a new layout, or remember a new login.

Email is the most underexploited interface in software

Email is a powerful and completely generalizable channel that every professional already uses. Instead of fragmenting services into custom interfaces, consider whether your core value could travel through a channel your users already live in.

The strongest argument for this design is not technical. It is behavioral. You are not asking users to change how they work. You are not asking them to adopt a new tool, learn a new interface, or build a new habit. You are inserting your service into a workflow that already runs every single day. The friction drops to near zero because the interaction model is one the user has practiced for decades.

This is what making a tool feel like a teammate actually means. Not a chatbot inside your app. Not a sidebar with a language model. A system that receives work through the same channel you use to communicate with every other person in your professional life, and delivers results the same way.

Always reason from first principles when you build a user interface. Too many of the interfaces you and I encounter are conventional without being thoughtful. The presentation is what makes or breaks the experience. If your app’s core function can travel through email, give it an email address.

But letting an agent have a full connection to your database is dangerous for obvious reasons. Agents can accidentally alter data without permission. So the challenge is: how do you give an LLM the full read picture while air-gapping it from accidental writes?

The answer is two connection strings.

All Commission Data Through One Agent

I built a platform for managing commission distribution at insurance firms. Sales, employees, agents, products, all in one Postgres database. Users manage commission flows from carrier to sales agent. The platform handles a dense stream of data: commission rates, agent information, performance analytics.

Users can navigate the UI manually. But the full capacity unlocks when an LLM agent works on behalf of the user. A task like migrating a commission rate across every configured agent, painful in a spreadsheet, trivial for the agent.

The system needs three foundational pieces to make this work.

Schema, Strings, and a Map

First, a schema that fits the application. Different apps need different schemas. An ecommerce store is fundamentally different from a commission distribution system, so the way you schematize your database is a contained problem for your specific app.

Second, two connection strings. The first is a read-only replica. This is the string the LLM uses for all queries that are not approved edits. The second is a standard connection string with write access, gated behind explicit human approval. This separation is the core of the design. It guarantees the agent can never run edits without approval, no matter what it generates.

Third, a continuously updated plain-text artifact of the schema, a markdown document describing every table, column, and relationship. This is what allows the LLM to instantly understand your specific database and write correct SQL against it. Without it, the LLM makes dumb errors.

Query Without Risk

Here’s how a query flows through the system. The user asks a question in a chat interface:

“What does sales amount to in January?”

The agent receives the question, references the schema artifact, and writes a SQL command:

psql "$SANDBOX_DB_URL" -c "SELECT SUM(amount) FROM sales WHERE date >= '2024-01-01' AND date < '2024-02-01';"

The agent executes that command against the read-only connection. The database returns:

    sum    
-----------
 452839.50
(1 row)

The agent then writes the response in plain language:

“Sales in January totaled $452,839.50.”

The full loop (receive question, write SQL, execute, return answer) takes roughly ten seconds. Each step is one LLM call. The agent receives the user question and writes the SQL. The database result triggers a second call, now containing the returned data. The agent then decides: retry with a different query, or write the final reply.

Writes Only on Approval

Now for the dangerous part: edits.

When a user asks for an action that requires modifying data, the agent runs through the same loop but against the read-only string. The system identifies when the generated SQL contains a write operation (INSERT, UPDATE, DELETE), blocks it from executing, and surfaces the proposed SQL in the user interface for explicit approval.

Only on human approval does the system switch to the standard connection string and execute the write.

This is a simple design, but it’s the most effective one I’ve found. The read-only replica makes it physically impossible for the agent to alter data during its reasoning and exploration. The approval gate on the write string means edits only happen with informed human consent. Two connection strings, one architectural guarantee: the agent cannot break what it hasn’t been allowed to touch.

Restriction Makes Agents Dumb

LLMs are probabilistic machines. They can generate completely random, potentially harmful SQL. The instinct is to restrict their access: limit what tables they can see, constrain what queries they can write. But restriction makes the agent dumber. It loses context. It can’t answer questions about data it can’t see.

The better path is full read access with an air-gapped write path. Let the agent see everything. Let it explore, join, aggregate, analyze. Then gate the single dangerous action, writing, behind a human checkpoint. You get the full intelligence of the agent without the risk.

In this article, I walk through a specific data flow from the insurance distribution business to present a framework for thinking about how data becomes useful. The goal is to make the example concrete enough that you can infer the key concepts and apply them to your own work.

Data Stuck in Spreadsheets

Inside the insurance distribution business, there is a vast information flow where sales, products, and commissions are transported through multiple steps. This data is communicated and shared inside spreadsheets, which are inherently difficult to work with in bulk.

The problem, stated simply: get the data out of the spreadsheets, clean up the structure, and funnel the data into a structured, queryable database like Postgres. What we end up with is a database that gets hydrated with new data at each period interval — data we can then work with downstream. Finding out who sells the most, what is selling the best, what products have a bad clawback rate, and more.

The goal of cleaning the data into a database is to create a single source of data truth. All data conforms to the same structure, allowing us to use the data stream inside our models to get insights.

An analogy: imagine buying a bag of M&Ms. You expect every piece of candy to conform to a specific shape, weight, and taste. But each piece has a different color. Now apply this to transforming spreadsheets into our database format. We build a transformer that reformats the data into our desired shape and form, but we retain the key value of each data point. We keep its colors intact. When we scale this across tens to hundreds of spreadsheets, this becomes critical. If even one source has been parsed incorrectly, our models lose accuracy. Uniform shape lets us compare the colors.

One Schema for All Sources

For our insurance data flow, I am working with commission spreadsheets. Each spreadsheet has a different structure because each source is a different insurance carrier, each with different data systems. The most important task is to retain high resolution from each data source while transforming it to conform to our uniform schema for commission data.

The transformation is done with a Python backend written specifically for each commission type, which transforms the source to fit our uniform schema. For maximizing convenience, I have built the entire backend in the cloud: when users have new files, they simply forward an email to a custom address (with the spreadsheet files), which triggers the system to validate, transform, and store the data into the database. From there, the next chunk of work can be initiated immediately.

My goal as a systems engineer is to make purely necessary steps — like the data transformation — blend into the background. Make them as unnoticeable as possible so the app experience does not feel like another spreadsheet.

It is helpful to revisit and iterate in circles: how we transform the data, what models we want, what interfaces will be used. Continuously go around again, modifying the way we transform data so that we have the necessary inputs downstream. Some data sources will not meet all requirements for our schema. This is fine. We build dedicated models into the transformation phase to either enrich the data with additional metadata, or create ways of grouping temporal data to conform to our schema.

Inference from Clean Data

Once the data has been transformed and stored, the creative part begins.

A model is how you take stored data from the database and infer something from it. The word infer — to deduce or conclude information from evidence and reasoning rather than from explicit statements — closely aligns with the overarching goal of working with data. We should spend time making sure we have a solid model, so that from there we can infer valuable information.

Each model should tailor to a specific desired output. For the commission data flow, we want to understand the different products that are selling, and quantify them over different temporal spans. Breaking this into a pseudo-spec: I need the product rows, some way of identifying different product types, and temporal metadata about the time interval each record belongs to.

The Interface Layer

The second category is the interface: where our model is presented to the user so it makes sense to them, meeting them where they are so they can understand and make decisions with the data.

An interface is a point where two things meet and interact. The overarching purpose is to be the portal for the user into accessing all of their data maximally. Maximally means the models should be customizable, interactive, and not static.

If the user gets an idea of how they would like to query the data, the interface must meet that demand.

I believe there are currently two key forms of interfaces: natural language, and user-configurable modules. The example interface uses a combination of the two — leveraging the strengths of the modular configurable interface, and enhancing it with the simplicity of natural language. Language is used to orchestrate what you want. The modular interface is used to display it.

Tables That Breathe

The interface is built with a hybrid of the accessible table-like design where content is displayed in a familiar spreadsheet way. Taller row heights and spacious column layouts are conscious decisions to make the spreadsheet visual a friendlier interface compared to a high-density spreadsheet. The overarching goal is to replicate the user’s familiarity with a table while reducing the energy needed for comprehension. A natural next step would be visual representations of the underlying model, perhaps clear visuals comparing and differentiating the data as graphs.

Natural Language Interaction

This is where the power comes in. Instead of the traditional model of a spreadsheet with columns, rows, sorting, and filters, natural language introduces higher resolution of intent. The user explains their desired outcome in their preferred language, rather than conforming to the rules of a spreadsheet.

Speed follows from this. Instead of the human manually configuring filters and rules, the underlying system can figure out user intent and build a model of the desired output at the speed of a hundred words per second. The difference in time spent compared to building the same view in Excel or Google Sheets is stark.

An example: a user might want to see all car insurance products from March to August with commission to agent under $100. Imagine the work of doing this across thirty-plus Excel files. But from the exact line of text — “all car insurance products from March to August with commission to agent under $100” — here, there is plenty of signal inferable for an intelligent system. Our example interface creates the view within seconds from synthesizing the user intent into a model. The user does not need to know any code, only a language.

For practical use, the users can download their view into an exported file PDF or Excel. This allows them to take the data further, into whatever work they are doing. Familiar formats win here because they make the data portable and easily sharable.

How It Works: SQL Under Natural Language

The LLM is the interpretation layer that receives the user intent, reasons over it, then begins an iterative process where it creates — through trial and error — the data model that satisfies the request. The interpreter returns the final data model back to the user interface for instant preview. A complete iteration takes less than thirty seconds.

An LLM most commonly refers to a Large Language Model, a type of Artificial Intelligence (AI) trained on massive amounts of data to understand, generate, and process human language.

Because of our heavy work upstream on clean and structured data, we have a perfectly predictable data source structure, as a result of our transformation, at this point in the pipeline. No matter the number of rows, the system iterates in seconds because of how the architecture is built.

The power here is the power of Postgres and SQL. Our LLM interpreter reads the user intent and writes SQL queries that shape our data model. Think of SQL like a language for prescribing data recipes: you write explicitly how to make a specific data dish. A recipe for making pizza scales no matter how many pizzas you make. SQL follows the same principle: no matter how much data you have, the recipe scales.

PostgreSQL (often shortened to Postgres) is a free, open-source object-relational database management system (ORDBMS). It is widely regarded as one of the most advanced and reliable database systems available.

SQL (Structured Query Language) is the standard programming language used to manage, query, and manipulate data within relational databases. It organizes data into structured tables with rows and columns, functioning as a sophisticated system for handling large volumes of information far beyond the capacity of traditional spreadsheets.

Beyond Commissions

The fundamentally interesting value of this interface is that it enables complete accessibility for any non-technical user to explore vast amounts of data. A few examples:

• Teachers with data about students and grades

• Accountants searching for specific receipts

• Researchers reviewing data quality

• Library inventory databases

• SaaS revenue analytics

• Investor modelling for capital allocation

Most data tools today assume a level of user expertise. I wanted to challenge that assumption. Instead of expecting the user to configure the system, why not let an AI do it? I believe this is the correct vision. The LLM becomes the configurator and the interpreter interface between raw human intent and structured data operations.

This interface is still in its infancy. There are many ways to improve the models, the interface design, and most importantly the accuracy of the LLM agent. At the boundary of any new technology, there is always a wide range of work to be done before the full value can be harvested. That is what makes it worth building.

Reading PDFs is a vital function of any application dealing with documents. Financial documents are commonly stored as PDFs (portable document format) because of the format’s permanent nature and simple storage. PDF is a great format for humans. It has a natural print layout and an easy-to-digest way of structuring text content.

But one of the bottlenecks of PDFs is their difficulty of utilization by computers or intelligent systems such as LLMs. PDFs are best when “looked at”, not when they are parsed by a machine. The analogy is clear: the PDF is for humans, not for computers.

Reality has it so that for more than a decade, PDFs have been the de facto standard for storing key information such as reports, accounting ledger summaries, information transfers, and more. This poses a challenge to anyone building a system that leverages PDF ingestion as part of its components.

In this article, I propose a three-step process for dealing with PDF files, and parsing key information from them, in a high-stakes financial system.

Setting the Scene

I have worked for over a year on a component of a system that deals with financial matters like thousand-plus row files for commission records. A common file size is 107+/- page PDF documents. The requirement is perfect extraction quality of key financial data points, which then get ingested into a downstream calculation component of the overall system.

Through trials and evaluations of parsing techniques, I have landed on a solution which I have complete trust in. Today, I will be sharing this system in depth.

Defining the Inputs and Outputs

In any system, defining the interfaces for each component makes data flow between components a smooth matter. Starting with the parsing requirements for the PDF parsing component is one of the most important decisions you make. Clearly defining the input and output requirements helps you reduce the inherent noise of data. Knowing exactly what you need from a file is the first step.

For this system, the PDF parsing component is required to output the following key data points:

• Agent number -- The identification of the sales agent from whom the parsed sales row is linked to

• Sales id -- Sales row identifier, for example, Sales #12345abc

• Product id -- “Car insurance, #999”

• Commission base -- The base amount, from which commission downstream gets computed with Python

• Commission amount -- Occasionally, the commission is pre-computed in the source file

• Period id -- Crucial identifier required in order to accept a sales row

• Commission rate -- Occasionally the rate is explicit in the source file

Having defined the exact data I need from a PDF, I can go about the business of building the parsing. Knowing exactly what I am after allows me to build a schema into the system that can be validated for each output of the parsing component. This prevents bad data from ever reaching the downstream components. Additionally, I can reject files up-front: if the files are not compatible with the required output (i.e., if the PDF is a news article or something unexpected), the parsing cancels immediately instead of initializing a full “dead” run.

Once I know clearly what I expect from this component, I can progress into designing the rest of the system.

Parsing Signal from Noise

Parsing PDF files is one of the most delightful problems in software systems. Inherently, the task of getting information from a PDF is all about extracting a clean signal (a specific data point) from the random and high-entropy nature of PDFs (unlimited formatting variations possible).

I favour three different parsing methods and services. Let’s walk through each one, keeping in mind our schema and output requirements.

Mistral OCR API

The Mistral OCR API is a fast and reliable OCR machine learning model service where a user can ingest a PDF file of any form and sort, and get a structured output in markdown (plain text format without any formatting or styling) or JSON (JavaScript Object Notation). Both of these output formats are easy to parse and read by a computer, because both get rid of any unnecessary styling, metadata, and formatting, leaving you with only the words, text, and numbers.

The Mistral OCR API is a favourite of mine because of how fast you can go from zero to working system. The service allows large batch uploads and completion within seconds.

Thinking about how the Mistral OCR service fits into the system: the API allows you to parse any PDF, extract tables, get images, and extract the entire PDF contents into the two much cleaner formats mentioned.

From here, the clean and simple output is ready for the next step in the system, which I will talk more about below.

Mathpix

I believe in redundancy as a key property of any system, deeply integrated into the design from the very first initialization. This is why I choose to have an additional OCR-based service in utilization -- not just as a fallback for Mistral failures, but in synergy with Mistral. Mathpix is fundamentally different from Mistral in that it leans more heavily towards maths and tables. This is perfect for a financial system that requires complete accuracy. And, doing the maths on the probability of system failure, adding a second OCR tool exponentially reduces the probability of complete failure.

Mathpix is similar to Mistral in that you can upload a PDF and receive a structured, plain-text version of the PDF contents.

Getting specific: inside the Mathpix API there is a rather “hidden” boolean you can enable: enable_tables_fallback=true. This option enables what Mathpix calls “an advanced table processing algorithm that supports very large and complex tables.” Thinking back to what our schema requirements are, and the types of files our PDF parsing component handles, enabling this option is a no-brainer for our 100+ page table-heavy files.

PDF to PNGs

To further bulletproof the system, I leverage the tool pdftoppm -- a CLI (command line interface) tool that runs locally to convert a PDF file into one PNG image screenshot per page. This is a very high-leverage tool in this system. The two previous parsing services allow us to rapidly extract the text and image content of the input files, but pdftoppm allows us to do further and more rigorous analysis of each page and its original structure.

From the command below, I get one PNG image per page, from where the system can easily review the specifics of each page:

pdftoppm -png -scale-to 1800 input.pdf output

-scale-to 1800 constrains the output image resolution to maximally 1800px on the tallest side, useful for consistency and for downstream API ingestion limitations which we will get to.

Putting It All Together

So far, I have talked about how to get information from a PDF file into a “friendly” format, more simply readable and understood by a computer. Once these parsers have done their job, we have the following pieces to work with:

• Mistral: cleaned markdown or JSON including extracted images

• Mathpix: clean markdown with additional “advanced” table extraction for edge cases and superior coverage for table-heavy files

• pdftoppm: one PNG screenshot per PDF page for deeper visual analysis of the original page structure

Orchestrating the Parsers

With these three parsed versions in hand, we can now orchestrate agents to do the inference of data from the PDF source file. We leverage a visual and text-based LLM model inside a sandbox running in the cloud to do this work, using the two current best tools for the job:

• Claude Agent SDK in Python

• E2B sandbox pre-built templates

Claude Agent SDK is a very powerful tool for processing data and getting a specific output. Once installed, the Claude Agent SDK bundles Claude Code’s claude tool, which is the current best coding and computer agent available. Additionally, using Claude, you instantly get access to the best LLM models on the market.

E2B is a service for spawning ephemeral small virtual machines in the cloud. A virtual machine is like your desktop computer, but on a smaller scale and for a different use case. When using an E2B sandbox, we can allow the Claude Agent to do virtually anything to “its sandbox”, because the only thing that matters is not what the agent does inside the sandbox, only what it produces and outputs from the work done inside of it.

Think of the sandbox as the “workshop” where the LLM has a range of tools (such as the three parsers we have talked about, and Python) and where it can break things, run experiments, and do virtually anything, without ever touching our application production code.

Instructing the Agent

Now that we have laid the foundation of the environment, we can begin instructing the Claude Agent on what we desire it to do. Here, we revisit the schema we defined earlier -- agent number, sales id, product id, commission base, commission amount, period id, and commission rate. These are the exact fields we need from any given PDF file, and they form the foundation of the instructions we give to the Claude Agent in the form of a formal document inside the sandbox -- the agent’s “work requirements.” This formal markdown (.md) document is similar to a document you hand to a co-worker when you need them to do a particular thing. The fewer questions the Agent might have about your document, the better the document is.

Think of this as the specific instructions covering:

1. How the agent will work. Exactly what steps it should take.

2. How the agent will create the output. Exactly how the desired output should look (use the schema rigorously).

3. What are common issues and things to know? How does the agent deal with failure? (i.e., if the file is not a financial document, or if a part of the system breaks)

These three points form the basis of a document which will be iterated on countless times through tests until the Claude Agent produces deterministically quality outputs. The Claude Agent is an LLM AI model, which means that the more specific and helpful you as the creator can be through the instructions, the more flawlessly the Agent can do the work for you.

Let me now outline how the Agent will do the work. I will not bore you with the full instructions, as these get long and specific for the system design requirements. By nature, your system will be of differing design, requiring a different set of instructions. But the following outline creates a high-quality, predictable set of examples you can build on.

Introduce how the component (the agent + sandbox) fits into the system:

• “You are reading a pre-parsed PDF file containing financial information.”

• “You are a component part of a larger system processing commission sales records using Python for calculation.”

• “Your task is to produce high-quality JSON extractions of a PDF file.”

• “The JSON file you write is used downstream inside the next system component to store and do Python calculations on. Doing any calculation with the data is NOT your task.”

Introduce the system environment:

• “You have three directories, each containing different parsing versions from the same file.”

• “Use all three directories to produce the final JSON parsing of the sales rows containing complete verbatim-JSON-transformed extractions from the source file.”

• “Use all three parsing versions in order to cross-verify the parsing of the file by processing each page in logical, self-determined chunks. Using all three parsing versions gives you a completely verifiable way of working, which is perfectly necessary for the complex financial data you are working with.”

Define the expected output requirements and where it should be produced:

• “You will be validated using programmatic JSON validation that the final JSON file you write is valid. Therefore, follow the assigned JSON schema rigorously.”

• “If you realize that the PDF source file will not be compatible with the assigned JSON schema, you simply escape and fail instantly. Failure is accepted if the file is clearly incompatible -- in fact, failure is desirable in such a scenario.”

• “All final JSON outputs must be written to a file called output.json inside the output directory in your file environment.”

Explain how to work and the minimum amount of verification until confidence can be produced:

• “Given the three versions of parsing, you should use all three versions iteratively to verify your inference of the data. This allows you to keep the data quality high.”

How to deal with large samples:

• “By running the tool estimate_tokens.py you can instantly get a recommendation as to how you can work with the assigned files. If the tool output is over the red line, you must spawn a team of agents and assign work specifically, with perfect overlap, and create rigorous validation for the final end product written to the output directory.”

• “Each agent writes their own JSON files, from where you do the merging once ALL data has been processed. If a part of the data fails, or you get stuck somewhere, you must fail quickly instead of trying to fix a massive run.”

How to deal with failure gracefully:

• “Simple failures are tolerated and should simply be ignored. For example, a tool use failure or similar -- if this happens, simply write your own tools on demand.”

• “Be flexible with the approach, but rigid in the output.”

Specific first steps:

• “Start first with checking the size of the assigned work. This will inform whether you can do the work alone, or if you need to spawn an agent team.”

• “Always prefer creating an agent team, versus using sub-agents, when it comes to specifically dealing with writing outputs and processing files.”

Making It Work

The above system is a result of trying and experimenting with countless approaches to the problem of dealing with PDFs. I believe this system produces a predictable and satisfactory solution.

In integrating this into the overall system, I have had success processing high-risk financial information like commission data with accuracy. Dealing with, and handling the risks of, inaccurate or false data extraction is my worst nightmare. Hence, this system design aims to eliminate such risks through cross-verification, frontier LLM models, and validations.

Because of the nature of PDFs, it is impossible to create a perfectly deterministic solution when one does not know what the PDFs will look like. The system above is designed specifically to solve the requirement of being able to read and process any incoming PDF containing financial data. Once tested to a sufficient degree, and error-corrected through iterative design as per your system requirements, this system delivers a satisfactory result.