A reference cuts the cost
Data can be an incredible asset to have when engineering an agentic workflow. The engineering question in this post is how to make the contextually right data available when the agent needs it, and what improvements this introduces.
Consider an agent doing a body of work consolidating commission books. It receives a request to investigate X correction in Y period. In this agentic body of work, there is no reference to what exactly the correction should look like. Here, the agent needs a reference data tool. A reference data tool provides context about specific things from a database. In this case, “corrections” is one of the reference tools. The agent can use it to access database records showing what a “correction” looks like in context.
Now consider a person doing the same work, opening a particular spreadsheet, PDF, or dashboard before making the next decision. Knowing where to look is part of knowing how to do the job.
I see reference data in an agentic system in the same way. While doing X task, the agent calls a tool to obtain Y information before making Z decision. The tool queries a source at runtime, so the reference can reflect fresh information available when the work is being done.
The word reference is a specific choice. In audio, a reference track gives an engineer a known basis for comparison. They can compare characteristics of the mix against a recording selected for that purpose. Reference monitoring, signals, and playback levels serve related roles in establishing known listening or measurement conditions. iZotope on reference tracks, Genelec on monitoring calibration
The useful part of that mental model is the relationship between the reference and the judgment. A familiar recording can help me judge a particular aspect of a mix. A rate schedule can help establish the applicable commission. An earlier correction can help me recognize a recurring problem. Each reference is useful because I know what I am consulting it for.
The reference data is the specific data you point your agent towards. Its usefulness depends on that specificity. Giving an agent vague access to all data does not tell it which records answer the question in front of it. A deliberately chosen query can make the relevant source and its meaning much more explicit.
There are three things I want the system to make clear: where to look, what to ask for, and what the result establishes. For the commission task, that means identifying the right person and period, retrieving the applicable rates and transactions, and understanding whether a correction has already been made and whether adding a correction is correct.
This does not require all reference data to be historical. The reference might be the latest status of a transaction, the agreement effective for a particular period, or a set of earlier cases. What matters is that the agent can consult it while doing the work.
The SQL query can remain the same while the data changes. A new correction becomes available through the same lookup. An amended agreement can be returned with the period to which it applies. The stable part is the meaning of the query. The result supplies the changing facts.
The specificity of the agentic work helps determine the reference data scope. When the task has a specific goal, such as making a correction correctly, we can identify the information retrieval needed to make that decision. Through iterative engineering, we can find what the reference dataset should be to make the task accurate, repeatable, and effective.
Engineering the reference query
A reference query is a deliberately designed request for the information a particular agent task needs. It defines the source and the parameters, and helps clarify the meaning of the returned result.
For a commission workflow, the available queries might cover agents (as in sales agents), agent details, agent rates, agent sales, and agent corrections. Broader queries can expose sales and corrections across the curated scope, with filters and bounded results. Here, agent refers to the salesperson in the business records. The AI agent consults those records through the client application.
Each query answers a different question. Which person does the request concern? Which rate applies to this period? Which sales contribute to the amount? What adjustments already exist? Have comparable cases been handled before?
The reference query can be exposed as a tool through the Model Context Protocol, or MCP. The server publishes the tool’s name, description, and input schema. The agent requests it through the client, the server authenticates and runs the query, and the result returns as context for the next step. MCP provides the interface and data, while the application determines the query. MCP tool specification
The engineering can be quite small:
A client tool called
get_agent_ratescould accept a person identifier and a period.The server limits the records to those the caller is allowed to access.
The server returns the relevant rate records immediately, without the model having to guess its way to discovering the right data.
Fundamentally, I view reference data as minimizing the amount of trial and error that the agent has to do to achieve its given goal.
A feature of engineered reference data tools is their ability to deal with inconsistencies in source identifiers, retrieve matching records across periods, and return a bounded result with summaries. I think built-in mechanisms for catching data errors in a SQL reference query do two things: a) stop data errors from reaching the agent, and b) help it immediately understand what ‘data error’ means in the given context when the query is exposed to it. That is recurring work reference queries can perform directly. The agent uses fewer resources and gets additional valuable, hard-to-define context.
Reference data allows us to instill judgment in the tool design.
We tested this by asking an agent to reconcile a commission table against a proprietary insurance commission dataset, kept private from publication. The task was specific: compare the submitted amounts for 50 agreements with the amounts recorded in the database, and return a complete comparison.
For each task, we selected 50 distinct agreement numbers from the sales table for one period. We used their recorded commission totals to construct a Power BI-style table, then artificially changed the submitted amounts for 12 or 13 agreements. Alternating between 12 and 13 gave us a 25% discrepancy rate overall. The agreement numbers and database records stayed unchanged. The agent was not told which amounts we had changed or how many.
We gave the agent two different ways to get the reference data. With get_committed_agreement_totals, it supplied the period and agreement numbers to a tool with prewritten SQL. The tool returned the recorded total for each agreement. With query_database, it wrote its own read-only SQL against the same sales table.
In both conditions, the agent received the same user message, the same plain SQL schema, and the same calculation instructions. The general SQL tool could retrieve all 50 totals in one call. It supported the filtering and aggregation needed for the task, so the agent could decide how to write the query without having to discover the database schema first.
One agreement could have several sales postings in the period, including negative and zero amounts. Separate postings with the same amount still had to be counted separately. The query had to sum all relevant postings before rounding the total to two decimal places. These were the rules we implemented in the reference query, and the rules the agent using general SQL had to implement itself.
Identify the period and agreements. With reference tools, the agent supplies tool arguments. With general SQL, the agent expresses the scope in SQL.
Include the right postings. A prewritten reference query or agent-written SQL.
Aggregate and round. A prewritten reference query or agent-written SQL.
Compare with the submitted table. The agent completes the same structured answer in both conditions.
Judge the answer. The same independent deterministic grader in both conditions.
The reference tool received no submitted amounts and no expected answer. It simply returned the database totals. The agent still had to compare those totals with the user’s table and complete the response.
We specified that response with a Pydantic schema. Each row had to contain the agreement number, submitted amount, recorded amount, difference, and a match or mismatch status. The difference was the submitted amount minus the recorded amount, in NOK. All 50 agreements had to appear exactly once, including those whose amounts matched. Both toolsets also included the same optional tool showing a valid 50-row response with generic values. This kept the schema guidance the same on both sides.
We calculated the correct answer separately in Python, adding up the individual database postings with exact decimal arithmetic. A passing answer had to satisfy the schema and match the expected period, agreements, and values in all 50 rows. The score was fully correct tables divided by attempted tables. One wrong field or a missing answer meant a failed attempt.
We ran 20 held-out tasks, each with a different set of 50 agreements, and repeated each task five times per toolset. That gave us 100 attempts with reference tools and 100 with general SQL. We used GLM 5.3 Flash through Baseten with the same model settings in both conditions. Pydantic AI called the Python tools directly in this experiment. MCP transport was outside the comparison scope given that the tools ran locally.
What we observed
Both toolsets produced 99 fully correct tables out of 100 attempts. With reference tools, the agent used a mean of 2.03 model requests per attempt. With general SQL, it used 3.90. That is about 48% fewer requests with the reference tool. Mean elapsed time per attempt was 42.72 seconds and 86.33 seconds, respectively. A turn in the figure means one request to the model.
Explore the interactive figure · Saved data (JSON) · Sources and measurement
One reference attempt timed out. One general SQL attempt returned a table with one incorrect row. Both counted as failed tables. The timeout also left us without a token-usage receipt for its final request. We recorded $0.42710 in inference cost for reference tools and $0.92795 for general SQL, but the reference total is incomplete. Its exact cost remains unknown.
The turn counts include requests used to recover from rejected SQL calls. After the run, we also checked the result with the affected attempts and their reference counterparts removed. Reference tools still required fewer turns. The rejection counts and that comparison are in the measurement notes.
Both conditions had access to the same proprietary dataset. What we tested was the value of engineering a specific way to use it. The reference query put the filtering, aggregation, and rounding into maintained code, with a tool description and a result shaped for the task.
Keeping the reference current
The operational system already maintains the records. When a new sale or correction is recorded, the same reference query can make it available to the agent. We do not need to keep copying updated values into its instructions.
A read replica can provide this access, with some delay as changes arrive from the primary database. The agent still needs to call the tool again when it needs an update. PostgreSQL hot standby documentation
Current data also needs the right scope. A correction for last month requires the rate that applied last month. The query’s scope and calculation rules still need upkeep when the schema or business rules change.
Looking ahead
I am interested in how much of the reasoning behind earlier work we can make available as a reference. For the commission task, that could mean retrieving a reviewed correction together with the records, agreement passage, and explanation that supported it. The next agent could see why the correction was made and assess whether the same reasoning applies.
This gives the system a way to accumulate useful references through the work itself. A reviewed decision adds more than a final amount. It preserves what was known, what was decided, and why. The reference tool would need to distinguish reviewed decisions from proposals the agent produced but nobody accepted.
Other tasks could draw on different kinds of data. A reference query might retrieve a current document passage, an image from an inspection, or an audio observation. The same design question applies: which part of that source does the agent need, and what can it establish from it?
A reference could also come from a computation performed at runtime. Before recommending a change, the agent could ask a simulation to evaluate it against the current operating state. The response would need to carry the assumptions and conditions behind the result. A prediction describes what a model expects to happen, and the agent needs to preserve that distinction when comparing it with observed records.
The direction I want to explore is how much useful context a system can make available at the moment it is needed. For each body of work, that means choosing the references the decision depends on, making them available through specific queries, and checking that the agent has used them correctly.
Sources and measurement
We ran the study on 25 September 2026 using GLM 5.3 Flash through Baseten, with max reasoning effort and temperature 0.2. Each attempt had a limit of 12 model requests, 16,384 output tokens per request, and 240 seconds. The 200 scored attempts cover 20 held-out tasks, five repetitions, and two toolsets. Each pair used the same submitted table and expected answer, with randomized pair and condition order. Each attempt started a new conversation. The comparison evaluates the complete tool design, including the query, its description, and the response shape.
The task inputs were constructed reconciliation requests using the proprietary insurance commission dataset. We selected the latest valid reporting period with at least 500 distinct nonblank agreement numbers, then used a seeded selection of disjoint groups of 50 from the holdout pool. Development and holdout agreements were separated. The submitted amounts changed, while the source records and agreement numbers remained unchanged. The 25% discrepancy rate was an experimental choice, not an estimate of how often reporting errors occur.
The saved measurements and analysis configuration (JSON) contain aggregate task summaries only. The proprietary dataset remains private, with no source agreement numbers, commission amounts, prompts, or model answers published. Each point in the figure represents one task and toolset, averaged over five attempts. Cost is inference USD per attempt, turns are model requests per attempt, and score is the fraction of complete 50-row tables that passed the independent grader. The turns/score view includes all 40 points. Cost views omit the one reference task with incomplete usage receipts. All scored attempts remain in the condition summaries.
The uncertainty intervals resample whole task groups, keeping both toolsets and all five repetitions together. With 20,000 bootstrap draws and seed 1701, the reference-minus-SQL mean turn difference is −1.87, with a 95% percentile interval of −2.13 to −1.61. The score difference is zero percentage points, with an interval of −3 to +3 percentage points. Equal observed scores do not establish equivalence.
The general SQL tool rejected 13 calls across 12 paired attempts. In a post hoc check, removing both sides of those pairs left 88 attempts per toolset. Mean turns were 2.03 for reference tools and 3.68 for general SQL, about a 45% reduction. Both passed 87 of the remaining 88 tables. This shows how the result changes when those attempts are excluded. It does not tell us how the agent would perform with a different SQL guard. Query text was deliberately not retained, so we cannot reconstruct each rejected query.
Costs use received token-usage receipts and the rates recorded for this run: $0.15 per million uncached input tokens, $0.03 per million cached input tokens, and $0.50 per million output tokens. The reference total is incomplete because the scored timeout has no final usage receipt. An aggregate provider usage check found a zero-token request during the timeout’s idle window but could not link it to that attempt. We therefore left the missing cost unknown. Database compute, tool execution, engineering, and maintenance are outside the cost measure.
One unserved HTTP 429 request was excluded and rerun. Its record remains in the audit trail, separate from the scored timeout. The replacement is included in the 200 scored attempts. Costs reported here cover the scored attempts with available receipts, not the entire provider invoice.
The tools queried a read replica at runtime. Each uninterrupted part of the run used one fixed, read-only database snapshot so the two toolsets would work against the same evidence. Across resumptions, the saved commitments matched the selected source records, submitted prompts, and expected answers in all three snapshots. This checks that the task evidence matched across the interrupted run. It does not establish that the rest of the database stayed unchanged, and the study did not test how agents respond to records changing during a task.
The harness retained no source records, submitted tables, or model answers as a dataset. Its local ledger preserves scores, usage receipts, timings, error categories, and keyed commitments that can be checked without storing the underlying financial records. Those commitments cannot reconstruct an earlier database state or an unsaved answer. Applying a different grader to the original answers would require data we deliberately did not retain.
I really like the audio reference track analogy, it clarifies what reference data should do. The commission example shows how much wasted effort an agent spends when it has to discover the right aggregation logic every time instead of calling a tool that already knows it.
We see something similar in matching projects. If you give an agent a vague "find duplicates" task and raw access to millions of records, it flails. But a narrow tool that retrieves candidate pairs based on blocking keys or returns similarity scores for specific attributes lets the agent focus on judgment, not discovery.
The 48% reduction in turns makes sense. How much of that gain do you think comes from skipping SQL trial-and-error versus having the rounding and aggregation rules baked in?