How Context and Retries Increase AI Agent Costs
AI agent token costs accumulate across the full execution of a question. The model reads instructions and context during discovery, reads much of them again when it writes a query, and may repeat the process after validation finds a mistake. A request that appears to contain one question and one answer can include several paid model calls underneath.
Supper reduces this repetition by limiting the context used at each stage and carrying known business logic into the analysis. The execution trace below shows where context accumulates, what a retry processes again, and which costs caching can lower.
One answer can contain several model calls
A basic chatbot may receive a prompt and generate a response in one call. An AI analytics agent usually has more work to do.
It may need to identify the relevant data, decide how a metric should be calculated, write a query, inspect the result, and determine whether the result supports an answer. Some systems divide those tasks across separate model calls so each step can use the output of the previous one.
Token usage is therefore cumulative:
Total token usage = input and output tokens across every model call
The visible response is only the final output. Earlier plans, tool requests, query drafts, validation results, and failed attempts also consume tokens.
The context usually grows as the analysis progresses. The discovery stage begins with the user's question and general instructions. The query stage adds selected schemas and business definitions. Validation adds the generated query and its result. A correction may include everything from the earlier steps plus an explanation of what failed.
Each output can become part of the next input.
An annotated AI agent execution trace
Consider this question:
How many customers who churned in Q2 had no product activity during their final 30 days?
Assume the company records churn on the effective contract end date. Product activity means a customer user completed a tracked action in the production application. Automated events and employee support sessions are excluded.
The agent must identify the official Q2 churn population, retrieve product activity for those customers, and apply the correct 30-day window to each account.
| Stage | What the agent does | Context processed by the model | Cost annotation |
|---|---|---|---|
| 1. Discovery | Identifies the subscription data, product-event data, and definitions required for the analysis. | System instructions, user question, available data descriptions, churn definition, and product-activity rules. | This is the first processing of the shared context. The output is a plan naming the likely sources and required calculation. |
| 2. Querying | Writes a query that finds customers who churned in Q2 and checks for product activity during each customer's final 30 days. | The shared context appears again, now joined by the selected schemas and discovery plan. | The model reprocesses the question and business definitions. The generated query adds new output tokens. |
| 3. Validation | Compares the query's churn population with the approved Q2 churn total. The counts disagree. | The model receives the question, definitions, query, returned data, and official control total. | Earlier context returns again. The query result and validation instruction increase the size of this call. |
| 4. Correction | Finds that the query used the cancellation-request date. It changes the cohort and activity window to use the effective contract end date. | The model receives much of the previous context plus the mismatch, date rule, failed query, and correction instructions. | The retry processes old material and adds evidence from the failure. The database query also runs again. |
| 5. Final response | Reports how many Q2 churned customers had no qualifying activity during their final 30 days. | The model receives the original question, validated result, relevant definitions, and any supporting breakdowns. | The final answer creates another call even though most of the analytical work has already occurred. |
The correction is only one step in the table, but it affects more than one cost. The agent revises the date logic, calls the data source again, reviews the new result, and prepares the final explanation from the corrected analysis.
If the second query also fails validation, that sequence can repeat.
Why retries process so much context again
A retry needs enough information to avoid repeating the original mistake. That usually includes the user's question, the definition being applied, the selected data structure, and the failed attempt.
The validation result becomes new context as well. In the example above, the difference between the query population and the approved Q2 churn count shows that the original cohort is wrong. The company's effective contract-end-date rule then becomes part of the correction.
This creates a growing context:
- The discovery call contains the question and general data context.
- The query call carries that material forward and adds a plan.
- The validation call adds SQL and returned data.
- The correction call adds the detected mismatch and revised instructions.
That sequence is intentional. The model needs some history to understand why it is being asked to revise its work. Removing too much context can cause the agent to make the same mistake again.
The cost problem appears when the system carries far more history than the current stage requires. A correction to the churn date may not need every table description considered during discovery. The final response may not need the failed SQL once the corrected result has passed validation.
Context selection determines how much of the earlier work gets paid for again.
How prompt caching changes the economics
Prompt caching lets a model provider reuse processing for an identical or sufficiently stable portion of an earlier input. Cached input is commonly priced below new input, though eligibility and pricing depend on the model provider.
Suppose an agent has 8,000 tokens of stable instructions, schema documentation, and business definitions. It uses that same material across 4 model calls.
Without caching, those calls process 32,000 repeated input tokens at the ordinary input rate. If the first call establishes a cache and the next 3 calls match the same prefix, 24,000 of those repeated tokens may qualify for cached-input pricing.
The model still receives the context. Its marginal price changes.
The variable parts of the execution remain new. The current question, discovery plan, query result, validation finding, and correction all differ by call. Those tokens continue to grow as the analysis proceeds.
Prompt structure affects whether caching works. Stable instructions and definitions should appear in a consistent order near the beginning of the prompt. Adding a timestamp, user-specific value, or changing instruction before the stable content may reduce the reusable prefix.
Caching therefore rewards consistency in how context is assembled. It provides less benefit when every step creates a completely different prompt.
Query and result caching affect other costs
Model tokens are only part of a retry. The agent may also rerun warehouse queries or call another service.
A query cache can reuse a result when the same approved query runs against the same data state. This removes the cost and delay of executing the calculation again. A result cache can make repeated reporting questions cheaper when the answer remains current and the user has permission to see it.
Freshness matters. A cached answer about customers who churned during a closed reporting period may remain valid after the quarter ends. A cached answer about customers showing signs of churn today can become stale as new activity and subscription data arrive.
Permission boundaries matter too. A result created for one user or team should only be reused when the next requester has access to the same underlying data.
Caching a mistake makes the mistake cheaper to reproduce. Cached results need clear expiration rules, and changes to metric definitions or reporting dates should invalidate affected entries.
Context still needs active control
Caching lowers the price of eligible repetition. It cannot prevent an agent from carrying unnecessary context through every stage.
The largest savings come when each call receives the information required for its specific decision. Discovery may need a catalog of possible sources. Query generation needs the selected schema and calculation rules. Validation needs the query, result, and an appropriate comparison. The final response mainly needs the confirmed result and enough supporting detail to explain it.
This stage-specific context keeps prompts smaller and makes failures easier to isolate. When validation finds an error, the system can retry the affected part of the analysis without restarting the entire execution.
It also improves cache behavior. Stable instructions can remain fixed while the smaller variable portion changes with the question and current result.
How Supper controls context and retries
Supper narrows the context before asking the model to reason. Its semantic model gives the agent the relevant business definitions and relationships for the current question, so the prompt does not need to carry every available definition.
Calculations happen close to the data source. The model can receive a compact result instead of thousands of raw records, which reduces the context added during querying and validation.
Supper also evaluates the work as the analysis develops. When a result fails a check, the system can correct the affected stage before the mistake travels into a longer explanation or another dependent calculation.
When a failure reveals a missing business rule, that rule can be added to the shared semantic model. Future questions can begin with the corrected context, reducing the chance that the same retry sequence occurs again.
This is how Supper limits AI agent token costs while preserving the context required for a reliable answer. The model receives what it needs for the current stage, and accepted business logic can be reused across future analyses.
Frequently asked questions
What are AI agent token costs?
AI agent token costs are the model costs generated by every input and output across an agent's execution. They can include calls for planning, query generation, validation, correction, and final response generation.
Why do retries increase AI agent token costs?
A retry often reprocesses the original question, instructions, business definitions, and relevant schema. It also adds the failed attempt and the information explaining why it failed. The new call can therefore contain more context than the original one.
What is prompt caching?
Prompt caching reuses model processing for a stable portion of an earlier input. Eligible cached tokens may be priced below new input tokens. The exact behavior depends on the model and provider.
Does prompt caching eliminate retry costs?
No. New context still requires processing, and a retry may also repeat database queries or other tool calls. Caching reduces the marginal cost of stable repeated input.
Which AI agent context is best suited to caching?
Stable system instructions, business definitions, and schema documentation are strong candidates when they remain unchanged across calls. Current query results and validation errors change during each execution and are less likely to receive the same benefit.
How can companies reduce AI agent token costs?
Companies can limit context to the current stage, keep reusable instructions stable, return compact query results, and validate work before the analysis expands. Corrections to shared business logic should also carry forward to future questions.
How does Supper reduce repeated context?
Supper selects relevant business logic from its semantic model, performs calculations close to the connected data source, and checks the analysis as it develops. This limits the amount of context carried into later calls and reduces repeated failures.