Select what matters. Compress selectively. Reuse what is stable. Expand when the outcome justifies it.
The goal is not the lowest cost per token. It is the lowest cost per accepted result.
A million-token context window can cost more and deliver less than retrieving 20,000 relevant tokens. With enough reuse, that larger context can also become the cheaper option.
Token count alone cannot tell you which strategy wins.
Million-token windows are usually presented as a capability milestone: more information, more possibilities. But capacity is not the same as value. A larger window expands what a system can process. It does not determine what the task needs.
Context is an allocation problem, much like capital allocation.
Tokens are not productivity. Context is not value. Like financial capital, context should be allocated where its expected contribution to the outcome justifies its cost.
That changes the question from "How much context can we fit?" to "Which combination of retrieval, compression, caching and broader context delivers an accepted result at the lowest fully loaded cost?"
The best context strategy is the lowest-cost combination that meets the task's quality, latency and risk requirements.
Sometimes that means retrieving a few passages. Sometimes it means reusing a large, stable knowledge pack. Sometimes the task genuinely requires a whole document or repository.
The point is deliberate allocation, not small context at any cost.
Content is what you have. Context is what you use.
Content is the information available to a system: documents, code, policies, conversations and other records.
Context is what the model receives for a particular inference. That includes instructions, the question, conversation history, selected documents, tool definitions and tool results.
The context window is the model's working capacity, measured in tokens. How input, reasoning and output share that capacity depends on the model.
Consider a support agent with access to thousands of product documents. A customer asks whether a particular subscription includes audit logs.
The content is the entire documentation collection. The necessary context may be a product plan, an entitlement table and a policy exception.
A practical flow looks like this:
Available content → retrieve and rank → filter or compress → assemble context → generate → validate
Reusing a stable prefix can reduce the cost of repeated processing. A broader request can be used when the selected evidence is insufficient.
This makes context an allocation decision, not a container to fill.
More context creates more work — but not through one simple cost curve
A model first processes the supplied input, a stage called prefill, and then generates its response through decoding.
With standard full self-attention, the number of token-to-token attention interactions grows approximately with the square of sequence length.
Twice the context therefore means roughly four times the attention-interaction work. Ten times the context means roughly one hundred times.
| Context length | Relative context length | Relative attention-interaction work |
|---|---|---|
| 100,000 tokens | 1× | 1× |
| 200,000 tokens | 2× | 4× |
| 1,000,000 tokens | 10× | 100× |
Illustrative scaling for the token-interaction component of standard full attention — not a prediction of total computation, latency or API charges.
Other prefill operations, including feed-forward layers, scale closer to linearly with sequence length. Model architecture, optimized kernels, sparse attention and caching also change the practical cost.
The distinction matters: quadratic attention does not mean a quadratic customer bill.
API charges follow provider pricing rules. Those rules can introduce their own boundaries.
All linked web sources were accessed October 18, 2026. The following are selected API pricing examples; rates are USD per million uncached input tokens. Provider prices and availability can change.
| Model | Prompt-length boundary | Rate at or below boundary | Rate above boundary |
|---|---|---|---|
| Gemini 3.1 Pro Preview | 200,000 tokens | $2.00 | $4.00 |
| GPT-6.1 Sol | 272,000 tokens | $2.00 | $4.00 |
| Claude Haiku 5.5 | 100,000 tokens | $0.10 | $0.50 |
Gemini 3.1 Pro Preview's output pricing, including thinking tokens, also rises when the prompt exceeds 200,000 tokens: from $12 to $18 per million output tokens.
Threshold rules are provider-specific. For Gemini 3.1 Pro Preview and GPT-6.1 Sol, crossing the stated input-token boundary applies the applicable long-context rates to the entire request, not merely the excess tokens.
Not every model has such a surcharge. Claude Sonnet 5.5 provides its full one-million-token window at standard rates.
A lower-priced tier is therefore a budget boundary, not a target to fill. The strategic implication is simple: know the boundary you are approaching before paying to cross it.
Three levers, three different effects
Context optimization is often discussed as if every technique does the same thing. It does not.
| Lever | What it changes | Main trade-off |
|---|---|---|
| Select and compress | Reduces the material supplied through retrieval, filtering, reranking or selective summarization | Can omit evidence or remove important detail |
| Reuse | Reduces repeated processing costs through prompt caching | Depends on stable prefixes, reuse and provider rules; cached tokens still occupy the window |
| Expand | Supplies broader documents, history or evidence when the task requires it | Can increase cost, latency and exposure to distracting material |
Prompt caching reuses previously computed intermediate state for a matching prefix. It is not simply returning a saved answer.
A useful arrangement is:
Stable instructions → stable reference material → changing question
But caching activation, eligibility and retention rules differ by provider and model.
Automatic caching also does not necessarily mean cheaper. On some OpenAI models, including GPT-6.1 Sol, cache writes cost 1.25 times ordinary input, even if the prefix is never reused. That write rate replaces the ordinary input rate for those tokens; it is not an additional charge of 1.25 times on top.
Compression needs the same economic discipline. Rewriting a stable prefix can break cache reuse, so fewer tokens do not automatically produce a lower bill.
Caching lowers the cost of reuse. It does not remove the model's technical context-window limit. Cached tokens still occupy capacity and can count toward pricing thresholds — as Anthropic explicitly specifies for Haiku 5.5.
One example: when a larger context becomes cheaper
Retrieval is not always the lowest-cost option.
Imagine a team asking repeated questions about a stable, 100,000-token knowledge pack. It can either cache the full pack or retrieve 20,000 relevant tokens for each question.
Using Claude Sonnet 5.5, assume these rates per million tokens:
- Ordinary input: $2.00
- Five-minute cache write: $2.50
- Cache read: $0.10
- Output: $10.00
Both approaches add 2,000 variable input tokens and produce 1,000 billed output tokens per question. Retrieval also receives an illustrative $0.01 per-question allocation for indexing, embedding and retrieval.
Assume both approaches meet the same acceptance criteria. The cached approach makes one initial write, and every later question fully hits the same prefix. Retrieved evidence varies between questions and receives no caching discount.
The cache requires a continuing cadence of use. Each subsequent question must start within five minutes of the previous cache-writing or cache-reading request. A hit refreshes that five-minute lifetime; a gap of five minutes or more can require another paid write.
Under those assumptions:
| Approach | First question | Each subsequent question |
|---|---|---|
| Cached 100,000-token pack | $0.264 | $0.024 |
| Retrieve 20,000 tokens per question | $0.064 | $0.064 |
The larger context costs more initially. Reuse changes the result.
| Questions answered | Cached full pack: cumulative cost | Retrieval: cumulative cost |
|---|---|---|
| 1 | $0.264 | $0.064 |
| 6 | $0.384 | $0.384 |
| 7 | $0.408 | $0.448 |
| 10 | $0.480 | $0.640 |
Caching ties retrieval at the sixth question and becomes cheaper from the seventh.
This is an illustration, not a universal crossover. Cache misses, updates, different output lengths and different acceptance rates can change the result.
The retrieval allocation assumes an operating volume over which upfront indexing and embedding costs are spread; at low volume, retrieval's fully loaded cost would be higher.
The comparison includes model charges and the stated retrieval allocation. It excludes failed attempts, retries, human review and tool-execution costs. It is therefore a request-cost illustration, not the fully loaded outcome metric used below.
The lesson is not "cache everything." It is that the lowest-cost choice depends on reuse, change frequency, pricing and accepted outcomes, not token count alone.
Research supports conditional allocation, not a single optimal context size
These studies tested models available in 2024 and early 2025. They provide useful architectural evidence, not forecasts of today's model performance.
Retrieve first, escalate selectively. Self-Route routes questions between retrieval and broader context. In arXiv version 1, Table 1, it uses roughly 60% fewer input tokens for Gemini 1.5 Pro and about 39% fewer for GPT-4o, with average benchmark-score changes of −3.29 and +0.22 points, respectively. These are input-token reductions, not measured dollar savings under current caching and pricing rules.
Self-Route qualification. Gemini's average score changes from 49.70 to 46.41, and GPT-4o's from 48.67 to 48.89. The version's results prose reports different performance figures, so these should be read as table-specific results, not a reconciled summary of the paper. They are benchmark-score points, not a single accuracy measure.
Cache stable knowledge when reuse is high. Cache-augmented generation experiments, reported in a 2024 arXiv preprint, used Llama 3.1 8B with approximately 21,000–85,000 reference tokens preloaded into KV caches. The tested setups demonstrated cases where avoiding runtime retrieval improved performance, but they do not establish the economics of current managed APIs.
Evaluate effective context, not advertised capacity. NoLiMa's 2025 ICML evaluation found that 11 of 13 tested models fell below half their short-context performance at 32,000 tokens when literal matching cues were removed. That is evidence for testing whether a model can use the supplied information — not a universal 32,000-token limit.
Together, the findings support a practical approach: select evidence, reuse stable material and test when broader context earns its cost.
What engineering teams actually do
Production accounts show the same emphasis on shaping context rather than indiscriminately expanding it.
Dropbox Dash consolidates and filters access. Dropbox's November 17, 2025 engineering account describes a unified retrieval interface, an index and knowledge graph, and a specialized search agent. The objective is to surface relevant information without making the main agent navigate every source independently.
Manus designs for reuse. In its July 18, 2025 engineering account, Manus reports an average input-to-output token ratio of roughly 100:1 and emphasizes stable prefixes, append-only history and recoverable external files. That is a company-reported operational figure, not a universal agent-design target.
Anthropic improves selection and tool discovery. Its Contextual Retrieval experiments, published September 19, 2024, reduced top-20 retrieval failure from 5.7% to 1.9% by combining contextualized chunks, semantic and keyword retrieval, and reranking. Separately, its advanced-tool-use account reports an 85% token-usage reduction in an illustrated tool-search setup that loads relevant tools on demand instead of supplying every definition upfront.
These are different interventions. Better retrieval improves evidence selection. Stable prefixes improve reuse. Tool discovery reduces unnecessary definitions.
None establishes a universal production savings rate. All demonstrate that context is something to engineer.
Find the strategy, then measure the result
Different workloads suggest different starting points.
| Workload | Starting strategy to compare |
|---|---|
| Repeated questions over a small, stable knowledge pack | Cached full context versus retrieval |
| Large or frequently changing corpus | Retrieval, reranking and selective context |
| Tool-heavy workflows | Consolidated or on-demand tools, with filtered results |
| Long-running agents | Reusable prefixes, recoverable memory and selective compaction |
| Whole-document comparisons or widely dispersed evidence | Broader context, tested against narrower alternatives |
These are starting hypotheses, not automatic rules.
Evaluate them on representative tasks, difficult cases and a separate holdout set. Count tool definitions, history and cached input when budgeting capacity, and reserve the output or reasoning space required by the model.
Most importantly, define an accepted result before optimizing.
For a support agent, acceptance might mean a correctly resolved case. For a coding agent, a validated change. For a review workflow, an assessment that meets an agreed standard.
The principal business metric is:
Cost per accepted result = fully loaded workflow cost ÷ accepted results
Fully loaded cost should include failed attempts, retries, retrieval infrastructure, cache writes, tool execution and human review.
Cost per token remains useful for explaining a bill. It cannot tell you whether the system delivered value.
Nor should the model's confidence alone determine whether the evidence was sufficient. Use task-specific checks, source verification and agreed acceptance criteria.
The FinOps opportunity is to treat context as an investment decision: compare alternatives, account for risk and fund additional processing where it improves the expected outcome enough to justify the cost.
A larger window expands what is possible. It does not determine what is necessary.
Select what matters. Compress selectively. Reuse what is stable. Expand when the outcome justifies it.