An employee asks, "How many vacation days do I have?"
The question is short. The model's input may not be. The optimization opportunity is not simply choosing a cheaper model — it is deciding what the model actually needs to read.
Consider an HR assistant that retrieves policy text on every request, even a balance lookup. Its illustrative input budget is:
- 10 tokens: employee question.
- 2,690 tokens: instructions, conversation history, and tool definitions.
- 3,000 tokens: retrieved policy excerpts.
- 300 tokens: authorized HR data and metadata.
That totals 6,000 input tokens, followed by a 500-token output.
Routing comes before chunking.
Retrieval-augmented generation, or RAG, searches a document collection and supplies selected passages to the model. Instead of sending the entire HR handbook, the application retrieves relevant sections.
But a current vacation balance comes from an authorized HR system, not the handbook.
Routing means choosing the workflow a question needs before assembling the model's context. A rule or classifier can direct a request to a system lookup, document retrieval, or both.
A balance-only request can use an authorized API call and a concise response format. Carryover, eligibility, or exception questions need current policy evidence.
Routing should remove unnecessary retrieval, not necessary verification. Authorization remains enforced by the application and source systems; the model must not invent balances or policies from memory.
Three ways teams improved context.
The next question is how to improve the context a workflow genuinely needs.
| Engineering example | What changed | What the evidence supports |
|---|---|---|
| Dropbox Dash: simplify retrieval | Consolidated app-specific retrieval tools into one universal search tool; used knowledge-graph ranking and a specialized search agent. | Dropbox reports clearer reasoning and more focused context. No controlled token- or cost-saving percentage is published. The search agent's model calls must also be counted. |
| Databricks: preserve structure | Used syntax-tree-based chunking with file paths and class/function hierarchy metadata instead of arbitrary text boundaries. | Fully correct answers rose from 59% to 70% on 46 model-judged questions. This was a code-assistant demo, not a production-wide result or cost measurement. |
| Spotify: curate knowledge | Created domain-specific clusters of datasets, approved question-SQL examples, and business definitions. | Experts accepted only 12.5% of proposed historical-query-derived examples. That measures curation, not answer accuracy; the candidate count was undisclosed. The system generates SQL, not document-based answers. |
Spotify's result is less about retrieval volume than context quality. Historical examples can be technically valid yet teach the wrong pattern, so domain owners should curate which examples, definitions, and schemas enter the prompt.
The shared lesson: available information is not automatically useful context.
What the research suggests.
| Research | Finding | Design implication |
|---|---|---|
| Adaptive-RAG | In its GPT-3.5 evaluation, adaptive routing versus always multi-step retrieval used 1.03 versus 2.81 average retrieval steps, with F1 scores of 50.91 versus 50.87. | Match effort to evidence needs. These are open-domain QA results, not HR savings. Its no-retrieval option is not a substitute for authoritative HR data or policy evidence. |
| Searching for Best Practices in RAG | In one chunk-size experiment, moving from 1,024- to 512-token chunks increased reported faithfulness from 94.26 to 97.59; 256-token chunks scored 97.22. | Smaller is not always better. This experiment used about 170 generated questions from one financial report and model-based judging. Faithfulness measures consistency with retrieved text, not verified correctness. |
| Sufficient Context | Stronger models often answered incorrectly rather than abstaining when retrieved evidence was insufficient. | Check whether the evidence can support an answer. Missing evidence calls for retrieval, clarification, or escalation, not merely a shorter prompt. This is a reliability finding, not a token-saving estimate. |
F1 measures answer overlap with a reference answer. These studies use different metrics and settings; their scores are not directly comparable.
Two questions, two optimization paths.
For a balance-only question, removing the unnecessary 3,000-token policy block reduces input tokens from 6,000 to 3,000: a 50% input-token reduction.
For a carryover or eligibility question, policy retrieval remains necessary. Halving that block saves 1,500 tokens, reducing input tokens from 6,000 to 4,500: a 25% input-token reduction.
Both scenarios hold the other input components fixed and exclude routing overhead. A model-based router may add a call, so evaluate the full workflow — not only the final prompt.
At 10,000 questions a month, the balance-only scenario would remove 30 million monthly input tokens (3,000 × 10,000). The policy scenario would remove 15 million (1,500 × 10,000). Each assumes every question follows that path; a mixed workload would depend on the share of each.
These are illustrative scenarios, not measured savings or equivalent cost reductions. Output, cached-input rates, retrieval services, and additional model calls affect the bill.
Measure the outcome, not the prompt.
Filter documents by permissions and applicable policy version. Keep rules with their exceptions and effective dates. Remove duplicate passages and unnecessary API fields.
Test each change against the same reference questions, keeping model versions and other settings fixed except for the component being tested. Track correctness, citation validity, permission violations, refusal behavior, end-to-end latency, billable input/output tokens, cache hit rate, and cost per successfully resolved, permission-safe request.
Include routing, retrieval, reranking, compression, and every model call. Reject any change that weakens permission boundaries or increases incorrect-answer risk.
The FinOps goal is not fewer tokens at any cost. It is less unnecessary context per correct, permission-safe answer.