In a prototype, nobody watches the bill or the clock. At scale, both decide whether a feature survives: a small cost per request becomes a large monthly line, and a slow answer makes every user wait. Most LLM apps waste tokens and time in the same few places, so the same few levers help almost everywhere.
This is part 5 of Level 4 (Expert), the last part of our AI Development path. You will learn where cost and latency come from, how to measure the cost per successful task, which levers cut tokens and waiting time, and how a routing cascade sends easy requests to small models. Our article on how we reduce AI development costs covers the build phase; this part covers the running cost of every request.
Where cost and time go in one request
APIs bill per token, with one rate for input and usually a higher one for output. Many reasoning models also produce hidden reasoning tokens, which are usually billed as output. Time follows a different logic. Before the first word appears, the model reads the whole input; this phase is called prefill and determines the time to first token (TTFT). Then it writes the answer token by token. The time per output token (TPOT) times the number of output tokens usually makes up most of the total time.
The rule of thumb that follows: input tokens often make up most of the volume, output tokens most of the waiting time. The support bot from Level 0 shows the first half: it reads nine times as many tokens as it writes, mostly the same instructions and help text in every request (see APIs, tokens and context windows).
Measure the cost per successful task
Before you change anything, measure a baseline. The price per token or per request is the wrong unit, because it hides retries, agent loops and failed answers. The useful unit is the cost per successful task: all spending for a feature — model calls, retries, tool calls, judge calls — divided by the number of tasks that met your quality criteria. Track it together with the success rate and the p95 latency, per feature and per model version, and set a minimum success rate that every option must reach.
The example shows two traps. The large model alone reaches the minimum success rate, but it costs the most per success. The small model alone looks cheapest, but it misses the minimum, so its low number does not count. The cascade reaches the minimum for clearly less money. You find such results only with evals and traces: Level 2 shows how to test AI features with evals, and the traces from Level 3 deliver the token counts.
Spend fewer tokens: budgets, compression, shorter answers
Token budgets put hard limits where costs can run away: a maximum number of output tokens per request, a daily budget per user and a maximum number of steps per agent task. An agent that loops is among the most expensive bugs in an LLM app.
Prompt compression removes what the model does not need. Start by hand: delete duplicate instructions, shorten examples, send only the tool definitions of the current route, summarize long chat histories and retrieve fewer but better document chunks. Research tools go further: LLMLingua uses a small model to drop unimportant tokens and reached up to 20-fold compression with little loss of quality on its test tasks (Source: Jiang et al., “LLMLingua”, EMNLP 2023). Any compression can remove the one sentence that mattered, so check it with your eval set.
Shorter answers save cost and time at once. Ask for a fixed format, such as JSON with defined fields or three bullet points, and set the output limit to match. For reasoning models, use the lowest reasoning effort that still passes your evals.
Reuse work and let it wait: caching and batching
Part 1 of this level introduced the three caches. Their effect on the bill differs. A prompt cache at the provider bills a repeated prefix at a lower rate and shortens the time to first token. It pays off when long, stable content comes first: instructions, tool definitions, examples, documents. An exact or semantic cache skips the model call entirely, but only for repeated questions and with the risks described in part 1.
Batching trades time for money. Many jobs do not need an answer within seconds: nightly classification, summaries of yesterday’s tickets, embeddings, eval runs. Several providers, including OpenAI, Anthropic and Google, offer batch interfaces: you submit many requests at once, receive the results within a time window of up to 24 hours and pay less per token than for live requests. On your own GPUs, larger batches raise the throughput, but each request waits longer. The DistServe paper calls the useful measure goodput: the highest request rate a system can serve while it still meets its latency targets (Source: Zhong et al., “DistServe”, OSDI 2024).
Route easy requests to small models
Most providers sell their models in size tiers. A smaller tier answers faster and costs less per token, and for many requests it is good enough. Two families from our LLM database, three tiers each, with the overall capability index (ECI):
| Model | Released | Weights | ECI | Rank |
|---|---|---|---|---|
| Claude Opus 5.5 Anthropic | Sep 22, 2026 | closed | 166.8 | 1 |
| Claude Sonnet 5.5 Anthropic | Sep 28, 2026 | closed | 164.4 | 6 |
| Claude Haiku 5.5 Anthropic | Oct 7, 2026 | closed | 159.0 | 14 |
| GPT-5.4 OpenAI | Mar 5, 2026 | closed | 156.7 | 23 |
| GPT-5.4 Mini OpenAI | Mar 17, 2026 | closed | 148.8 | 64 |
| GPT-5.4 Nano OpenAI | Mar 17, 2026 | closed | 146.1 | 79 |
ECI = Epoch Capabilities Index, one overall score across many benchmarks; rank among all models with an ECI. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.
Model routing decides for each request which model answers. A cascade is the most common form: the small model goes first, a check decides whether its answer is good enough, and only the rest moves up to a larger model.
Research supports the idea. In the FrugalGPT experiments, a learned cascade matched the best single model of the time with up to 98 percent lower cost on the tested tasks (Source: Chen, Zaharia and Zou, “FrugalGPT”, 2023, arxiv.org). RouteLLM trained routers on human preference data, and in some tests they cut costs by more than half without lowering response quality (Source: Ong et al., “RouteLLM”, ICLR 2025).
Routing saves more than money. Fewer tokens on large models also mean less electricity and, through power generation and cooling, less water. That footprint is no longer small: one study estimates that AI systems caused 32.6 to 79.7 million metric tons of CO₂ in 2025 (Source: de Vries-Gao, Patterns, December 2025), and according to the International Energy Agency, electricity demand from data centers is rising four times faster than demand from all other sectors (Source: reported by The Guardian, March 2026).
The check is the weak point. It can be a schema validation, a rule, a confidence score or a small judge model. If it accepts wrong answers, the cascade saves money and silently loses quality. Escalated requests also take longer, because they run through two models. So evaluate each route on its own and watch the escalation rate: when it climbs, your traffic or a model has changed. For a narrow task with high volume, the next step is distillation: you train a small model on the answers of a large one. Level 3 compares this with fine-tuning, RAG and prompting; check the provider’s terms before you train on its outputs.
Shorter waits for users
Latency has its own levers. Streaming shows the first words after the time to first token, so the wait feels shorter, although the total time and the cost stay the same. Independent steps, such as several tool calls or document lookups, can run in parallel instead of one after another. Work that does not depend on the user’s question, such as summaries of known documents, can be computed in advance. And a self-hosted model can use speculative decoding, described in part 2 of this level.
The table shows what each lever changes. Every lever has a quality risk, so every change goes through your evals first:
| Lever | Lowers cost | Lowers latency | Quality risk | Best for |
|---|---|---|---|---|
| Token budgets, output limits | Yes | Yes, shorter outputs | Cut-off answers | Every feature, agent loops |
| Prompt compression | Yes, input | Yes, time to first token | Lost context | Long prompts, RAG, long chats |
| Prompt caching | Yes, repeated input | Yes, time to first token | Low | Long, stable prefixes |
| Exact or semantic cache | Yes | Yes, no model call | Stale or wrong hits | Repeated questions |
| Batching | Yes | No, results come later | Low | Offline jobs and evals |
| Routing and cascades | Yes | Mostly; escalations take longer | Wrong answers accepted | Many easy, few hard requests |
| Streaming | No | Felt waiting time only | Low | Chats and long answers |
| Distillation or self-hosting | Yes, at high volume | Often | Narrower skills, more operations | Narrow, frequent tasks |
Work through the levers in order
Start with the levers that are cheap to try and easy to undo, and move to the heavier ones only when the numbers ask for it:
Change one lever at a time and compare the cost per successful task, the success rate and the p95 latency with the baseline, ideally in a canary release. The infrastructure for the heavier steps is part of our comparison. According to the vendors, Runpod rents GPUs billed by usage for self-hosted or distilled models, Pinecone is a vector database for retrieval and semantic caches, and Netlify hosts the web app and its serverless functions:
Short descriptions from our tool comparison, based on the vendors' information — not tested by ABXK.