Cost Control for LLM Apps: Caching, Batching and Smaller Models

It is easy to build a demo that costs pennies and then discover that the production version costs thousands. Language-model pricing is usually based on tokens: you pay for the text you send and the text you receive. As usage grows, small inefficiencies become big bills. The good news is that most apps can reduce costs substantially without hurting quality.

Understand where the money goes

Before optimising, measure. For each feature, track:

Number of requests
Input tokens and output tokens per request
Model used and price per token
Cost per user, per conversation, per task
Cache hits and retries

A typical surprise: a "simple" chatbot sends the full conversation history, a long system prompt and retrieved documents on every turn, so input tokens dominate costs. Output tokens are often priced higher per token, so verbose answers also matter.

1. Choose the right model for each job

Not every task needs the largest model.

Use small, fast models for classification, routing, extraction and simple Q&A.
Reserve larger models for complex reasoning, long documents and high-stakes writing.
Model routing: a cheap first step decides which model should handle the request.
Cascade: try the small model first; if confidence is low or validation fails, escalate to the larger one.

Evaluate each model on your own test set. You may find the smaller model is good enough for 80 percent of requests.

def route(task: str, text: str) -> str:
    """Pick a model tier based on the task and size."""
    if task in {"classify", "extract", "language_detect"}:
        return "small"
    if len(text) > 8000 or task in {"legal_review", "code_refactor"}:
        return "large"
    return "medium"

MODELS = {"small": "claude-haiku-5-5", "medium": "claude-sonnet-5-5", "large": "claude-opus-5-5"}
print(MODELS[route("classify", "Where is my parcel?")])

Always check current model names and prices in your provider's documentation.

2. Shorten prompts

Remove unnecessary instructions and repeated examples.
Use concise wording; replace long paragraphs with short bullet rules.
Send only relevant context. Retrieve the top few chunks rather than whole documents.
Truncate or summarise long conversation history.
Strip boilerplate such as email signatures, HTML tags and repeated headers from inputs.

Test that shorter prompts keep quality, since cutting too much can hurt.

3. Control output length

Set a sensible max_tokens.
Ask for brevity: "Answer in at most 3 sentences."
Request structured output (JSON with short fields) instead of long prose when machines consume it.
Use stop sequences to end generation as soon as the answer is complete.

4. Cache aggressively

Response caching

If the same or very similar request repeats, reuse the previous answer. FAQ-style bots often see many repeated questions.

import hashlib
import json

cache = {}      # use Redis or a database in production, with expiry times

def cache_key(model: str, prompt: str, params: dict) -> str:
    raw = json.dumps({"m": model, "p": prompt, "x": params}, sort_keys=True)
    return hashlib.sha256(raw.encode()).hexdigest()

def cached_call(model, prompt, call_fn, **params):
    key = cache_key(model, prompt, params)
    if key in cache:
        return cache[key], True          # hit
    result = call_fn(model, prompt, **params)
    cache[key] = result
    return result, False

Be careful to cache only what is safe: avoid caching personalised or sensitive answers across users, and set expiry so stale information is refreshed.

Semantic caching

Match new questions to previous ones by embedding similarity, which catches paraphrases such as "What are your hours?" and "When do you open?". Set a conservative similarity threshold to avoid wrong matches.

Provider prompt caching

Some model providers offer prompt caching, which makes repeated long prefixes (system prompt, reference documents) cheaper and faster. Structure prompts so that the stable part comes first and the variable part comes last, to benefit from it. Check each provider's rules.

5. Batch and schedule work

Batch APIs: if results are not needed immediately (nightly summaries, bulk classification, content generation), use batch endpoints, which are often cheaper than real-time calls.
Combine small items into one request where possible, for example classify 20 messages in one prompt instead of 20 calls, while checking accuracy.
Run heavy jobs off-peak and rate-limit to prevent spikes.

6. Use retrieval instead of stuffing

Rather than putting an entire knowledge base in every prompt, store it in a search index and retrieve only the passages needed. This reduces input tokens and often improves accuracy.

7. Add limits and guardrails

Per-user and per-day quotas
Maximum message length
Maximum tool-call loops for agents
Timeouts and retry limits with backoff (uncontrolled retries multiply costs)
Authentication to prevent anonymous abuse
Budget alerts and automatic shutoff for runaway spending

8. Avoid unnecessary calls

Use rules or regular expressions for simple tasks such as detecting phone numbers or yes/no replies.
Validate input before calling the model (empty messages, spam).
Reuse results across steps instead of re-asking.
Do not call the model for every keystroke; debounce user input.

9. Consider fine-tuning or open models for high volume

For a stable, repetitive task at large scale, a small fine-tuned model or a self-hosted open model can be cheaper per request than a large general API model, although you must include engineering and hardware costs. Compare total cost of ownership, not only token prices.

10. Measure the impact

Establish a baseline, change one thing at a time and track cost per task and quality metrics together. Cost savings that harm quality can cost more through lost customers.

A quick estimator helps with planning:

def monthly_cost(requests_per_day, in_tokens, out_tokens, price_in_per_m, price_out_per_m, days=30):
    daily = requests_per_day * (in_tokens * price_in_per_m + out_tokens * price_out_per_m) / 1_000_000
    return daily * days

# Example with made-up prices: replace with your provider's real prices
before = monthly_cost(5000, 3000, 400, price_in_per_m=3.0, price_out_per_m=15.0)
after = monthly_cost(5000, 1200, 250, price_in_per_m=3.0, price_out_per_m=15.0)
print(f"Before: ${before:,.0f}  After: ${after:,.0f}  Saved: {(1 - after / before):.0%}")

A quick optimisation checklist

[ ] Do we log tokens and cost per feature?
[ ] Are we using the smallest model that meets quality needs?
[ ] Are prompts trimmed, and history summarised?
[ ] Is output length limited?
[ ] Do we cache repeated requests safely?
[ ] Do we use batch processing for non-urgent jobs?
[ ] Do we use retrieval rather than full documents?
[ ] Are there quotas, timeouts and alerts?

Frequently Asked Questions

What is the fastest way to cut costs?

Measure first, then reduce prompt length and add caching. These often deliver big savings with little risk.

Will smaller models reduce quality?

For simple tasks, often not noticeably. Test on your own data and use routing for harder cases.

Is caching safe?

Yes if you avoid sharing personalised or sensitive responses and expire entries appropriately.

How do I forecast costs?

Estimate requests, average tokens and prices, add a safety margin, and compare with actual logs weekly during the first months.

Conclusion

LLM costs are manageable when you measure usage, choose models wisely, shorten prompts, cache, batch, retrieve and set limits. Treat cost as a design constraint from the start, and keep quality metrics beside it.