How I Cut My OpenAI API Bill by 71% Without Changing the Product

The workload: a support-answering feature making roughly 40,000 calls per month, each with a long system prompt and a retrieved knowledge chunk.

1. Prompt caching — 34% of the savings

Related: Midjourney vs DALL-E vs Flux: A Practical Image-Tool Comparison Framework →

Related: Claude 4.5 vs GPT-5.5: Which AI Coding Model Wins in 2026? →

My system prompt was 1,900 tokens and it was identical on every single call. I was paying full input price for it 40,000 times a month.

The fix is structural, not clever: put everything static at the beginning of the prompt and everything variable at the end. Cached input tokens bill at a fraction of the normal rate on every major provider now, but caching only triggers on a stable prefix.

messages = [
  {"role": "system", "content": STATIC_INSTRUCTIONS},   # never changes
  {"role": "system", "content": STATIC_EXAMPLES},        # never changes
  {"role": "user", "content": f"{retrieved_context}\n\nQ: {question}"},  # varies
]

I had the examples after the user question originally. Moving two blocks saved a third of the bill.

2. Model routing — 26%

Related: 9 Free AI Coding Tools Every Developer Should Try in 2026 →

Related: How to Use ChatGPT Effectively in 2026: 7 Prompt Patterns That Actually Work →

Not every request needs the flagship model. I classified requests with a cheap first pass:

def route(question, context_len):
    if len(question) < 120 and context_len < 800:
        return "small"      # ~90% cheaper
    if needs_reasoning(question):
        return "flagship"
    return "mid"

62% of my traffic turned out to be short factual lookups that the small model answered identically. I verified this by running 300 requests through both and diffing the answers — 291 were equivalent, 9 were better on the flagship, and those 9 all contained multi-step reasoning.

Do the diff before you trust the routing. Assuming which requests are "easy" without measuring is how quality quietly drops.

3. Capping output tokens — 14%

Related: Sora 2 Review: OpenAI's Video Model Is Finally Useful for Real Work →

Related: Google Veo 3 vs Sora 2: The AI Video Generator That Wins in 2026 →

Output tokens cost several times more than input tokens, and models pad when you let them. I set max_tokens to a realistic ceiling and added one line to the prompt: "Answer in at most 120 words unless asked for detail." Average response went from 310 to 148 tokens with no complaints from users.

4. Caching whole answers — 11%

Related: How to Clone Your Voice with AI in 2026 (Free and Paid Options) →

Related: ChatGPT Plus for Free in 2026: What's Actually Legit →

18% of my questions were literal repeats or near-repeats. A normalised hash of (question + context id) as a Redis key with a 7-day TTL caught most of them:

key = hashlib.sha256(f"{norm(q)}|{ctx_id}".encode()).hexdigest()
if hit := redis.get(key): return hit
ans = call_model(...)
redis.setex(key, 604800, ans)

For semantic near-duplicates, an embedding cache with a 0.95 similarity threshold caught another slice — but be careful, a threshold that is too loose returns confidently wrong answers to slightly different questions. I tested down to 0.90 and had to walk it back.

5. Trimming retrieved context — 9%

Related: Apple Sues OpenAI, Alleges Ex-Engineer Stole ChatGPT Trade Secrets →

I was sending 8 retrieved chunks. Testing showed 4 gave the same answer quality on 95% of questions. Retrieval quality beats retrieval quantity: improving the ranking let me halve the context.

6. Batch processing for non-urgent work — 4%

Related: OpenAI Ships GPT-5.6, GPT-Live and ChatGPT Work in Coordinated Enterprise Push →

Nightly summarisation and tagging jobs do not need a response in two seconds. Batch endpoints run at roughly half price with a delayed completion window. Moving two background jobs took an afternoon.

7. Streaming and early cancellation — 2%

When a user navigates away mid-response, abort the request. Small, but free.

What did not work

• Aggressive prompt compression. I shortened instructions by 40% and quality dropped on edge cases. The saving was trivial because those tokens were cached anyway.
• Switching to the cheapest provider available. Two of them had latency and rate-limit behaviour that cost more in engineering time than the savings.
• Fine-tuning a small model. Genuinely promising, but the break-even for my volume was somewhere past 300k calls a month. Below that, routing is better value.

The measurement setup that made this possible

You cannot optimise what you do not log. Per request I store: model, input tokens, cached input tokens, output tokens, latency, route decision, cache hit. One table, one daily rollup query:

select model, count(*) calls,
       sum(input_tokens) in_tok, sum(cached_tokens) cached,
       sum(output_tokens) out_tok
from llm_calls where created_at > now() - interval '1 day'
group by model order by out_tok desc;

The first time I ran this I found a debug endpoint in staging making 4,000 calls a day against the flagship model. That alone was $40/month.

FAQ

Does caching hurt answer quality? Prompt caching does not change output at all — it is a billing mechanism. Answer caching can serve a stale answer, so scope your TTL to how fast your data changes.

Is routing worth it under $100/month? Probably not. Under that, spend the time on the debug-traffic audit instead — it is the highest ratio of savings to effort.

How do I know cached tokens are actually being counted? The API response usage object reports cached input tokens. If it stays at zero, your prefix is not stable — check for a timestamp or a UUID sneaking into your system prompt. That was my bug for two days.