What a prompt costs: tokens, caching and the bill

In the laboratory of prompt engineering, developers effortlessly send thousands of words of instructions to large-scale language models. System prompts with elaborate persona definitions, dozens of few-shot examples and complete JSON schemas fly across the API lines. As long as an application is in the development phase with dozens of test requests per day, the financial effect is barely noticeable. But as soon as an AI application scales to production and handles tens of thousands of users, the prompt turns from an innocent piece of source code into the largest variable cost item on the infrastructure bill.

Determining the real price of a prompt requires a thorough understanding of the underlying token economy, the asymmetric pricing of AI providers and the technical workings of GPU memory caching. A prompt is not a static sequence of characters; it is a dynamic instruction set that the inference engine has to process on every call. In this handbook we dissect how tokens are counted, why the structure of your system prompt directly determines whether you qualify for hundreds of euros in caching discounts, and how to keep your prompts tightly hydrated without collapsing reasoning quality.

How tokens are counted: the processor under the hood

To understand where the bill comes from, we have to look at how a Large Language Model (LLM) reads text. An LLM does not process raw ASCII or UTF-8 characters, nor does it read whole words. Instead, a subword tokenizer (such as BPE or WordPiece) cuts the input text into numbers corresponding to a fixed vocabulary table. One token equals roughly 0.75 English words or about 4 characters on average.

For Dutch-speaking developers there is a direct financial catch here. Most popular tokenizers from OpenAI, Anthropic and Meta are trained mainly on English-language corpora. Dutch words are far more often split by these algorithms into several separate subword fragments. Where the English word sustainability often counts as one or two tokens, the Dutch translation duurzaamheid can be broken into three to four tokens. This means a Dutch-language system prompt can naturally end up as much as 30 percent more expensive to process than a substantively identical English prompt.

To estimate up front how many tokens a specific instruction set takes up, you can use the prompt token counter to analyze your texts quickly. This tool gives you immediate insight into the ratio between characters and token counts for different tokenizers, so you spot unexpected cost increases during the design phase.

The anatomy of API costs: input vs. output pricing

API providers almost without exception apply a two-part rate structure: cost per million input tokens and cost per million output tokens. Output tokens are structurally 3 to 5 times as expensive as input tokens. This price difference is caused by the hardware reality of autoregressive generation.

When processing the input (the prompt), the GPU can handle all tokens in parallel in a single forward pass through the Transformer layers. This is a prefill phase that makes extremely efficient use of the massive parallel compute power of modern AI accelerators. In the generation phase (the output), however, the model has to compute token by token, sequentially. Every newly generated token requires a full pass through the entire network, with all weights loaded from HBM memory again.

At the infrastructure layer you can monitor consumption according to the guidelines for managing rate limits, tokens and costs. This guide offers concrete handles for setting budget limits and rate limiting at the API level, which prevents runaway output loops from draining your budget.

Prompt caching and prefix caching dissected

The most important innovation in token economics of recent years is prompt caching (also called prefix caching). Because the prefill phase of a long prompt demands a lot of GPU compute, providers store the intermediate computations of the Transformer network — the so-called key-value (KV) cache — in the VRAM of the server cluster.

When a follow-up call contains exactly the same opening text (prefix), the model does not have to run those tokens through all layers again. The provider reads the KV cache straight from memory. This delivers not only an enormous latency gain (often up to 80 percent faster time-to-first-token), but at providers such as Anthropic, OpenAI and Google it also translates into a direct discount of 50 to 90 percent on the price of those specific input tokens.

For an in-depth analysis of the server-side workings of KV caches we refer to the overview on context caching in LLM APIs explained. There you can read exactly how providers hold key-value pairs in the memory of their GPU clusters and which minimum thresholds apply to activate caching.

Architecture for maximum cache hits: the fixed prefix

Many developers unknowingly make the mistake of placing dynamic variables at the top of their prompt, such as the current date, the unique user ID or the user's question. This changes the input from token 1 onward, so a cache hit never occurs.

To benefit optimally from caching, you should build your prompts like a layered pyramid. The most static, heavy components belong right at the top, while the dynamic, fast-changing data moves to the bottom.

The optimal order for prompt construction

1. Static system prompt (top): The general role description, behavioral rules and fixed formatting instructions. This block changes rarely or never.
2. Fixed documents / knowledge base: Large reference works, API documentation or policy terms that stay identical across multiple requests.
3. Few-shot examples: Fixed question-and-answer pairs that demonstrate the desired format.
4. Historical dialogue context: Earlier messages in a multi-turn conversation (where applicable).
5. Dynamic user input & variables (bottom): The user's specific question, including temporary variables such as the date or the user profile.

By cutting your instructions into fixed blocks, the approach in building prompts from reusable components helps keep your prefix predictable. This modular structure ensures that a large part of the prompt stays unchanged, which is crucial for effective caching.

// VOORBEELD: Slechte promptstructuur (Cache miss bij elke aanroep)
{
  "system": "Datum: 2026-08-07. Gebruiker: ID-8942.", // Dynamisch bovenaan = BREEKT CACHE
  "prompt": "[Lange statische systeemprompt van 3000 tokens...]"
}

// VOORBEELD: Optimale promptstructuur (High Cache Hit Ratio)
{
  "system": "[Lange statische systeemprompt van 3000 tokens...]", // Vaste prefix = CACHE HIT
  "user_context": "Datum: 2026-08-07. Gebruiker: ID-8942.",      // Dynamisch onderaan
  "user_query": "Wat zijn de voorwaarden voor retourneren?"
}

Saving tokens without losing quality: finding the limit

Besides caching, simply shortening the total prompt size is the most direct way to push costs down. Recklessly cutting instructions carries major risks, however. Once a prompt is written too sparingly, the model loses essential context, resulting in style breaks, format errors or hallucinations.

When reducing token counts, always keep an eye on performance, as discussed in the article on prompt length versus answer quality in practice. That article shows at what point cutting context starts to damage the accuracy of the output.

Proven techniques for token pruning

• Remove politeness and filler: Sentences such as "Please read the attached text carefully and try to formulate the following answer as accurately as possible" can be reduced without any quality loss to "Analyze the text and answer the question."
• Use compact data structures: Instead of elaborate JSON objects with long key names in your few-shot examples, a tight CSV notation or YAML is often enough, saving up to 40 percent on syntax tokens.
• Prune the few-shot examples: Test whether three excellent, varied examples do not produce exactly the same result as ten overlapping examples.
• Enforce concise output: Because output tokens are the most expensive, you can demand explicitly in the system prompt: "Answer in no more than 2 sentences" or "Return only the JSON array with no introductory text".

Worked examples and scenario analyses from practice

To make the financial impact of token structure and caching tangible, we analyze three common production scenarios. We calculate here with a fictional but representative pricing of € 2.50 per 1M input tokens, € 0.25 per 1M cached input tokens and € 10.00 per 1M output tokens.

Scenario 1: customer service RAG bot

An organization handles 50,000 customer questions per day. Every request contains a static system prompt plus relevant knowledge base articles totaling 4,000 tokens. The user asks a question of 100 tokens and the bot gives an answer of 200 tokens on average.

• Without caching:
  • Input: 50,000 x 4,100 tokens = 205M tokens = € 512.50 per day.
  • Output: 50,000 x 200 tokens = 10M tokens = € 100.00 per day.
  • Total per month (30 days): € 18,375
• With optimized prefix caching (90% cache hit rate on the 4,000 static tokens):
  • Uncached input: 50,000 x (100 user + 400 static miss tokens) = 25M tokens = € 62.50 per day.
  • Cached input: 50,000 x 3,600 cached tokens = 180M tokens = € 45.00 per day.
  • Output: 10M tokens = € 100.00 per day.
  • Total per month (30 days): € 6,225 (a saving of 66%)

Scenario 2: automated code review assistant

For specific applications where a lot of fixed context is supplied, we refer to the guide on writing prompts for code generation. That guide shows how to structure sizeable code bases efficiently within the context window.

When a developer has a pull request of 500 lines of code reviewed, a system prompt of 2,000 tokens (containing all company coding guidelines) is combined with 8,000 tokens of code context. If the developer places the company coding guidelines at the top of the prompt, that 2,000-token prefix can be cached across hundreds of reviews per day, producing a continuous reduction in base costs.

Pitfalls and the downside of token optimization

Attractive as cost savings are, aggressive token optimization and blind trust in caching carry serious risks.

1. Cache invalidation triggers

Provider-specific thresholds can quietly wreck your caching strategy. Many APIs require a prefix to be at least 1,024 tokens long before it qualifies for caching at all. If your system prompt counts 800 tokens, you pay full price every time, unless you deliberately extend the prompt with structured documentation or examples to reach the threshold.

2. Cache eviction at low frequency

A KV cache on a provider's GPU clusters is not kept indefinitely. If no new requests with the same prefix arrive for 5 to 10 minutes, the cache is automatically cleared (eviction) to make room for other users. Caching therefore only really pays off with requests at a constant, high frequency.

3. Over-optimization and loss of nuance

When you prune a system prompt too far to save tokens, the subtle boundary conditions ("edge case handling") often disappear. The result is that the model makes mistakes more often, which leads to follow-up calls, retries or manual corrections by developers. A failed call that has to be rerun costs twice as many tokens as a slightly longer prompt that succeeds the first time.

Conclusion & checklist for cost-efficient prompt design

A professional prompt engineer looks not only at how well a model performs a task, but also at what that processing costs at scale. By building prompts with a strict dividing line between static prefixes and dynamic variables, you transform expensive API calls into efficient, cached operations.

With these principles anchored in your software architecture, your AI application stays scalable and fast — and the monthly API bill stays fully under control.