Prompt compression for long contexts: summarize or prune

Why context compression remains necessary with large context windows

The availability of context windows of 128k to 1M tokens suggests that developers no longer need to worry about the size of their prompts. That is a persistent misconception. A large context window does not guarantee that a model processes all information with equal accuracy. The well-known lost in the middle phenomenon shows that transformer models retrieve information at the start and the end of a long prompt considerably better than facts buried somewhere in the middle.

The compute required for the prefillphase of inference also scales linearly with context length (and quadratically in the pure self-attention layers without optimizations). A prompt of 80,000 tokens causes a noticeable delay in time to first token (TTFT), which makes interactive applications and chatbots feel slow and sluggish. To analyze the real financial impact of large contexts, the article on what a long context window really costs provides a detailed economic calculation.

Finally, needless context pollution leads to hallucination and instruction drift. When a prompt contains thousands of lines of irrelevant documentation, superfluous log lines or outdated conversation turns, the attention mechanism becomes overloaded. Targeted compression acts as a signal-to-noise filter: it forces the model to concentrate on the core instructions and the variables it actually needs.

Semantic compression: recursive and hierarchical summarization

Semantic compression uses a language model to rewrite a long text into a more compact representation while preserving its meaning. This usually happens through a secondary, cheaper LLM call (a compact flash or haiku model, for instance). Two common patterns can be distinguished here: rolling buffers and hierarchical MapReduce summaries.

In a rolling summary buffer , the oldest interaction is merged into an existing summary after every conversation turn. The prompt format therefore keeps a fixed structure: a permanent system prompt, an updated status paragraph and the two to four most recent dialogue exchanges. This keeps the total token count within a predictable range.

In hierarchical summarization , a long document is cut into thematic chunks. Each chunk is summarized in parallel, after which the intermediate results are condensed in a second step into an overarching abstract. This pattern is well suited to long reports or transcripts, but it introduces specific risks:

• Information decay (the telephone effect): Across several successive layers of summarization, subtle nuances, negations or preconditions are irretrievably lost.
• Hallucination accumulation: If an earlier summary contains a small factual error, that error is treated as absolute truth in the subsequent steps.
• Latency and cost overhead: Generating summaries requires extra model calls, which raises both total processing time and API consumption per session.

Syntactic pruning: deterministic pruning and token truncation

Unlike semantic compression, syntactic pruning relies not on a language model but on deterministic code and heuristics. The aim is to eliminate tokens without a single extra LLM call. That yields an immediate reduction in latency and processing cost.

Deterministic pruning can be set up at various levels:

• Structural data compression: Converting bulky JSON payloads into more compact formats such as YAML, TOON or tab-separated tables. Simply removing superfluous whitespace, braces and repeated keys can save 25% to 40% of the tokens.
• Stop-word and style elimination: Programmatically filtering out pleasantries, repeated headers, boilerplate disclaimers and standard HTML markup in source documents.
• Attention-aware token dropping: More advanced systems compute the information density of sentences through TF-IDF or embedding similarity and remove paragraphs whose relevance score is too low relative to the current user question.

The great advantage of syntactic pruning is absolute precision around variables. Numbers, source URLs, identifiers and variable names in source code stay exactly the same, whereas a summarizing model can easily round such details off or transcribe them incorrectly.

Compression in autonomous workflows and agentic loops

In autonomous AI systems that execute several steps in succession, context growth is one of the main causes of failure. For a deeper look at how these systems work, the overview article on AI agents and autonomous assistants shows how memory and task execution interact.

During a run, an agent continuously collects observations, raw tool outputs, stack traces and interim conclusions in its scratchpad. After six or seven iterations, the context holds tens of thousands of tokens of stale JSON responses. If that context is not actively cleaned up, context pollution sets in and the agent gets stuck in redundant actions. For developers dealing with stalled systems, the guide to debugging agentic loops offers concrete diagnostic strategies for tracking down memory leaks and circular reasoning.

A comparison: summarizing, pruning and RAG extraction

To determine which method best suits a specific architecture, we have to weigh the properties of semantic summarization, syntactic pruning and dynamic RAG filtering against one another. The table below sets out the main operational differences.

A practical implementation of a hybrid compression pipeline

In production applications, a hybrid approach gives the most stable result. Deterministic filters at the front are combined with a sliding window and a periodic summary buffer. The Python pattern below illustrates how such a pipeline can be set up without external libraries.

import re

class ContextCompressor:
  def __init__(self, max_tokens=2000, system_prompt=""):
    self.max_tokens = max_tokens
    self.system_prompt = system_prompt
    self.summary_buffer = ""
    self.recent_turns = []

  def _prune_text(self, text: str) -> str:
    # Verwijder meervoudige witruimtes en lege regels
    text = re.sub(r'\n\s*\n', '\n\n', text)
    text = re.sub(r'[ \t]+', ' ', text)
    # Verwijder standaard HTML- en markdown-ruis
    text = re.sub(r'<!--.*?-->', '', text, flags=re.DOTALL)
    return text.strip()

  def _estimate_tokens(self, text: str) -> int:
    # Snelle benadering: 1 token ~= 4 karakters
    return len(text) // 4

  def add_turn(self, role: str, content: str):
    clean_content = self._prune_text(content)
    self.recent_turns.append({"role": role, "content": clean_content})
    self._rebalance()

  def _rebalance(self):
    total = self._estimate_tokens(self.system_prompt) + \
            self._estimate_tokens(self.summary_buffer)
    
    # Bereken tokens in recente beurten
    for turn in self.recent_turns:
      total += self._estimate_tokens(turn["content"])

    # Als limiet wordt overschreden, verplaats oudste beurt naar samenvatting
    while total > self.max_tokens and len(self.recent_turns) > 2:
      oldest = self.recent_turns.pop(0)
      # In productie: vervang onderstaande append door een compacte LLM-samenvatting
      self.summary_buffer += f"\n[{oldest['role']}: {oldest['content'][:120]}...]"
      total = self._estimate_tokens(self.system_prompt) + \
              self._estimate_tokens(self.summary_buffer) + \
              sum(self._estimate_tokens(t["content"]) for t in self.recent_turns)

  def build_prompt(self) -> str:
    prompt_parts = [f"SYSTEM: {self.system_prompt}"]
    if self.summary_buffer:
      prompt_parts.append(f"HISTORIE SAMENVATTING: {self.summary_buffer}")
    for turn in self.recent_turns:
      prompt_parts.append(f"{turn['role'].upper()}: {turn['content']}")
    return "\n\n".join(prompt_parts)

This architecture ensures that the active interaction always stays sharp and unaltered for the most recent dialogue exchanges, while older interactions condense step by step into a compact historical overview.

Quantifying and evaluating information loss

Every compression step carries the risk that essential details disappear. To keep optimizations from degrading the quality of the final answers, a structured evaluation method is necessary. Never rely on subjective spot checks; measure performance systematically.

The most reliable measurement methods include:

• Needle in a haystack (NIAH) tests: Hide a specific fact (a unique access code or date, for instance) deep in the context and check whether the model can still reproduce the correct answer after compression.
• Entity F1 score: Compare the set of entities (names, numbers, identifiers) in the uncompressed source with the output after compression to check that no critical variables have dropped out.
• Task-specific A/B tests: Evaluate parallel prompts with a fixed test framework. To set up a reliable test rig, use the method for A/B testing of prompts to record statistical differences in quality systematically.

In every evaluation, always measure three variables at once: token reduction (as a percentage), TTFT latency (in milliseconds) and accuracy on your validation set. Only when accuracy stays within acceptable margins is a compression method ready for production.

Context caching and prefix stability: the hidden pitfall

A common mistake when implementing dynamic summary buffers is disrupting the prefix cache. Many modern LLM providers offer prompt caching: static text at the start of a prompt is cached on the server, so that repeated input tokens are processed roughly 50 to 90% more cheaply and considerably faster.

When a rolling summary is placed directly after the system prompt and changes with every turn, the cache breaks for the entire prompt. The cost saving on a few hundred compressed tokens then often fails to outweigh the loss of the caching discount across tens of thousands of static tokens. For more background on the exact rate structures and cache mechanisms, the article on what a prompt costs in tokens and caching offers practical calculations.

For directly applicable practical tips on cutting token consumption without losing performance, the overview of proven token-saving tricks provides additional guidance on smart cache alignment.

The solution to this dilemma is to position static blocks before dynamic elements. Keep the fixed system prompt and the unchanged documentation exactly the same at the start of the payload, and place summary buffers and shifting conversation histories only at the tail end of the prompt.

A decision tree: which method do you choose, and when?

The choice between summarizing, pruning or a hybrid depends primarily on two factors: the required processing speed and the extent to which the task depends on exact numbers and identifiers.

• Choose syntactic pruning when the interaction has to happen in real time (< 1 second latency), when the text contains a lot of raw data such as logs or JSON, or when exact numerical values absolutely have to be preserved.
• Choose semantic summarization for long narrative conversations, helpdesk interactions or meeting transcripts, where the thread and the intent matter more than literal dates or individual field codes.
• Choose a hybrid pipeline for autonomous agent architectures and complex applications: prune all incoming tool data deterministically at source level and summarize only completed task steps, periodically.

By treating context not as a bottomless pit but as scarce working memory, applications stay fast, affordable and reliable, whatever the scale of the underlying models.