How to Manage an LLM Context Window

How to Manage an LLM Context Window

Managing an LLM context window means deciding what the model sees, in what order, and how many tokens you reserve for the answer. The goal is not to fill the largest possible window. The goal is to give the model the smallest reliable context that supports the task.

If you are shipping LLM-powered features, prompts, agents, or AI workflows, context management should be treated as part of your application architecture. A clean context plan improves reliability, latency, cost, and evaluation quality.

What the context window actually contains

An LLM context window is the maximum number of tokens the model can process in a single request, including:

• System messages
• Developer or instruction messages
• User input
• Retrieved documents
• Examples used for in-context learning
• Tool schemas
• Tool results
• Conversation history
• The model’s output

The output tokens count too. If a model has a 128k token window and your request consumes 127k input tokens, you have left very little room for the response. This often causes truncated answers, skipped reasoning steps, malformed JSON, or incomplete tool calls.

Start with a token budget

Before you optimize prompts, create a budget. A token budget forces you to decide what each section is allowed to cost. It also gives your team a concrete way to review prompt changes during code review.

Here is a sample budget for a support triage agent using a 32k token context window:

In this example, the planned input is roughly 17k tokens with 2k reserved for output and 3k to 5k held as buffer. That is safer than sending 30k tokens into a 32k window and hoping the model behaves.

Use a context plan, not a context dump

A context plan defines what gets included for each request type. This is more reliable than passing every available message, document, and tool result into the model.

A useful context plan answers five questions:

1. What is the model being asked to do? Classify, extract, answer, plan, write code, call tools, or verify output.
2. What facts does the model need? Include the minimum evidence needed to complete the task.
3. What can be summarized? Compress prior chat, large tool responses, and long documents.
4. What should be excluded? Remove stale instructions, unrelated retrieved chunks, repeated messages, and raw logs.
5. How much output space is required? Reserve tokens before building the input payload.

Annotated prompt and message payload

The payload below shows one practical structure for a retrieval-based support agent. The annotations use fields inside the JSON so the example remains easy to copy into an internal design doc.

{
  "model": "example-llm",
  "max_output_tokens": 1800,
  "messages": [
    {
      "role": "system",
      "content": "You are a support triage assistant. Follow the routing policy exactly. Return valid JSON only.",
      "_note": "Keep stable behavior rules short. Avoid mixing product docs into the system message."
    },
    {
      "role": "developer",
      "content": "Task: classify the ticket, identify missing information, and recommend the next action. Use only the provided policy and evidence. If evidence is insufficient, set confidence below 0.7.",
      "_note": "Task-specific instructions belong here. Make success criteria explicit."
    },
    {
      "role": "user",
      "content": "Customer says SSO login fails after switching identity providers. Error: invalid_audience.",
      "_note": "Raw user request. Do not rewrite it before the model sees it unless you preserve the original."
    },
    {
      "role": "user",
      "content": "Conversation summary: Customer migrated from Okta to Azure AD yesterday. They changed callback URLs but have not confirmed the audience value in the SAML app settings.",
      "_note": "Use a compact summary instead of the full chat history."
    },
    {
      "role": "user",
      "content": "Relevant policy excerpt 1: SSO invalid_audience errors usually indicate a mismatch between configured audience URI and service provider entity ID...",
      "_note": "Retrieved chunk trimmed to the relevant section."
    },
    {
      "role": "user",
      "content": "Relevant policy excerpt 2: For identity provider migrations, request screenshots of audience URI, ACS URL, and entity ID before escalating...",
      "_note": "Second retrieved chunk supports the likely next action."
    },
    {
      "role": "user",
      "content": "Return JSON with keys: category, priority, missing_information, recommended_action, confidence.",
      "_note": "Put output shape close to the end so it remains salient."
    }
  ]
}

This structure separates durable instructions, task instructions, user input, summaries, evidence, and output format. That separation makes it easier to debug which part of the context caused a bad response.

Before and after: reducing bloated context

Here is a common production failure. A team builds a support agent and passes the full ticket thread, every retrieved document, all available tool schemas, and raw API responses into each request.

Before: bloated context

System:
You are a helpful support assistant. Answer accurately.

Conversation:
[Full 37-message ticket thread, including greetings, repeated error text, unrelated billing question, and old debugging attempts]

Retrieved documents:
[10 chunks from SSO docs]
[6 chunks from billing docs]
[4 chunks from user management docs]
[3 chunks from changelog pages]

Tools:
[All 18 tool schemas, including billing tools, CRM tools, deployment tools, and identity provider tools]

Tool results:
[Raw 9,000-token account object]
[Raw 5,000-token organization settings object]
[Raw 3,000-token prior ticket list]

User:
What should we do next?

This request uses many tokens, but much of the content is low value. The model has to sort through stale messages, irrelevant retrieval results, and oversized tool output. The answer may look confident while using the wrong evidence.

After: reliable context plan

System:
You are a support triage assistant. Return valid JSON only.

Developer:
Classify the ticket and recommend the next action. Use the supplied SSO policy excerpts and account summary. If required evidence is missing, ask for it.

Conversation summary:
Customer migrated from Okta to Azure AD. SSO now fails with invalid_audience. Customer confirmed ACS URL was updated. Audience URI has not been confirmed.

Relevant evidence:
1. SSO policy: invalid_audience usually means the audience URI does not match the service provider entity ID.
2. Migration checklist: after IdP migration, confirm ACS URL, entity ID, audience URI, and certificate.
3. Escalation rule: escalate only after customer provides screenshots of IdP SAML settings and the values still match.

Tool result summary:
Account plan: Enterprise.
SSO enabled: true.
Configured entity ID: urn:app:prod:customer-123.
Recent auth failures: invalid_audience for Azure AD users.

Available tools:
- request_customer_information
- create_escalation

Output format:
{
  "category": "sso_migration",
  "priority": "low|medium|high",
  "missing_information": [],
  "recommended_action": "",
  "tool_to_call": "",
  "confidence": 0.0
}

The reduced version is shorter and more useful. It keeps the facts that affect the decision, removes unrelated billing and user management docs, summarizes tool results, and limits tools to the ones the agent can actually use for this route.

Practical techniques for managing context

1. Reserve output tokens first

Start with the model’s context limit, subtract your required output size, then build the input within the remaining budget.

For example, if your model supports 32k tokens and your JSON response may take up to 2k tokens, treat your input limit as 30k tokens. If you need a long final report, reserve 4k to 8k tokens. Do not let retrieval consume that space.

2. Rank context by task value

All tokens are not equally useful. A 200-token policy rule may matter more than a 4,000-token transcript. A single database field may matter more than an entire API response.

Rank each item by whether it changes the model’s decision. If the answer would be the same without a chunk, remove it.

3. Summarize conversation history

Do not dump full chat histories by default. Most old messages contain acknowledgments, repeated facts, or abandoned paths.

Keep a rolling summary with these fields:

• User goal: What the user is trying to accomplish
• Known facts: Verified information
• Open questions: Missing information that affects the answer
• Decisions already made: Prior recommendations or tool calls
• Constraints: User preferences, permissions, account limits, deadlines

For agent workflows, update the summary after each major turn. Store the raw history separately for audit and debugging.

4. Trim retrieved chunks before insertion

Retrieval quality does not end when the vector database returns results. You still need to filter, deduplicate, and trim chunks before adding them to the prompt.

A good retrieval pipeline should:

• Remove chunks below a relevance threshold
• Drop near-duplicate chunks
• Prefer current docs over old docs
• Trim long chunks to the paragraphs that match the user’s issue
• Preserve source IDs so the model can cite or reference evidence

Mixing unrelated retrieved chunks is one of the fastest ways to degrade answer quality. If the user asks about SSO, billing docs should not enter the context unless the task specifically connects SSO to billing.

5. Control tool-result size

Tool results often become the largest hidden source of context growth. Raw API responses can include unused fields, repeated metadata, long arrays, and internal IDs the model does not need.

Instead of reinserting a full account object, transform it into a task-specific summary:

Raw tool result:
{
  "account": {
    "id": "acct_123",
    "plan": "enterprise",
    "created_at": "2021-04-02",
    "features": [...300 lines...],
    "users": [...1200 lines...],
    "auth_settings": {...},
    "billing_history": [...900 lines...]
  }
}

Context-safe summary:
Account acct_123 is on Enterprise.
SSO is enabled.
Identity provider: Azure AD.
Configured entity ID: urn:app:prod:customer-123.
Last 25 login failures share error code invalid_audience.

If you use tool standards such as Model Context Protocol, still treat tool outputs as context that needs budgeting. A cleaner protocol does not remove the need to summarize large results.

6. Keep tool schemas route-specific

Agents often expose too many tools. Each tool schema costs tokens, and extra tools increase the chance of a bad call.

If the current route is “SSO support,” the model probably does not need invoice tools, marketing tools, or deployment tools. Pass only the tools that are valid for the task. For many workflows, 2 to 5 tools are easier to control than 20.

7. Put the most important instructions in stable positions

Models can lose track of details in very long contexts. Place durable behavior rules at the start. Put the current task, evidence, and output format near the end. Avoid burying the actual user request between long retrieved documents.

A practical order often looks like this:

1. System rules
2. Task instructions
3. Short conversation summary
4. User’s current request
5. Relevant evidence
6. Tool results or available tools
7. Output format and final constraints

8. Use examples sparingly

Few-shot examples can improve consistency, especially for classification, extraction, and formatting tasks. They can also waste space if they do not match the current task.

Use 2 or 3 high-quality examples instead of 10 average examples. Remove examples that teach behavior you no longer want. Track example performance in evals rather than assuming more examples always help.

Common mistakes to avoid

• Filling the whole window: A full context window leaves little space for output and increases the chance that important details get diluted.
• Assuming all tokens are equally useful: Prioritize tokens that affect the task outcome.
• Dumping entire chat histories: Summaries usually work better for long-running conversations.
• Mixing irrelevant retrieved chunks: Retrieval noise can cause wrong answers even when the right document is present.
• Forgetting output-token reservations: Always budget for the response before adding optional context.
• Ignoring tool-result size: Summarize tool outputs before reinserting them into the next model call.
• Treating a larger window as a substitute for evaluation: Bigger context can hide problems during demos and create new failures in production.

Evaluate context changes like code changes

Every context change can alter model behavior. Adding a retrieved chunk, changing summary format, or exposing a new tool should go through LLM evaluation.

Build eval sets around real failure modes:

• Long user messages with irrelevant details
• Conflicting retrieved documents
• Missing information that should trigger a clarification
• Large tool responses that need summarization
• Conversation histories with stale decisions
• Tasks that require strict JSON output

Measure more than answer quality. Track input tokens, output tokens, latency, cost, tool-call accuracy, JSON validity, citation accuracy, and refusal or escalation behavior.

A simple context management checklist

Use this checklist before shipping a prompt or agent update:

• Have you reserved enough output tokens?
• Is there a hard cap for each context section?
• Can old conversation history be summarized?
• Are retrieved chunks relevant to the current task?
• Are tool results compressed into task-specific summaries?
• Are only valid tools exposed?
• Is the output format close to the end of the prompt?
• Do evals cover long-context and noisy-context cases?
• Are token counts, latency, and cost tracked over time?

Use the smallest reliable context

A larger context window is useful, but it does not remove the need for engineering discipline. Reliable LLM systems use clear budgets, route-specific context, compact summaries, filtered retrieval, controlled tool results, and evals that catch regressions.

If you manage context this way, your prompts become easier to review, your agents become easier to debug, and your team can improve quality without guessing which tokens helped.

PromptLayer helps AI teams manage prompts, trace requests, compare versions, run evaluations, and understand how context changes affect production behavior. If you are building LLM applications or agents, create a PromptLayer account and start tracking your context, prompts, and evals in one place.