OpenAI API openai tokens context-window
Lena Schreiber Infrastructure Engineer

InvalidRequestError: context_length_exceeded

Encountering InvalidRequestError: context_length_exceeded means your API prompt exceeds the model's token limit; this guide explains how to fix it.

What This Error Means

As an Infrastructure Engineer, I've seen this error pop up frequently when integrating with OpenAI's API, especially in applications that handle user-generated content or complex data processing. Simply put, InvalidRequestError: context_length_exceeded means that the total number of tokens in your request – encompassing your input prompt, system messages, and any chat history – has exceeded the maximum limit that the specific OpenAI model you're using can process in a single API call. It's essentially the model saying, "Your message is too long; I can't fit it all into my working memory."

This isn't an error due to network issues or malformed JSON; it's a semantic limit. The model has a fixed "context window," and once your input goes beyond that window, the API rejects the request rather than processing an incomplete or truncated version of your prompt.

Why It Happens

Every large language model operates with a predefined "context window" size, measured in tokens. A token is roughly equivalent to 4 characters for English text, or about ¾ of a word. When you send a request to the OpenAI API, your entire message – including any system instructions, the user's prompt, and previous turns in a conversation (if you're managing chat history) – is converted into tokens.

This error occurs when the sum of these tokens surpasses the model's maximum allowed input length. Different models have different context window sizes. For example, gpt-3.5-turbo typically has a smaller context window than gpt-4 or specific gpt-4-turbo variants. If you switch models without re-evaluating your prompt size, you're likely to hit this limit.

In my experience, developers often underestimate how quickly tokens accumulate, especially when dealing with verbose inputs or extended conversational threads. It’s not just the new input that counts, but everything you've sent as context for the current turn.

Common Causes

Identifying the root cause is the first step to resolution. Here are the most common scenarios I've encountered:

  1. Overly Long Input Prompts: The most straightforward cause. If you're feeding entire documents, long articles, or extensive data sets directly into a single API call, you're highly susceptible to hitting the token limit.
  2. Extensive Chat Histories: In conversational applications, developers often send the entire conversation history with each API call to maintain context. As the conversation lengthens, the cumulative token count of past messages can easily exceed the limit. This is a very common pitfall in chatbot development.
  3. Including Too Much "System" or "Context" Information: Some applications prepend a large block of system instructions, examples, or external data (e.g., retrieved relevant documents) to every prompt. While helpful for grounding the model, this introductory context can consume a significant portion of the available token budget.
  4. Using a Model with a Smaller Context Window: If your application was initially developed with a model that had a larger context window (e.g., gpt-4-32k) and then switched to a smaller one (e.g., gpt-3.5-turbo-16k) without adjusting prompt engineering, this error will surface.
  5. Not Accounting for Output Tokens: While the error specifically relates to input context length, it's worth noting that a portion of the context window can be reserved for the model's expected output. If you're setting max_tokens very high and your input is already near the limit, there might not be enough room left for a substantial response, potentially leading to errors, though typically context_length_exceeded is about the input not fitting.

Step-by-Step Fix

Addressing this error requires strategies to manage and reduce the token count of your requests.

  1. Identify Current Token Count:
    Before making changes, determine how many tokens your current requests are actually consuming. OpenAI provides tiktoken for this.

    ```python
    import tiktoken

    def count_tokens(text: str, model: str = "gpt-3.5-turbo") -> int:
    """Returns the number of tokens in a text string."""
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))

    Example for a simple prompt

    prompt = "Summarize this very long document about infrastructure architecture..."
    token_count = count_tokens(prompt)
    print(f"Prompt token count: {token_count}")

    Example for a chat message list

    messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me about infrastructure as code."},
    {"role": "assistant", "content": "Infrastructure as Code (IaC) is managing and provisioning infrastructure through code..."},
    {"role": "user", "content": "What are its benefits?"}
    ]

    def count_chat_tokens(messages: list, model: str = "gpt-3.5-turbo") -> int:
    """Returns the number of tokens in a list of messages."""
    try:
    encoding = tiktoken.encoding_for_model(model)
    except KeyError:
    encoding = tiktoken.get_encoding("cl100k_base") # Fallback for unsupported models

    num_tokens = 0
for message in messages:
    # Each message typically adds a few tokens for metadata (role, name)
    num_tokens += 4 # every message follows <im_start>{role/name}\n{content}<im_end>\n
    for key, value in message.items():
        num_tokens += len(encoding.encode(value))
num_tokens += 2 # every reply is primed with <im_start>assistant
return num_tokens

    chat_token_count = count_chat_tokens(messages)
    print(f"Chat message token count: {chat_token_count}")
    ```

  2. Choose an Appropriate Model:
    If possible, upgrade to a model with a larger context window. For example, gpt-4-turbo variants often offer 128k token contexts, which is significantly more than gpt-3.5-turbo's 16k. Be aware that larger models generally come with higher costs and potentially slower response times.

    ```python

    Ensure you are using a model with sufficient context

    model_name = "gpt-3.5-turbo" # often 16k context

    model_name = "gpt-4-turbo" # often 128k context

    openai.chat.completions.create(model=model_name, messages=...)

    ```

  3. Implement Input Truncation or Summarization:

    • Truncation: For long inputs that don't need absolute fidelity, simply cut off the text after a certain token count. This is a blunt instrument but effective.
    • Summarization: If the core meaning is important, consider summarizing portions of the input before sending them to the main model. You could even use another, smaller model to do the summarization.
    • Chunking + Embeddings: For very large documents, break them into chunks, create embeddings, and retrieve only the most relevant chunks based on the user's query. This is the basis of Retrieval-Augmented Generation (RAG).

  4. Manage Chat History:
    This is critical for conversational AI.

    • Sliding Window: Maintain a fixed number of recent turns. When the history exceeds a threshold, remove the oldest messages.
    • Summarize Past Turns: Periodically summarize older parts of the conversation. For example, after 10 turns, summarize the first 5 turns into a single "summary" message and replace the original 5 messages with it.
    • Prioritization: In my experience, it can be useful to prioritize keeping the system message and the most recent user/assistant turns, as these usually carry the most weight for the current interaction.

    ```python
    def reduce_chat_history(messages: list, max_tokens: int, model: str = "gpt-3.5-turbo") -> list:
    """
    Reduces chat history by removing oldest messages until total tokens are below max_tokens.
    Prioritizes keeping the system message and recent user/assistant turns.
    """
    if not messages:
    return []

    # Keep system message if present
system_message = None
if messages[0]["role"] == "system":
    system_message = messages[0]
    messages_to_process = messages[1:]
else:
    messages_to_process = messages

current_messages = [system_message] if system_message else []
current_tokens = count_chat_tokens(current_messages, model)

# Add messages from most recent to oldest
for i in range(len(messages_to_process) - 1, -1, -1):
    msg = messages_to_process[i]
    msg_tokens = count_chat_tokens([msg], model) # Estimate tokens for individual message

    # Check if adding this message exceeds the limit
    # We add a buffer of 100-200 tokens for the potential new response
    if current_tokens + msg_tokens + 200 < max_tokens: 
        current_messages.insert(len(current_messages) if system_message else 0, msg)
        current_tokens += msg_tokens
    else:
        break # Stop adding if we exceed the limit

# Ensure system message is always at the beginning if present
if system_message and current_messages[0] != system_message:
    current_messages.remove(system_message) # Remove if it got re-inserted somewhere else
    current_messages.insert(0, system_message) # Insert at the beginning

return current_messages

    ```

  5. Refine Prompt Engineering:

    • Be concise. Can you phrase your instructions or examples more efficiently?
    • Remove redundant information.
    • Use examples judiciously; sometimes one good example is better than five mediocre ones that eat up tokens.

Code Examples

Here are some concise, copy-paste ready examples for common scenarios.

1. Counting Tokens with tiktoken

Essential for debugging and proactive management.

import tiktoken

def num_tokens_from_string(string: str, encoding_name: str = "cl100k_base") -> int:
    """Returns the number of tokens in a text string."""
    encoding = tiktoken.get_encoding(encoding_name)
    return len(encoding.encode(string))

def num_tokens_from_messages(messages: list[dict], model: str = "gpt-3.5-turbo") -> int:
    """Returns the number of tokens used by a list of messages."""
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        encoding = tiktoken.get_encoding("cl100k_base")

    num_tokens = 0
    for message in messages:
        num_tokens += 4  # every message follows <im_start>{role/name}\n{content}<im_end>\n
        for key, value in message.items():
            num_tokens += len(encoding.encode(value))
    num_tokens += 2  # every reply is primed with <im_start>assistant
    return num_tokens

# Example usage
text_to_analyze = "This is a sample sentence to count tokens."
print(f"Tokens in text: {num_tokens_from_string(text_to_analyze)}")

chat_messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
    {"role": "assistant", "content": "The capital of France is Paris."},
    {"role": "user", "content": "And Germany?"}
]
print(f"Tokens in chat messages: {num_tokens_from_messages(chat_messages)}")

2. Truncating Long User Input

A simple function to ensure user input doesn't exceed a specific token limit.

import tiktoken

def truncate_text_by_tokens(text: str, max_tokens: int, model: str = "gpt-3.5-turbo") -> str:
    """Truncates text to ensure it does not exceed max_tokens."""
    encoding = tiktoken.encoding_for_model(model)
    tokens = encoding.encode(text)

    if len(tokens) > max_tokens:
        truncated_tokens = tokens[:max_tokens]
        truncated_text = encoding.decode(truncated_tokens)
        # Often helpful to add an ellipsis or similar indicator
        if not truncated_text.endswith("..."):
            truncated_text = truncated_text.rsplit(' ', 1)[0] + "..." # Try to cut at word boundary
        return truncated_text
    return text

# Example
long_text = "This is a very, very long piece of text that definitely goes over the limit for demonstration purposes. We need to cut it down to size. Imagine this is a whole document or an extensive log file." * 10
truncated = truncate_text_by_tokens(long_text, 50)
print(f"Original length: {len(long_text)} chars, Truncated length: {len(truncated)} chars")
print(f"Original tokens: {num_tokens_from_string(long_text)}, Truncated tokens: {num_tokens_from_string(truncated)}")
print(truncated)

Environment-Specific Notes

The fundamental solution to context_length_exceeded remains the same across environments, but how you monitor and manage it can differ.

  • Cloud Deployments (AWS, GCP, Azure):

    • Monitoring: Implement logging for API request sizes. CloudWatch, Stackdriver, or Azure Monitor can track payload sizes if properly configured. Set up alerts for InvalidRequestError types.
    • Dynamic Context Management: For applications that deal with highly variable input lengths (e.g., summarization of user-uploaded documents), consider implementing dynamic logic to select the appropriate model (e.g., a cheaper, smaller model for short inputs, a more expensive, larger model for long inputs).
    • Serverless (Lambda, Cloud Functions): Be mindful of cold starts if you're loading tiktoken or other tokenizer libraries on the fly. Package them efficiently.

  • Docker Containers:

    • Logging: Ensure your application's logs, especially those indicating prompt length or truncation events, are correctly forwarded to your container orchestration's log aggregation system (e.g., ELK stack, Grafana Loki).
    • Resource Limits: While not directly related to this error, ensure your container has sufficient memory and CPU if you're doing complex pre-processing or summarization locally before sending to the API.
    • Dependencies: Pre-install tiktoken and other necessary libraries in your Dockerfile to avoid runtime installation issues.

  • Local Development:

    • Immediate Feedback: Leverage print statements and local tiktoken calculations during development. It's much easier to debug locally than in a production environment.
    • Testing with Edge Cases: Deliberately create test cases with very long inputs or extensive chat histories to ensure your context management logic holds up before deployment.
    • Development Tools: Utilize IDE features for stepping through code and inspecting variables to understand exactly what's being sent to the API. I often use a simple print(json.dumps(messages, indent=2)) right before the API call to visualize the full payload.

Frequently Asked Questions

Q: What exactly is a "token"?
A: A token is a fundamental unit of text that large language models process. It can be a word, a part of a word, a punctuation mark, or even a single character. For English text, one token generally corresponds to about 4 characters or 0.75 words.

Q: Can I increase the context length limit for my OpenAI model?
A: No, the context length limit is a fixed property of the specific model you are using and cannot be altered by the user via API parameters. Your only options are to choose a model with a larger inherent context window or to reduce the size of your input.

Q: Does the model's output count towards the context length limit?
A: The InvalidRequestError: context_length_exceeded specifically refers to the input you send to the model. However, when you make an API call, the total context window (e.g., 16k tokens) must accommodate both your input and the model's potential output (controlled by max_tokens in your request). If your input is very close to the model's maximum and max_tokens is also high, you might encounter other errors, but context_length_exceeded is primarily about the input.

Q: How do I choose the right model given my context length requirements?
A: Evaluate your typical input size. If your prompts are consistently short (e.g., under 4000 tokens), gpt-3.5-turbo-16k might be sufficient and cost-effective. For applications requiring extensive context (e.g., processing long documents, detailed conversations), gpt-4-turbo or its 128k context variants are usually necessary despite the higher cost. Always benchmark costs and performance.

Q: Is there an easy way to just "summarize" old chat messages automatically?
A: Yes, a common pattern is to detect when the token count is approaching the limit, then take older turns (e.g., everything except the last 3-5 turns) and send them to the model with a prompt like "Summarize the following conversation history concisely for context:" and then replace those old turns with the generated summary. This keeps the history compact.

(none)