Context Window: Why the Model Forgets

Reading time: approx. 5 min

In the previous moments, we have learned about AI fundamentals and the importance of tokens as building blocks. Now we will dive into the context window, one of the most critical aspects for understanding why AI models sometimes "forget" previous instructions or information. The context window is the limited amount of text (number of tokens) that an AI model can hold in its "working memory" simultaneously. Mastering this is crucial for you to be able to formulate effective prompts and get consistent answers in the classroom.

1. What is a context window?

A context window is the maximum amount of data, measured in tokens, that an AI model can "see" and process at any given time. This includes both your prompt, any previous messages in the conversation, and the model's own responses.

• Different sizes: The size of the context window varies greatly between different AI models.
  • Older models like GPT-3.5 often had a window of about 4,000 tokens.
  • Newer, more advanced models like GPT-4 Turbo can handle up to 128,000 tokens, which is a significant difference.
  • Some locally running models or specialized AIs can have windows from 2,000 up to 32,000 tokens.

What happens when the window fills? If the total amount of tokens (your prompt plus chat history) exceeds the model's context window, the oldest tokens will "fall out" of the window and thus be "forgotten" by the model. It can then no longer refer to that information.

2. Consequences of a limited context window

That an AI model has a limited context window can lead to several challenges in the classroom:

1. Earlier instructions are forgotten: In a long conversation or if you give a very extensive first prompt, the information at the beginning will disappear from the model's "memory" when you reach the token limit. This means the model can lose important instructions, roles you assigned to it (for example, "You are a history teacher..."), or details you specified early.
2. Incomplete or irrelevant answers: Since the model no longer has access to all previous information, it can answer without taking into account important details that now lie outside the context window. This can lead to answers that feel irrelevant, contradictory, or that miss the point of your original question.

3. Example in practice

Let us see how this can manifest in a classroom situation:

Scenario: You start a conversation with the AI with the prompt: "As an experienced social studies teacher for grade 8, create a detailed lesson plan in three steps for a lesson on democracy and citizenship in Sweden. Include discussion questions, a practical exercise, and assessment criteria. Focus on student participation."

You then continue to have a long dialogue with the AI, where you discuss different aspects of the lesson plan, ask it to clarify parts, and add new moments. After generating several long answers and instructions, you may have passed 4,000 tokens total.

The problem: When you later ask the AI: "Can you now summarize the three most important discussion questions from our original lesson plan?", it can happen that the AI gives a generic answer or does not remember at all that it should act as an "experienced social studies teacher for grade 8" with a focus on "democracy and citizenship". The reason is that the original instruction has now fallen out of the model's context window.

4. Strategies for working with the context window

To effectively manage the context window and avoid the AI "forgetting" important information, you can use the following strategies:

• Short, but complete prompts: Try to put all necessary information at the beginning of each prompt, rather than spreading it out over several messages if it is critical. Be concise but clear.
• "RAG" technique (Retrieval Augmented Generation): This is a more advanced technique where you first search for relevant information in external documents (for example, curricula, articles) and then feed only the relevant parts together with your prompt to the AI. Instead of uploading a whole book, you send only the relevant paragraphs.
• "State management": In more complex applications, you can program the AI to only send the latest messages and a summary of the most important context from the beginning of the conversation. For you as a teacher, this means you sometimes may need to repeat key instructions.
• Chunking (segmentation): If you have very long texts (for example, a whole novel or an extensive research report) that the AI should process, divide them into smaller "segments" or "chunks". Process each part separately or summarize them and then send the summaries to the AI.
• Token budgeting: Plan how you allocate your available tokens. Reserve space for prompt (20-30%), previous conversation (30-40%), and response (30-40%). This helps you maintain control over what remains in the context window.
• Limiting response length: Remember that setting a maximum length on the AI's response does not make the response more concise, but only interrupts it when the token limit is reached. For shorter responses, specifically ask to "summarize in 3 sentences" instead of just limiting the number of tokens.

5. Practical tips for the classroom

Here are some concrete tips for how you can apply this knowledge in your teaching:

• "Checkpoint" exercises: Let students deliberately test the context window. Ask them to input a long text or a long dialogue and then ask a question that refers to something they know is early in the conversation. Then discuss why the AI might not remember it. What disappeared?
• AI-assisted summaries: Encourage students to regularly ask the AI to summarize longer chat history or texts they are working with. These shorter summaries can then be pasted again into the chat to keep the relevant context alive within the window.
• Modular material upload: If students are to analyze a longer text, for example, a chapter in a textbook, instruct them to upload the text in smaller, but coherent, parts instead of trying to paste the whole chapter at once. Then you ensure that the AI can process each part properly.

6. Reflection exercise

To deepen your understanding of the context window, reflect on the following:

1. Choose an existing AI conversation you have had (or start a new long one). Continue to input text (for example, long answers or new prompts) until you experience that the AI starts to forget information from the beginning of the conversation. What type of information disappeared first?
2. Think of a lesson plan or a project where you would normally give students a lot of background information. How could you restructure your prompts to the AI (or students' prompts) to ensure that the core instructions are always within the model's context window?
3. Discuss with a colleague: Which strategies for managing the context window do you think would be most useful in your specific subjects and for your student groups? Are there subjects where a larger context window is more critical than in others?

Next moment: Zero-shot prompting: When AI understands without examples. We move forward and learn how you can formulate powerful prompts that the AI understands and can act on directly, even without you giving any previous examples or special training data.