There's a pattern most people run into pretty quickly with AI. Early in a conversation, it's sharp: it follows instructions, gives structured answers, tracks the thread of what you're talking about and mostly makes sense with its output. But as the session gets longer, the quality of the output shifts. It starts repeating itself, drops things you told it three messages ago, and the answers get noticeably more generic and far less precise.
Matt Pocock calls this the "smart zone" and the "dumb zone." Early in a session, you're in the smart zone. As the conversation grows, you drift into the dumb zone.
That framing is memorable and it captures a very real issue with how LLM's work. What's actually happening is a gradual decline in quality as the conversation gets longer. Understanding the how this occurs, and why, changes how you use these tools.
The context window: a fixed-size workspace
When you send a message to an AI, the model doesn't just see your latest message. It sees everything: your message, the full conversation history, any system instructions, and all of its own prior responses. All of that gets packed into a single block of text called the context window.
The context window has a fixed size, measured in tokens, which are roughly chunks of words. (Chapter 1 of the Fundamentals covers how text gets broken into tokens.) Modern models have big context windows, but to some extent context window size is a glamour metric much in the same way reporting on bandwidth rather than latency is a glamour metric for networks: bigger doesn't mean better.
Why more context means worse output
When a model processes your question, it doesn't weigh every token equally. It uses a mechanism called attention to decide which parts of the context matter most for the current task. (Chapter 3 covers how attention works in detail.)
Think of attention like a spotlight with a fixed amount of brightness. Point it at three things and each one is well-lit. Point it at a hundred things and everything gets dimmer.
Of course reality is a little more nuanced than that. A few factors work together to slowly cause this drop in quality:
Training bias toward shorter contexts. These models were mostly trained on shorter documents and conversations. They're best calibrated for the context lengths they saw most during training. Push a model to 100,000 tokens and you're in territory it has less experience with.
Positional encoding fidelity. Models use positional encodings to track where each token sits in the sequence. These work well at shorter distances but lose precision at extreme lengths.
Finding needles in haystacks. The more content in the context window, the harder it is for the model to find the relevant pieces. Research shows that models are particularly bad at using information placed in the middle of long contexts, a phenomenon called the "lost in the middle" effect.
The result is context degradation: a gradual erosion of output quality as the context fills up.
Watch it happen
Drag the slider below to fill up a context window. Watch what happens to the attention distribution. The important stuff (your current question, key instructions) gets squeezed as noise and old messages take up more of the budget.
With 5% of the context window used, the model is sharp and focused. Each concept gets , and the model can easily track the relationships between every idea in the conversation.
What you can do about it
Once you understand the mechanism, the strategies are straightforward.
Be concise
Every token you send takes up space in the context window. You don't need to be cryptic, but bloated prompts with unnecessary repetition have a real cost. Say what you need to say, clearly and directly.
Front-load what matters
Because of the "lost in the middle" effect, put your most important instructions at the beginning or end of your message. Models pay more attention to the edges of their context than to the middle.
Start fresh
A fresh start for a new conversation is your most effective fix. A fresh start means a new context window which means you're back to front-loading context and tightening up the relationship between tokens.
Hand off compressed context
Starting fresh works well, but sometimes you're mid-task and can't afford to lose the thread. You could ask the model to summarize everything and paste that into a new conversation, but that's manual and easy to botch.
Pocock's handoff approach is more deliberate. Instead of a loose summary, you create a focused document that captures only what the next session needs: the purpose of the work, key decisions so far, and pointers to relevant files. Then you start a new session with that document as its opening context.
The new session starts in the smart zone with just the context that matters. Think of it as the difference between handing someone a clean brief versus forwarding them your entire email thread.
It's a gradient, not a switch
The "smart zone / dumb zone" framing is catchy, and it points at something real. But there's no moment where the model flips from one to the other. It's more like turning up the heat slowly. You don't notice you're boiling until you're well past comfortable.
The degradation is real and it affects every large language model. But it's manageable once you know what's going on.