Treating the token budget as structure, not a feature
Growing the context window defers the problem. Decide what not to include first.
Contents
Growing it looks like a fix
Longer prompts cost more. But cost is not the real problem.
String prompt = systemInstruction + conversationHistory + retrievedDocuments;As retrievedDocuments grows, the ratio of what the model must read to what it may ignore
gets worse. Doubling the context does not double the answer quality; it doubles the room to blur.
Give each part a budget
A prompt that gains a paragraph per feature will not last. Fix the slots first, then let content compete inside them.
| Slot | Cap | On overflow |
|---|---|---|
| System instruction | fixed | Cannot overflow. If it does, the instruction is wrong |
| Conversation history | 30% | Fold the oldest into a summary |
| Retrieved documents | 50% | Drop the lowest scoring first |
| Headroom | 20% | So the answer is never truncated |
Keep the rule in config, not in code
Hard-coded caps mean a deploy every time the model changes. In config you can tune them in production and learn which cut actually hurts the answer from real traffic.
Finding what cannot be cut is the design work. Growing the window postpones it.