Tokens and Context: The Real Constraints

Two numbers govern every LLM application: the context window (how much text the model can see at once) and the token cost of everything you put in and take out. Design ignores these at your wallet's peril.

Tokens are not words

A token is roughly ¾ of a word in English, but code, URLs, and non-English text tokenize worse — sometimes 2-3 tokens per word. You will feel this in your bill: a "short" prompt with a stack trace can cost more than the rest of the request combined. tiktoken (OpenAI) or the provider's tokenizer tells you the real count; measure before you estimate.

Context windows are not free attention

Models attend poorly to the middle of long contexts — the "lost in the middle" effect is measurable: put critical instructions and the most relevant retrieved content at the start or the end. A 128k window you fill is a 128k window you pay for and get worse recall from. Retrieval exists to shrink what the model sees, not to fill it.

Try it yourself

Rough token estimation is genuinely useful for cost math:

// ~4 chars per token is the standard English approximation
const estimate = (s) => Math.ceil(s.length / 4);
console.log("~tokens:", estimate("Summarize this document"));
console.log("~tokens for a 10k-char doc:", estimate("x".repeat(10000)));
// At $3/1M input tokens, 1000 calls with a 10k doc ≈ $7.50 — add output and
// system prompt before you quote a feature's cost.

Field notes

• Truncate inputs, never instructions: the system prompt should survive every truncation strategy.
• Cache prefixes: providers cache identical prompt prefixes — put the stable system prompt first and varying content last.
• Count tokens in CI. The day your "small prompt" feature 10x's its input is the day your finance team finds you.