enough ball knowledge to larp about understanding llm backends
a backend engineer's guide to tokens, caching, routing, batching, streaming, and the words people keep throwing around.
i've spent the past few months building ai systems (aka backends which make an api call to anthropic and do reasonable things with the response) and somewhere i felt like the whole field has been blown disproportionately out of the water in terms of perceived complexity. not the research but the application layer where most of us are working right now, the serving of tokens and all the fancy words associated with it. the point of this post is not to teach you transformer architecture or model fine-tuning but to give you a handbook that makes you understand what goes on inside this blackbox we call a "model" and give you enough lingo to not get cooked in a systems conversation when your job requires you to have an opinion about it, ball knowledge as i call it. larping, but the informed kind.
tokenization
just like interpreters and compilers, the first step to understanding language is tokenization. splitting text into mini chunks the model can consume. you hear the word token all the time while working with llms and simply put, a token is the smallest unit of text the model consumes.
(representation of a sentence in tokens by the gpt-4o tokenizer)
the reason we need tokens is that language models, or any machine learning models work with numbers. therefore we need a method of converting words into numbers.
each token is assigned a unique integer, these integers are then vectorized and then fed into the language model. the language model processes these vectors and outputs integers again, which are subsequently converted back into text through our look up table with the unique token identities.
the two modes hiding inside every request
every call you make to an llm has these two different phases with two entirely different personalities.
prefill is what happens to your input. since your model already has the entire prompt in hand, it processes all of it in parallel. this is a compute problem, since we're multiplying matrices here and gpus are good at multiplying matrices. more tokens in the prompt means more work but the work parallelizes cleanly.
decode is what happens to the output. the model generates one token and then the model uses that token to generate the next token. obviously this step cannot be parallelized due to its sequential nature, this is what being autoregressive means. the annoying part here is not the compute of the math, it's pulling out the entire history of the conversation encoded as this thing called a kv cache, back out of the memory just to know what's already been said. this is the real bottleneck, decode is memory bandwidth bound, not compute bound.
the problem we face here isn't a new one, like a db with fast writes but slow reads.
this is the reason as your conversation with an llm gets longer, its response time gets slower.
why your prompt structure matters
this is not me trying to explain prompt engineering to you but something which is very important for you to keep in mind when working with llms.
every token the llm processes turns into a kv cache entry, a little summary of what the current token means with respect to all the tokens before it. the cache entry of the nth token depends solely on tokens 1 to n and nothing that comes after it. if two requests share tokens 1 to x where x is less than n then work done to process the first x tokens is identical in both requests.
hash chains also kinda work like this if you're familiar with them.
this is why your system prompt and tool definitions belong at the top of your prompt, so they can be cached and the variables like session id, timestamp, the actual user input belong at the bottom.
look at anthropic's pricing for cached reads as compared to normal ones. writing something into the cache costs more than a normal request but reading from a cache costs a fraction of a normal read. this is why you need to cache requests which you believe are going to be repeated in your system.
(anthropic model pricing)
routing requests
if you're self hosting the model on your own gpus rather than calling an api, each replica will keep its own kv cache in its own memory. now if we make a request to replica A and then a follow up request to replica B, replica B will have no idea of what happened before, this means then we won't have cache hits and end up processing the request entirely from scratch.
traditionally, the thing a load balancer had to care about while routing was incoming traffic but now we also need to take care of matching follow up requests to their old replica.
a router tracks which replica has warm cache per session and tries to send you back there, the same idea as session-affinity in load balancers i.e you direct all the requests from a specific user to a specific endpoint (sticky sessions). the thing we need to keep in mind is that routing requests purely on the basis of cache affinity can wreck traffic load distribution (imagine a lot of users using a chatapp with the same system prompt), so if one replica happens to be the one with the relevant cache it gets hammered while others stay idle. real systems score replicas based on something like cache hit benefit minus current load, sometimes called load-aware cache-affinity routing.
if you're just using an api then this genuinely isn't your problem to solve, but still worth knowing.
two numbers to keep in mind
time to first token (ttft) as the name suggests is how long before you get your first token. this includes how long your request waits in queue, the prefill time on the uncached part of prompt and the first decode step to get the very first token out.
time to complete is ttft plus every single decode step after that until response completion. each subsequent decode step gets a little slower due to the growing kv cache read we talked about in decode, and each subsequent token is generated at a slower pace.
caching and routing as we discussed only help us with ttft since they barely affect the decode speed. time to complete is where a long response in a conversation actually costs you, every extra token multiplied by however much slower decode has gotten by that time in the conversation.
streaming
simply put you just see the tokens as they are generated instead of once the entire response is completed. this is no computational optimization i.e the model itself doesn't generate the tokens faster but you just get to see them earlier.
this is like chunked transfer encoding, so when you don't know the content length already, chunked transfer encoding allows the server to stream dynamically generated content without knowing the total response size in advance except in our case the payload is being generated token by token.
through streaming our perceived wait time for a response comes down to nearly ttft instead of time to completion and every token after appears as you're reading it (as you must have observed in chat applications).
usually though, streaming each token as it's coming isn't a good idea because the words and sentences wouldn't make sense. for example when building voice agents: you can't hand raw individual tokens to a text to speech engine and expect it to sound natural. so we start chunking, we buffer the stream until we hit a comma or a full-stop for example, then flush that chunk to speech, while the model generates the next chunk to speak.
batching
since decoding is memory bandwidth bound and not compute bound, the gpus themselves are sitting mostly idle at this stage, waiting on a kv cache read from memory. since we're paying the memory anyways, we might as well pull the kv caches for multiple requests at once and generate the next token for them in the same breath instead of dealing with one request at a time, amortizing an expensive read across more useful work per trip.
the old way of doing this was static batching, so we lock a fixed group of requests and don't release any of them until the whole group finishes. this falls apart fast when the output lengths vary, which they always do ofc. the batch is as slow as the longest request i.e the one with the most tokens, a classic case of head-of-line blocking.
the fix is continuous batching, so the batch gets reassembled at every single decode step instead of at fixed boundaries. a request finishes and a new request waiting in line slides into the freed slot the very next moment (kinda like connection pooling). this is how most llm providers work, and how they are able to serve so many people so fast at the same time.
the tradeoff is latency since in exchange for better throughput i.e more tokens being served per second to more people there is slightly worse latency per request. valid tradeoff to make in this situation.
fancy words people throw around
the machine learning and ai space has nomenclature which was defined by people who grew up on star wars and it's not as straightforward as the terms we're familiar with in the backend space. i love that though.
so this is a quick detour through some important words which you should know about but i won't be going deep into cause that's not what this blog is about.
start with inference. it just means running the model to get an output instead of training it. everything related to serving you've read in this post so far, tokenizing, prefilling, decoding, that's all inference. so when someone says inference costs, they are referring to the cost of serving these requests and not training the model.
attention is the mechanism that lets the model figure out how relevant every other token is to the one the model is currently processing. you don't need the math behind it but we already have a gist of why as context grows, the model takes more time to process the next token and also the quality of the response degrades because the model now has to do more work to figure out what's actually relevant to all the tokens you hand to it.
every token has to weigh its relevance against every other token, that's why processing a prompt is quadratic, double your context and prefill does four times the work. each decode step after that grows linearly with context, but stack up a whole response and a whole conversation and the totals get quadratic fast.
weights are the actual numbers baked into the model from training, this is what people mean by "a 100b model," 100 billion of these numbers, and they're the same thing as what gets called the parameter count.
you'll often hear about rlhf (reinforcement learning from human feedback). the rough outline of it is that after a model's base training, which is just predicting the next token on a huge pile of text and makes it really good at just completing text, humans rank a bunch of model's outputs against each other for the same prompt. the preference data trains a separate reward model to score outputs just the way humans scored them. the original model then gets fine-tuned again, using reinforcement learning, to produce outputs that score well against the reward model. this cycle repeats until we get something which is actually useful to us.
distillation is training a smaller model (student) to mimic a bigger model's (teacher) outputs, this is how we get very small cheap models (like a qwen or llama) which are surprisingly good at their specified tasks. this helps us because now we can get better results than a same sized model which was trained on hard labels alone instead of a richer signal and at a fraction of the inference cost of a larger model but with similar capability.
quantization is the shrinking of the precision of weights instead of reducing the total weights themselves like in distillation. this is how running big models is possible on consumer gpus. an FP16 7B model needs ~14GB just for weights, quantized to INT4, that drops to ~3.5-4GB. we're trading off quality and precision for lower size and better inference latency.
hallucination is probably the single most overused buzzword in the space right now and very often incorrectly used to blame all issues pertaining to llms. it's not a bug in the traditional sense, in fact there's decent theoretical work arguing they're basically inevitable just because of the nature of these glorified probabilistic token predictors, openai's why language models hallucinate is worth a read if you want the actual argument. it's the model producing something wrong very confidently and fluently, which is just what generation looks like when the next highly likely token happens to be untrue. we can however, mitigate hallucinations. something like asking the model to cross-reference its results against a given source, asking it to explain its chain of thought or simply just asking it to not hallucinate works like banging on a broken tv.
last one for this detour, agentic, the adjective everyone's currently obsessed with slapping onto every product description. more on that in a bit.
some parameters to take note of
these are the runtime knobs, config flags that tune how the model behaves for each specific call without touching its weights. same model gives you different results purely based on the parameters you pass in the request.
temperature controls how sharp or flat the model's probability distribution is before picking its next token. in english, how much freedom are we giving the model to say something novel or dumb. low temp means picking the highest probability token, so we get a similar predictable response for the same request. high temp means that even the lower probability tokens get a real shot and you get more varied responses. zero temperature doesn't technically guarantee determinism since floating point operations on gpus aren't perfectly reproducible across different runs but it's pretty damn close to it.
top_p (nucleus sampling) trims the same distribution a different way, instead of reshaping the whole curve like temperature, it only samples from the smallest set of tokens whose combined probability crosses p. yes, that didn't mean anything to me either.
in english: a low top_p(0.1) makes the model pick only from a pool of safe obvious words. good for stuff like data extraction or any place where there is an obvious correct answer. a high top_p(0.99) means the model is willing to reach further down the list to pick the weirder "creative" responses and this means a higher chance of the model saying something which you didn't want it to.
often top_p and temperature are used one at a time in practice since tuning both the knobs which exist for a similar function makes it harder to know which knob caused a given change in output.
max_tokens is the hard ceiling for how much the model can generate. this is usually why your model might just cutoff mid sentence and it's often nice to set since tokens cost money and you should have an idea of the kind of responses you're expecting for the kind of requests you're making in your application.
stop sequences tell the model to cut generation as soon as it produces a specific string. useful in places where you know when to stop the model.
choosing a model
highly depends on your budget and the task at hand, whether you are hosting locally or paying for api credits. i won't go into whether claude or gpt is better because that info is not gonna hold up for longer than a week.
few things you need to look into:
context window. the number everyone looks at first. a bigger window means more token capacity, it doesn't mean the model attends to all of it equally well though. quality tends to degrade the further back something sits in a long context, same attention cost story from the fancy words section. a bigger window buys you capacity, not attention quality.
parameter count. a higher parameter count isn't necessarily a decent proxy for better quality. the biggest reason not to trust it is that a lot of frontier models are mixture of experts now, meaning the model might have 400b total parameters but only activate a small fraction of them, say 40b, for any given token. so two models both labeled "400b" can have completely different actual compute costs per token. the number on the label doesn't even reliably tell you the cost anymore, let alone the quality.
latency, cost and quality. generally, a frontier model will get you quality but will be slower and cost more. a smaller model will be cheaper but will come at the cost of raw capability. for a lot of production tasks, especially the highly specific narrow ones, the cheaper and faster model is the correct engineering call and always reaching for the big guns is counterproductive.
rag vs fine-tuning
the way i like to think of it is making a complex sql query or baking the data into the schema itself. rag is a query, you retrieve relevant context at request time and stuff that into the prompt, cheap and easy, the model's actual behavior doesn't change. fine-tuning is closer to baking something into the schema, you're changing the model's actual weights to tune its behavior or style the way you want. expensive but the change persists across requests without needing special context every time.
general rule of thumb is that if the problem is the model doesn't know this fact, that's rag. if the problem is the model doesn't behave the way i want it to, that's fine tuning. though you can get pretty close to fine-tuning results with rag itself.
tool use and agents
function calling isn't one single mechanism across every provider. some serving stacks literally mask which tokens are legal at each decode step (constrained decoding), forcing the model onto rails so it physically cannot generate a token that breaks the schema. others just fine-tune hard on schema-following and validate the output after the fact. this is why tool calls are reliable but not infallible, and why you still see the occasional malformed call in production.
agents are the buzzword of the decade and all they are is an llm with access to tool use in a loop (ReAct loop: reason, action, observation, repeat). i've already talked extensively about agents in my previous blog about an agent development kit i built so check that out if you wanna go deeper.
one thing which people often tend to confuse is that one request made to an agent is not one model request that internally does stuff, it's multiple hops to the llm provider in a loop. n hops means n requests, and each one pays its own queue wait, its own prefill, its own ttft, its own decode, on its own clock. a 4 hop tool chain isn't one request's worth of latency, it's four requests' worth stacked one after another. a fast single request tells you nothing about how slow a multi hop tool chain will actually feel in production.
so basically
none of the stuff we discussed here will make you qualified to work at an ai lab, but it does make you qualified to nod at the right moments, correct someone about batching, and larp with confidence. informed larping is just understanding with humility. hope you got enough ball knowledge out of this one.
thanks for reading. see ya :)
if you found this write-up useful, feel free to fund my caffeine addiction: