Selling tokens on an open marketplace is a different business from running inference for your own product, and the difference trips up good engineers constantly. The trap is that it looks like an optimization problem (pick the cheapest hardware, serve the model efficiently, undercut the listings) when it is really a demand problem with an optimization problem attached.
A profitable configuration nobody routes traffic to sells zero tokens. Zero tokens at any margin is zero.
What is the KV cache? When a model reads your prompt, every token it attends to produces a key and a value vector at every layer. Those vectors are the model’s working memory of the conversation so far: they are what lets token 900,000 attend to token 12. Together they are the KV cache. It is not the model weights, which are fixed and shared across every request; it is per-session state. It grows linearly with context length, and at long context it dwarfs the weights. Throw it away and the only way to get it back is to run the entire prompt through the model again. That re-run is called prefill.
Start from Demand, Not from Hardware
The natural engineering instinct is to start with the fleet: here is the hardware I have or can buy; what can I serve profitably on it? That sequence produces configurations that are locally optimal and commercially dead.
Invert it. Start from what the market is actually routing:
- Which models have live, sustained demand?
- What is the current price floor on each?
- Can I serve one of those, below that floor, at a margin I will accept?
Only then does hardware selection begin. Hardware is the last decision, not the first, because it has the most flexibility and the least information about whether you will sell anything.
Working Through It, Step By Step
The calculator at open.lightbits.ai is built for exactly this sequence.
Step 1: pick the model from the demand-ranked dropdown. The list is ordered by live routed demand, not by benchmark score or release date. Start at the top and work down until you find something you can credibly serve. Resist picking the model you find most interesting; pick the one with traffic.
Step 2: describe the fleet. Either a cloud instance type (if you are renting) or a hardware BOM (if you are buying). Be honest about the BOM: include the network and the storage, not just the cards. The most common modelling error here is costing eight GPUs and forgetting the fabric that connects them.
Step 3: set a list price and read your rank. The tool ranks your price against live listings and estimates your capture at that price. This is the step that kills bad ideas fastest. A configuration that is profitable at rank 40 will sell nothing, and it is much better to learn that here than after the hardware arrives.
Step 4: read cost per 1M tokens, with and without cache tiering, then find your floor. The difference between the two lines is the recomputation you are currently paying for. Then work the other direction: ask the tool for the lowest price at which you still clear a 30% margin. That number is your real competitive position: not what you would like to charge, but how low you could go and survive. Knowing it tells you whether you are in a price war you can win.
Step 5: press Auto-optimize. Set the objective to maximum profit and give it a fixed budget, and it will search across model, quantization, and hardware jointly. This matters because the three interact: a quantization that halves memory may let you serve a larger model on the same cards and land in a higher-demand tier, which no amount of optimizing one variable at a time would find.
Why Cache Reuse is Unusually Valuable on Marketplaces
The general argument for tiering the KV cache is that returning sessions skip prefill. On marketplace traffic, that argument is stronger than average, and it’s worth understanding because it determines how much of the benefit you will actually capture.
A large and growing share of routed marketplace traffic is agentic. Coding agents, research agents, tool-using loops: these do not send one prompt and collect one answer. They send a trajectory, and on every step they resend the entire trajectory so far plus the newest observation.
Step 20 of an agent loop is a prompt containing all of steps 1 through 19. On a discard-the-cache architecture, each of those steps prefills the full accumulated history from scratch. The same prefix is recomputed twenty times in a session, each time slightly longer than the last.
This is close to the ideal case for reuse: near-total prefix overlap, short intervals between turns, and a prefix that grows monotonically. The reuse share on agentic traffic is high, which means the prefill saving applies to most of the work rather than a corner of it.
And the trajectory keeps growing, which compounds it. Prefill cost carries a term that grows with the square of the context, so step 20 of an agent loop is not merely one more prefill; it is the most expensive one in the session. That is also why the savings grow as the loop runs: published joint results with FarmGPU and ScaleFlux put time-to-first-token improvements between 100x and 280x across models from 131K to 1M tokens, and beyond 1000x at 10M tokens. For the whole turn, the share that reaches your throughput is the prefill share of turn time, which, on long agentic sessions, is most of it. Model it with your measured reuse share and your measured context growth, not with a number from a blog post.
Calibrate Before You Trust It
Every number above is a simulation. The calculator is parameterized by measured benchmarks on reference hardware, and your fleet is not reference hardware. Your interconnect, storage, quantization, and traffic shape all move the result.
The .pop probe exists for this. It is a short profiling run against your own fleet and your own traffic, and feeding its output back into the calculator roughly halves the error band on the projections. If you are about to make a capital decision on these numbers, run it first. It is a small amount of work against a large commitment.
Methodology note. The calculator is a simulation parameterized by measured benchmarks, not a measurement of your fleet. Treat its output as a hypothesis worth testing, and validate with a proof of value before committing capital.
Common Ways This Goes Wrong
Optimizing a model with no demand. Covered above, and still the most common failure. Check routed demand before anything else.
Costing the GPUs and forgetting the rest. Fabric, storage, power delivery, and the facility all land in cost per million tokens. Leaving them out produces a floor price you cannot actually survive at, which is a bad thing to discover after you have published it.
Assuming your reuse share. It is workload-specific and the single largest driver of benefit. Measure it.
Quoting a multiple without its sequence length. The gain grows with context, so a figure measured at 1M tokens does not describe your 8K traffic. Model at the context lengths you actually serve.
Racing to the floor without checking you can live there. Know your 30%-margin price before you enter a price war, not during.
Calibrate Before You Commit Capital
Run the five steps above with your real numbers. Then share the session link the calculator produces with whoever signs off on the capital, so they are looking at the same model you are rather than a summary of it.
If you want the error band tightened before that conversation, send the .pop output.
Look at open.lightbits.ai, I have <hardware>, <goal>. |
To learn more about optimizing KV cache for superior tokenomics, read the blogs: