The terms I use, defined once and used the same way everywhere.
Total spend on an AI feature (model tokens, retrieval, infrastructure, evals, retries and human review) divided by the number of requests that produced a correct, usable result. Unlike cost per request, it counts failures and rework, so it shows what each useful answer really costs the business.
Why it matters: a cheaper model that fails more often can raise this number while lowering cost per request.
A fixed set of real inputs, usually taken from production traces, each paired with the expected output or a pass/fail rule. Teams run it against every prompt, model or configuration change to see whether quality held. It is small enough to run often and stable enough to compare results over time.
Why it matters: without one, a cost cut can't be shown to be safe.
An automated check in CI or deployment that runs the golden eval set against a proposed change and blocks it if the pass rate drops below an agreed threshold. It turns "did quality hold?" from a judgement call into a pass/fail step, so cost changes ship only when the evidence supports them.
Why it matters: it lets teams cut costs quickly without quietly breaking quality.
Sending each request to the cheapest model that can handle it, based on the task, the input or a classifier, instead of sending everything to the largest model. A router might send simple lookups to a small model and keep the large model for multi-step reasoning, with the golden eval set confirming each route still passes.
Why it matters: it is often the largest single saving in an LLM feature, and the easiest one to get wrong without evals.
Tracing each LLM request end to end (prompt, retrieved context, model, tokens, latency, cost, retries and the outcome) so a team can see what a feature does in production and why it fails. It joins cost data and quality data per feature, rather than per provider invoice.
Why it matters: you can't fix failures or costs you can only see as a monthly total.
Reporting cloud and AI costs back to the teams, features or customers that caused them, without actually charging their budgets (that is chargeback). It relies on tagging and allocation, so each LLM feature's spend is visible to the people who can change it.
Why it matters: you can't cut what nobody can see is theirs.
Articles on building LLM systems that hold up in production, and what they cost. Each one by email as it goes live; nothing else.
Articles are published on Hashnode.
