[Topic]
Cost & efficiency
Where LLM spend goes and how smaller tuned models, distillation, caching and serving choices reduce it.
PlaybookIntermediate
How to reduce enterprise LLM costs without sacrificing quality
Most LLM spend comes from a few repeated patterns: long prompts resent on every call, verbose outputs, retries, and agents that multiply calls. Measure cost per successful task first. Then apply the cheap levers (output limits, prompt caching, batch APIs) before the expensive ones (routing, smaller tuned models, self-hosting), and re-run a quality eval after every change.
ExplainerAdvanced
Model distillation: building smaller, cheaper, specialized models
Distillation trains a smaller student model to reproduce a larger teacher's behavior. Today that usually means fine-tuning the student on filtered teacher outputs and, increasingly, letting the teacher grade the student's own samples. It works best on narrow, well-measured tasks, and commercial API terms can restrict using a provider's services to build competing models.