How to reduce enterprise LLM costs without sacrificing quality

To reduce LLM costs without losing quality, first measure cost per successful task. Then work through the levers in order of effort: cap output length, cache repeated prompt prefixes, batch work that can wait, route easy requests to cheaper models, and only then consider smaller tuned models or self-hosting. Re-run the same quality evaluation after every change, or the savings may simply be a worse product.

The order matters. The cheap levers are low-risk and quick to reverse. The expensive ones (new models, new infrastructure) can save more but take longer and are easier to get wrong. Teams that jump straight to “let’s self-host” often haven’t yet captured the batch and cache discounts already available on their current API.

Where does LLM spend actually come from?

Every API bill reduces to the same formula. For one task:

cost per task = Σ over calls [ uncached_input × p_in
                             + cached_input   × p_cache_read
                             + cache_writes   × p_cache_write
                             + output         × p_out ]
cost per successful task = total cost / tasks that met the quality bar

Five patterns drive most of it:

• Input tokens resent on every call. System prompts, instructions, few-shot examples, retrieved documents and tool schemas often dwarf the user’s message, and they are identical from call to call.
• Output tokens. They usually cost several times more than input. On Anthropic’s list prices as of November 2026, output costs five times input for Claude Opus 5.5 ($4 vs $20 per million tokens), Sonnet 5.5 ($2 vs $10) and Haiku 5.5 ($0.10 vs $0.50 for prompts up to 100K tokens), according to its pricing page. Verbose answers, unrequested explanations and long reasoning traces add up fast.
• Calls per task. A pipeline that classifies, retrieves, drafts, critiques and rewrites makes five calls where a user sees one answer.
• Retries. Malformed JSON, failed validations and timeouts trigger repeat calls. A 20% retry rate is a 20% surcharge, and it usually signals a quality problem rather than a cost problem.
• Agent loops. An agent re-reads its growing context at every step. If step k resends everything from steps 1 to k−1, input tokens grow roughly with the square of the number of steps. Anthropic reported that its agents typically use about 4x more tokens than chat interactions, and its multi-agent systems about 15x (Anthropic engineering).

Step 0: measure before you cut

You can’t cut what you can’t attribute. Before changing anything:

1. Log tokens per call by feature. Record input, cached input and output tokens, the model, latency and the outcome for every call, tagged with the product feature and the task id. All major APIs return cached-token counts in the usage object.
2. Compute cost per successful task for each feature. A cheap call that fails half the time is expensive.
3. Freeze a quality baseline. Assemble a few hundred real, representative requests with expected outputs or a scoring rubric, and record today’s score. Every later change is judged against it. How to build a custom evaluation framework explains how.
4. Rank features by spend. Usually a handful of features account for most of the bill. Start there.

Fill in this template for each top feature. The example column is hypothetical:

Line item	How to measure	Example (hypothetical)	Your value
Requests per month	request logs	2,000,000
Calls per request	traces	1.3 (incl. retries)
Input tokens per call (uncached / cached)	usage object	1,500 / 0
Output tokens per call	usage object	300
Price per 1M tokens: input / cached / output	provider pricing page, dated	from your contract
Success rate on quality eval	frozen eval set	92%
Monthly cost ÷ successful tasks	computed

Which cost levers should you pull first?

Lever	Effort	What it saves	Quality risk
Output limits and tighter formats	Hours	Output tokens	Low if evaluated
Prompt cleanup and compression	Days	Input tokens	Low to medium
Prompt caching	Days	Repeated input	None: same tokens, same output
Batch APIs	Days	50% on deferrable work	None; latency up to 24 hours
Routing or cascades	Weeks	Expensive-model calls	Medium: misroutes
Smaller tuned model or distillation	Weeks	Price per token	Medium: needs proof
Quantization (self-hosted)	Days to weeks	GPU memory and throughput	Low to medium
Self-hosting	Months	Price per token at high utilization	Operational

Cap output length and tighten formats

Ask for exactly the output you use. If downstream code reads three JSON fields, request those three fields with a schema, not a paragraph followed by JSON. Set max_tokens (or the equivalent) to a sensible ceiling for each feature. Remove “explain your reasoning” from prompts whose reasoning nobody reads. Reasoning sometimes improves accuracy, though, so measure before you remove it. Since output usually costs several times more than input, this is often the highest-return change per hour of work.

Clean up and compress prompts

Audit the static part of each prompt. Delete stale instructions, duplicated rules and few-shot examples that no longer change the score. Trim retrieved context to the passages that are actually used, and send tool results in summarized form rather than raw.

Automated compression exists too. LLMLingua reported up to 20x prompt compression with little performance loss on its benchmarks (GSM8K, BBH, ShareGPT, Arxiv). Compression works against caching, though: a compressed prompt that changes on every call won’t hit the cache. Compress the variable parts and cache the static ones.

Use prompt caching

Providers can skip reprocessing an identical prompt prefix and bill it at a discount. As of November 2026:

• OpenAI enables caching by default on supported models. The minimum cacheable prompt is 1,024 tokens for GPT-5.6 and later. OpenAI says cached input is “discounted up to 95%”: on GPT-5.6 and later, reads cost 0.1x the normal input rate on most models and 0.05x on a few. On those models, cache writes cost 1.25x for the default 30-minute lifetime; earlier models have no write charge (OpenAI prompt caching).
• Anthropic offers automatic caching with one top-level cache_control field, or up to four explicit breakpoints. Writes cost 1.25x the base input price for a 5-minute TTL and 2x for a 1-hour TTL. Reads cost 0.1x on most models (0.05x on Opus 5.5 and Sonnet 5.5). The minimum cacheable length ranges from 512 to 4,096 tokens depending on the model (Anthropic prompt caching).
• Google enables implicit caching by default for Gemini 2.5 and newer models, with minimums of 2,048 or 4,096 tokens depending on the model (Gemini context caching).

Worked example (Anthropic multipliers, 5-minute TTL). A 10,000-token static prefix sent twice costs 20,000 token-units uncached. With caching it costs 12,500 for the write plus 1,000 for one read, 13,500 in total. After ten reuses it costs 12,500 + 10 × 1,000 = 22,500 instead of 110,000. A cached prefix pays for itself on its first reuse within the TTL.

Caching only pays if prompts are built for it. Each provider’s guidance comes down to the same rules:

• Put stable instructions, tool definitions and shared reference material first.
• Put timestamps, user ids and per-request data last.
• Append new conversation turns rather than rewriting earlier ones.
• Keep tool definitions in a fixed order.

Check the cached-token counts in your logs. A hit rate near zero usually means something at the start of the prompt changes on every request.

Move deferrable work to batch APIs

Evaluations, backfills, nightly classification, document enrichment and synthetic data generation rarely need an answer within seconds. Both OpenAI’s Batch API and Anthropic’s Message Batches API charge 50% less than the synchronous endpoints, in exchange for completion within 24 hours. Anthropic says most of its batches finish within an hour. OpenAI runs batches against a separate, higher rate-limit pool. Anthropic states that batch and caching discounts stack, but cache hits in batches are best-effort because requests run concurrently. Anthropic reports typical hit rates of 30% to 98%.

Route easy requests to cheaper models

Traffic is rarely uniformly hard. Two research approaches are worth knowing:

• Cascades. Try a cheap model first, check the answer with a scorer or verifier, and escalate only when the check fails. FrugalGPT (Chen, Zaharia and Zou, 2023) reported matching the best individual model it tested with up to 98% lower cost on its benchmarks.
• Learned routers. A classifier decides, before any call, whether a request needs the strong model. RouteLLM (Ong et al., 2024) trained routers on human preference data and reported cost reductions of over 2x in some cases without lower response quality.

The headline numbers come from benchmark mixes, not your traffic. A cascade adds latency on escalated requests and needs a check you can trust. A router needs labeled examples of your easy and hard requests. Measure the misroute rate on your own evaluation set, because a misrouted hard request is a quality regression that customers will see.

The simplest routing decision is often the most valuable one: test whether the next tier down in the same model family passes your quality bar for a given feature. Many features run on a flagship model only because that was the default when they were built.

Use smaller tuned models and distillation

For a narrow, stable, high-volume task, a small model trained on that task can match a much larger general model there, at a fraction of the price per token. There are two ways to get there:

• Fine-tune an open-weight model on your own labeled data. The LoRA and QLoRA tutorial walks through it.
• Distill a large model’s behavior into a smaller one. See model distillation, including the terms-of-use limits on training with commercial model outputs.

Hosted fine-tuning of proprietary models is no longer a safe default. OpenAI is winding down its fine-tuning platform. It is closed to new users, and eligible existing customers can create training jobs until January 6, 2027, unless the base model shuts down earlier (OpenAI SFT guide). Open weights you control don’t disappear when a vendor changes its roadmap. The trade-offs are in open-weight vs proprietary LLMs.

A tuned model saves money only if it holds quality, so the work is mostly evaluation. This is the approach Attempts takes: it tunes an open-weight model on a client’s data, measures it against the client’s current prompt and model on the same test set, and recommends against training when prompting or retrieval already closes the gap. Lower cost is something to measure, not something to assume.

Quantize self-hosted models

When you serve your own weights, lower-precision formats reduce memory and can raise throughput. For example, AWQ uses 4-bit weight-only quantization that protects the most important weight channels, and its TinyChat implementation ran more than 3x faster than the Hugging Face FP16 implementation. Quality loss varies by model, method and task. Evaluate the quantized model on your test set as a separate candidate.

When does self-hosting an LLM pay off?

The core difference: an API charges per token, while self-hosting charges per GPU-hour whether or not requests arrive. Serving software matters too. vLLM’s PagedAttention reported 2–4x throughput over earlier serving systems at the same latency. Still, the deciding variable is utilization, the share of paid capacity that serves real traffic.

self-hosted cost per 1M tokens = (GPU $/hour × GPUs) / (throughput tokens/s × 3,600 × utilization) × 1,000,000

break-even utilization u* = (GPU $/hour × GPUs) / (throughput tokens/s × 3,600 × API blended $/token)

Self-hosting is cheaper per token only when your sustained utilization exceeds u*. “Throughput” here means the sustained tokens per second that your deployment achieves on your real input/output mix at your latency target. Measure it with a load test, don’t take it from a spec sheet.

Worked example. All infrastructure figures below are assumptions for illustration, not quotes.

• Traffic: 2M requests a month, each 1,500 input and 300 output tokens. That is 3.0B input and 0.6B output tokens a month, about 1,370 tokens/s on average. Peaks will be several times higher.
• API baseline (cited list prices, November 2026, before any caching or batch discount): on Claude Sonnet 5.5 at $2 input and $10 output per million tokens, the month costs 3,000 × $2 + 600 × $10 = $12,000, a blended $3.33 per million tokens. On Claude Haiku 5.5 ($0.10 / $0.50), the same traffic costs $600, a blended $0.17 per million.
• Self-hosted (assumptions): two GPUs at an assumed $3.00 per GPU-hour each, so $6.00 an hour or about $4,380 a month running 24/7. Assume a measured sustained throughput of 5,000 tokens/s for a fine-tuned small model at your latency target.

Utilization	Self-hosted $ per 1M tokens	vs Sonnet 5.5 ($3.33)	vs Haiku 5.5 ($0.17)
10%	6 / (5,000 × 3,600 × 0.10) × 10⁶ ≈ $3.33	break-even	~20x more expensive
27% (this traffic’s average)	≈ $1.23	~2.7x cheaper	~7x more expensive
60%	≈ $0.56	~6x cheaper	~3x more expensive

Under these assumptions, self-hosting beats the mid-tier API from about 10% utilization, and never beats the small API model’s list price at any realistic utilization. Caching or batch discounts on the API side would push the break-even point higher still. The comparison only counts if both candidates pass the same quality bar. A self-hosted 4B model that fails the eval saves nothing.

The formula leaves out several costs. Add them before you decide:

• Redundancy. You need at least two replicas for availability, plus headroom for peaks, which lowers average utilization.
• People. Engineering time for deployment, autoscaling, monitoring, security patching and on-call.
• Lifecycle. Evaluation and retraining when the data drifts, plus upgrades when better base models ship.
• Commitments. Reserved capacity lowers the hourly rate but locks in spend, so the utilization risk moves to you.

SOSP '23 | Efficient Memory Management for Large Language Model Serving with PagedAttentionvLLM authors, UC Berkeley · ACM SIGOPS · YouTube · 2024

Why must quality be measured on every cost change?

Every lever above can lower quality without anyone noticing at first:

• A truncated output drops a field.
• A compressed prompt loses an instruction.
• A router sends a hard contract question to a small model.
• A quantized model gets slightly worse at arithmetic.

None of these shows up on the invoice. They show up as support tickets, churn and retries. Retries feed back into cost, so a quality regression can erase the saving that caused it.

Make cost changes the way you make code changes:

• Gate each change on the frozen eval set. Report quality and cost per successful task side by side, with a confidence interval when the difference is small. How to measure whether fine-tuning actually improved your model covers the statistics.
• Change one lever at a time so you can attribute both the saving and any regression.
• Shadow or canary the change on a slice of live traffic before rolling it out fully, and keep the old path ready for rollback.
• Re-measure monthly. Prices, models and traffic mix change, and the right configuration changes with them.

A 30-day cost reduction plan

Week	Actions	Exit criterion
1	Instrument tokens by feature; compute cost per successful task; freeze eval sets for the top features	You can name the features that make up most of spend
2	Output caps and schemas, prompt cleanup, cache-friendly prompt layout	Cached-token share rising; quality unchanged on eval
3	Move deferrable jobs to batch; test the next-cheaper model tier per feature	Batch share of deferrable work near 100%
4	Prototype routing or a tuned small model for the largest stable feature; model the self-host break-even with measured throughput	A go/no-go decision backed by eval results and the cost model

Steps 1 to 3 are low-risk and quick to reverse. Treat step 4 as an experiment with a written hypothesis, a baseline and a stopping rule. That is how lower cost stays a measured result rather than an untested hope.