LLM cost optimization: how to cut inference spend without wrecking product quality

Why LLM costs sneak up on early AI SaaS teams

LLM cost optimization matters early because inference cost behaves differently from every other line item a SaaS founder is used to managing. Hosting a web app scales sublinearly with users because a lot of infrastructure is shared. Inference cost scales close to linearly with usage, sometimes worse than linearly if a feature involves loops, retries, or agentic tool calls that trigger multiple model calls per user action. A support chatbot that costs three cents per conversation looks irrelevant at fifty conversations a day and looks like a real number at five thousand.

The second reason costs sneak up is that the demo path and the production path are usually not the same path. A demo is one clean prompt with a short context window. Production adds retrieved documents, conversation history, system instructions, retry logic for malformed outputs, and often a second model call to validate or format the first one's response. Each of those additions is individually reasonable and collectively expensive. Nobody sat down and decided to spend five times more per request than the demo suggested, it accumulated one sensible-looking pull request at a time.

The fix is not to avoid these additions, most of them genuinely improve the product. The fix is to price and measure them as they go in, the same way a hardware company would cost out a bill of materials before shipping a product, rather than discovering the true cost after the invoice arrives.

Model routing: send the request to the cheapest model that can do the job

Model routing is the practice of maintaining more than one model behind a feature and choosing which one handles a given request based on how difficult that request actually is. Most AI products default to calling the most capable model available for every request, because it is the simplest thing to build first, and that default is frequently the largest source of avoidable spend in the whole system.

A workable routing setup usually has three tiers: a small, fast, cheap model for classification, extraction, and simple rewrites, a mid-tier model for the bulk of normal user requests, and a frontier model reserved for requests that are genuinely hard or that a cheaper model has already failed on. The routing decision itself can be as simple as a rule based on input length and task type, or as sophisticated as a lightweight classifier that predicts task difficulty before the main call happens. Either approach beats sending everything to the most expensive option by default.

A useful way to find routing opportunities is to sample a week of real production requests and manually check what the cheapest available model would have produced for each one. In most AI SaaS products, a large share of traffic, often the majority, is simple enough that a small model handles it just as well as a frontier one, and the cost difference between tiers is commonly five to twenty times per token. Routing even sixty percent of traffic to a cheaper tier can cut total inference spend by more than half while leaving output quality unchanged for the user, because the requests that got downgraded were never hard enough to need the expensive model in the first place.

Rule of thumbStart routing with the requests you are most confident are easy, like intent classification, short summarization, and structured extraction. Save the frontier model for open-ended reasoning, long-form generation, and anything a cheap model has already failed at once.

Caching: stop paying twice for the same answer

Caching, in the context of an LLM product, means storing the output of a previous request so that an identical or near-identical future request can be served from storage instead of triggering a new paid model call. It sounds obvious, but a surprising number of AI SaaS products call the model fresh on every single request even when the same question, the same document, or the same input is seen repeatedly.

Caching pays off fastest on features with a small number of frequently asked questions, shared reference documents, or repeated onboarding flows. It pays off least on features that are genuinely unique per request, like personalized writing generation, where two requests rarely look alike enough to share a cached response.

• ▸Exact match caching: hash the full prompt (system instructions plus user input plus any retrieved context) and store the response keyed on that hash. This catches literal repeats, which are common for FAQ-style features, support bots answering the same top questions, and any feature that reprocesses the same input document more than once.
• ▸Semantic caching: instead of an exact hash, embed the incoming request and compare it against recently cached requests using vector similarity, serving the cached answer when similarity crosses a threshold. This catches near-duplicates like slightly reworded questions, which exact match caching misses entirely.
• ▸Prompt caching offered directly by model providers: several providers now let you mark a portion of a prompt, typically the system instructions and any static context, as cacheable, so repeated calls with the same prefix are billed at a steep discount for that portion. This is close to free to implement if your system prompt is long and stable, which it usually is.
• ▸Cache invalidation discipline: set a sensible time-to-live for cached responses and invalidate immediately when the underlying data changes, for example when a document a feature answers questions about gets edited. A stale cached answer is worse than a slow correct one.

Prompt compression: pay for fewer tokens without losing capability

Prompt compression is the practice of reducing the number of tokens sent to and returned from a model on each call while preserving the information the model actually needs to do the task well. Since most API pricing is per token, a shorter prompt is a direct, immediate cost reduction on every single call, with no infrastructure change required.

• ▸Trim the system prompt: long system prompts accumulate over months of iteration as instructions get added and old ones never get removed. Reread it and cut anything the model would produce correctly without being told, and anything that duplicates instructions already stated elsewhere.
• ▸Limit conversation history: sending the full chat history on every turn of a multi-turn conversation gets expensive fast as conversations get longer. Summarize older turns into a short recap instead of resending them verbatim, refreshing the summary every few turns.
• ▸Retrieve less, retrieve better: if a feature uses retrieval augmented generation, sending ten loosely relevant document chunks instead of three tightly relevant ones inflates cost with no quality benefit and sometimes hurts quality by diluting the model's attention with noise.
• ▸Cut redundant few-shot examples: few-shot examples in a prompt are often copied from an early prototype and never revisited. Test whether the model performs just as well with two examples instead of six, which is common once the underlying model has improved since the examples were written.
• ▸Ask for shorter outputs when a shorter output is genuinely sufficient: output tokens are typically billed at a higher rate than input tokens, so an instruction to answer in two sentences instead of a default long-form response saves money on every reply, not just on the prompt side.

Rule of thumbTrack total tokens per feature over time, not just dollars, because token count is the number you can directly act on with a prompt edit, and dollar totals mix in provider pricing changes that have nothing to do with your product decisions.

Batching and streaming solve different problems

Batching means grouping many requests together and processing them asynchronously, usually at a discounted rate offered by the provider for work that does not need an immediate response. Streaming means returning a model's output token by token as it is generated, rather than waiting for the full response before showing anything to the user. Founders sometimes conflate the two because both involve how responses are delivered, but they solve different problems and are not substitutes for each other.

Batching is the right lever for anything that does not need to happen in real time: nightly summarization of the day's activity, bulk re-scoring of a dataset after a prompt change, generating embeddings for a backlog of documents, or sending overnight digest emails. Major providers offer batch APIs at a meaningful discount, commonly around half the price of the standard synchronous endpoint, in exchange for results arriving within a window of a few hours instead of instantly. Any feature currently running on a schedule or a cron job through the normal API is a candidate for moving to batch pricing.

Streaming does not reduce cost at all, the total tokens billed are the same whether they arrive all at once or one at a time. What streaming buys you is perceived speed: a user watching a response appear word by word feels like they waited less time than a user staring at a blank screen for the same number of seconds. Streaming is a user experience investment, not a cost optimization, and it is worth implementing for that reason even though it belongs in a different conversation from the one about your bill.

Usage caps and rate limits are a margin protection tool, not a growth restriction

A usage cap is a defined limit on how much of a metered resource, in this case model calls or tokens, a customer on a given plan can consume before being throttled, charged an overage, or asked to upgrade. Founders often resist adding usage caps because they worry it looks stingy or hurts the free trial experience, but without a cap, a single unusually heavy user on a flat-rate plan can consume more in inference cost in a month than that customer paid you, and there is no natural ceiling stopping it from happening again the next month.

• ▸Set caps at the plan level based on the cost of goods you can tolerate for that price point, not on a round number that feels generous. Work backward from your target gross margin, not forward from what sounds nice in a pricing table.
• ▸Differentiate soft caps from hard caps: a soft cap slows a user down or nudges them to upgrade, a hard cap stops further usage until the next billing cycle or an upgrade. Free and trial tiers should generally use hard caps, since there is no revenue offsetting cost overrun on a free plan.
• ▸Monitor for abuse patterns separately from genuine heavy usage. A legitimate power user hitting a cap is a sales conversation. A script hammering your API with automated requests is a security and cost problem that needs a different response, typically rate limiting by request frequency rather than just total volume.
• ▸Communicate caps clearly before a customer hits them, with a visible usage meter inside the product. A surprise throttle after the fact damages trust far more than a visible limit stated up front.

Per-user margin math: know your real cost of goods sold

Per-user margin math is the exercise of calculating gross margin for an individual customer's actual usage pattern, rather than for an average or blended customer across the whole product. AI products need this calculation more urgently than most SaaS products because the cost of serving two customers on the same plan, doing what looks like the same job, can differ by an order of magnitude depending on how they use the product.

Start by identifying every model call a typical customer action triggers, including calls that happen invisibly, like a background classification step, a moderation check, or a retry after a malformed response. Multiply each call's estimated token count by current provider pricing to get a cost per action, then multiply by how many times an average customer on each plan performs that action per billing period. Compare that total to what the customer pays. If the number is uncomfortably close to the plan price, or above it for your heaviest users, you have found the segment where growth is actively working against you rather than for you.

Do this exercise separately for at least three customer profiles per plan: a light user, a typical user, and your heaviest realistic user, not just an average across all of them. A blended average margin can look perfectly healthy while a specific segment of your customer base is quietly unprofitable and getting more expensive to serve every month as they grow more comfortable with the product and use it more.

Rule of thumbIf you are building or refining pricing tiers around these numbers, it is worth revisiting how the tiers themselves are structured, not just what they cost internally to deliver, alongside the cost work above.

When fine-tuning actually saves money, and when it does not

Fine-tuning is the process of further training a pretrained model on your own examples so it performs a specific task with a shorter prompt or more reliable output format than prompting alone would achieve. Founders sometimes reach for fine-tuning as a cost optimization technique because it can shrink the prompt needed to get a desired behavior, but it is a narrower tool than it is usually treated as, and it comes with real ongoing costs of its own.

Fine-tuning tends to save money when the task is narrow, repetitive, and currently requires a long, detailed prompt with many examples to get right, because a fine-tuned model can often achieve the same behavior with a much shorter prompt, cutting input token cost on every call. It also tends to help when output format consistency is the main problem, since a fine-tuned model is more reliable about following a specific structure without needing extensive formatting instructions repeated in every prompt.

Fine-tuning tends not to help, and can actively cost more, when the underlying task still requires broad reasoning, when your product needs to keep up with rapid improvements in general-purpose models, or when you do not yet have a large, clean, representative set of examples to train on. A fine-tuned model is locked to a specific base model version, and every time the underlying provider ships a meaningfully better general model, you face a choice between re-running the fine-tuning process or falling behind on quality while competitors on prompting alone upgrade for free. For most early AI SaaS teams, prompt engineering plus routing plus caching solves the majority of cost problems before fine-tuning is worth the added maintenance burden. Reach for it once you have a stable, high-volume, narrow task and have already exhausted the cheaper levers.

Monitoring cost per action instead of total monthly spend

Cost per action is the average inference cost required to complete one meaningful unit of product usage, such as one support ticket resolved, one document summarized, or one generated draft delivered. It is a more useful number to track day to day than total monthly spend, because total spend conflates cost efficiency with growth, and a founder watching only the total can make exactly the wrong decision at exactly the wrong moment.

Building this instrumentation is a few hours of work if you do it before launch and a much larger project if you do it after a founder has already been surprised by an invoice. It is also one of the more common gaps we see in products going through review on LaunchLoop: the demo is strong and the pricing page is confident, but nobody in the team can answer what a single core action actually costs to run, which is a hard question a sharp early customer or investor will eventually ask.

• ▸Log the model, token counts, and estimated cost for every call at the point it happens, tagged with the feature and action it belongs to, rather than trying to reconstruct this later from a provider invoice that only shows totals.
• ▸Chart cost per action over time by feature. A rising total spend alongside a falling or flat cost per action means the business is growing and getting more efficient at the same time, which is a healthy pattern worth celebrating rather than panicking about.
• ▸Chart cost per action against the plan price for the customer segment that uses that feature most, so you can see gross margin trending in the direction you want before it becomes a problem, rather than after.
• ▸Set an internal alert threshold for cost per action per feature, so a regression, like a prompt change that accidentally triples token usage or a new bug that causes retry loops, gets caught within days instead of surfacing a month later as an unexplained spike in the bill.

A simple order of operations for a team with limited engineering time

Cost optimization work competes with feature work for the same limited engineering hours, so sequencing matters. Doing the highest-leverage, lowest-effort items first protects margin quickly without derailing the roadmap.

• ▸Week one: add per-request cost logging, even if it is just a line written to a database on every model call. You cannot optimize what you cannot see.
• ▸Week one or two: trim the system prompt and any obviously stale few-shot examples across your top three features by call volume. This is nearly free and immediate.
• ▸Week two or three: add exact-match or provider-level prompt caching to any feature with repeated or templated prompts, especially support and onboarding flows.
• ▸Week three or four: introduce a two-tier model routing setup for at least one high-volume feature, sending simple requests to a cheaper model.
• ▸Month two: move any non-urgent, bulk, or scheduled work to a batch API if your provider offers one.
• ▸Month two or three: run the per-user margin math on your three heaviest customer profiles and set or adjust usage caps accordingly before it becomes urgent.
• ▸Ongoing: revisit fine-tuning only after the above levers are in place and you have identified a specific, narrow, high-volume task where a shorter fine-tuned prompt would meaningfully cut cost.