AI Cost Starts with Architecture

I do not think AI cost is just a finance problem.

It is mostly an architecture problem.

If you are building an AI product, the cost starts with decisions your engineering team makes every day. Which model handles a request? How much information does it receive? How many times does it run? What happens when the first answer is wrong?

By the time the monthly bill arrives, those decisions are already in production. You can negotiate a better price, and that may help. But you still need to understand why your system is doing that much work.

Match the Model to the Task

Imagine a company building a support assistant. A customer asks where their order is. The application sends the conversation, account history, and company policies to its strongest model. Then another model checks the answer.

Maybe the customer only needed an order status from a database. Before comparing model prices, I would check whether this request needed generation at all.

Now consider the requests that do need a model. Classifying a message, extracting an order number, and investigating an unusual complaint are different tasks. I would not assume they all need the same level of capability.

Test a smaller model on the routine tasks. Keep the stronger model for cases where it makes a meaningful difference. The quality requirement should come from the application and the mistakes you need to avoid.

There is research behind this. RouteLLM, published at ICLR 2025, learned to select between stronger and weaker models. Its benchmark results reported more than twofold cost savings while largely preserving response quality. That is encouraging. It does not establish what your production savings will be.

There is also a difference between choosing a model before generation and trying a smaller model first. The second approach may require checking its answer and calling a stronger model afterward. If that happens frequently, you are paying for both attempts and adding another sequential step before the customer gets an answer.

This is why evaluating the whole application matters. A model can be cheaper per call and still make the product more expensive if customers repeat their questions or someone has to fix its answers.

I would also keep model selection easy to change. The principle of building for replaceability applies here. If testing shows a better combination of cost, quality, and latency, your team should be able to use it without rebuilding the product.

Stop Repeating Work

Suppose every support request includes the same long instructions and reference documents. Prompt caching can reuse processing of that repeated input. Anthropic’s documentation explains how stable prompt prefixes make this possible, while also setting minimum lengths, expiration rules, and cache-write charges.

So check whether requests actually reuse the cached content. A feature being enabled does not tell you whether it is saving money.

Caching a finished answer is a separate decision. If an approved answer remains valid for a repeated question, the system may serve it without another generation call. AWS includes this pattern in its production guidance. But account-specific information and changing policies need explicit rules for freshness and access.

The context you send deserves attention too. Does our support assistant need the complete conversation to extract one order number? Maybe it does in some cases. Test that. Adding more documents should have a reason you can explain.

Decide What Can Wait

The work a user waits for is usually called the request path. Our customer needs an answer about their order. Preparing a management report or enriching documents for tomorrow can happen separately. Microsoft’s asynchronous request-reply pattern describes acknowledging work, processing it independently, and exposing its status.

Some of that background work can use batch processing. Google’s Gemini Batch API advertises 50 percent of the standard cost, with a target turnaround of 24 hours. That can fit evaluations, document processing, or other work without an immediate-response requirement.

Putting a task in a queue does not automatically make its model calls cheaper. You need an execution mode that offers the saving. You also need to handle failures and tell people when the work is complete.

Every Extra Step Has a Cost

Agents make these decisions even more important. A request can turn into planning, several tool calls, reviews, and further attempts. Anthropic recommends starting with the simplest workable solution and adding complexity when it improves the outcome. For a predictable task, a defined workflow may be enough.

In its research-system report, Anthropic observed that agents used roughly four times the tokens of chat interactions, and multi-agent systems roughly fifteen times. Those are observations from its own workloads. They are not universal billing ratios. But they show why additional steps need to justify their cost.

Retries need limits too. AWS warns that retries at multiple layers can compound and worsen an overloaded system. It recommends bounded retries with increasing delays and randomized timing. For our support assistant, I would set a deadline and decide what happens when it cannot complete the request.

If you host models yourself, model precision becomes another architecture choice. The Llama quantization tutorial covers reducing precision and serving the model with vLLM. Memory requirements, throughput, and deployment settings then become part of the cost discussion.

But operating that infrastructure takes engineering time. A lower infrastructure bill may still be a poor tradeoff if keeping it running consumes the team that should be building your product.

Measure the Cost of a Resolution

To judge all these decisions, I would measure cost per successful outcome. The FinOps Foundation calls this use-case economics. For our assistant, that could mean cost per resolved support request, including failed attempts, supporting infrastructure, and human correction where relevant.

Then trace the expensive and unsuccessful cases. Did the request reach the wrong model? Did retrieval send unnecessary context? Did the workflow keep retrying? Those questions tell your team what to investigate.

Finance should help define budgets and understand margins. Engineering should be able to explain what each workflow consumes and why. Traffic, prices, and contracts still affect the bill. Architecture determines how the product responds to them.

If you are responsible for an AI product, bring these decisions into the design review:

• Use the least costly model that meets the quality requirement.
• Reuse work while it remains valid.
• Move non-urgent work out of the request path.
• Set limits on tool calls, retries, and escalation.
• Measure cost per successful resolution.

Then test whether your choices deliver the result your users need.

An Architecture Review Map

Review how much work the system performs and how it runs that work. Both affect the cost of a successful resolution.

The monthly bill is useful feedback. Many of the decisions behind it were made much earlier.