I recently shared a short post:
Running a model locally removes an API call. It does not remove the work.
I want to expand on that, because getting a model to run and taking responsibility for it in production are different commitments.
Imagine a team building a document assistant. An engineer downloads a model, starts an inference server, and connects the application. The answers look good. Someone suggests replacing the external API.
That may be a reasonable decision.
But before making it, I would ask what happens after the demonstration.
Follow the Request into Production
One person submits a document. The model responds quickly.
Now several customers submit long documents at the same time. Requests wait. Some time out. The application retries, adding more work to a server that is already busy.
Who investigates?
Your team needs to understand capacity, concurrency, memory usage, and what the application should do when inference cannot finish. Someone needs to monitor the service, investigate failures, and plan upgrades.
The external provider previously handled part of that work. Your engineers now handle it.
You still depend on hardware, model releases, and inference software. You also become responsible for how those dependencies work together.
Count the Engineering Time
In AI Cost Starts with Architecture, I argued that operating infrastructure takes engineering time, and that time belongs in the cost discussion.
Apply that to the comparison.
What does the hardware cost? How much capacity sits unused? How much spare capacity do you need? Who tests an inference-server upgrade? What product work waits while that person investigates a deployment failure?
A lower API bill can be useful. You still need to understand the total cost of delivering an acceptable answer.
Security belongs here too. Keeping inference inside your environment gives you control over where it runs. Your team must then manage access, patch dependencies, and decide what gets logged.
As I discussed in Your AI System Needs a Deletion Path, prompts, extracted content, and saved answers can remain in several places. Hosting the model yourself does not remove those retention decisions.
Let the Requirement Justify the Work
Of course, local inference may still be right.
Maybe the product must work offline. Maybe customer data must stay within a controlled environment. Maybe predictable demand makes dedicated infrastructure economical. Maybe you need control over the model and its deployment.
Those are requirements worth evaluating.
Before committing, I would test the actual workload, including concurrent requests and failures. Then identify who owns operations and how much time they have.
This connects to Before Hiring More Developers, Find Out What Is Blocking Delivery: assigning additional responsibility to an engineer who is already fully occupied does not create capacity.
If the business needs your team to operate inference, make room for that work. Give it an owner, a budget, and expectations the team can meet.
Running locally can be a good engineering decision.
But you did not remove the vendor.
You made your engineering team the vendor. Plan accordingly.