Why I Moved My LLMs to Local Inference
The case for running large language models on your own hardware — and the engineering it actually takes to pull it off.
Six months ago, every LLM call in our stack went to a managed API. It was fast to build on, predictable in behavior, and we didn't have to think about hardware. Then our bill crossed four figures per month, a customer asked where their data was being processed, and I started running the numbers on self-hosting.
The cost argument is real but not simple
At our request volume, the managed API cost roughly $0.002 per 1k tokens. Over a million requests per month, that compounds fast. A single mid-range GPU — an A10 or even a used A100 — pays for itself in raw compute cost within a few months at that scale.
But the simple math leaves out ops cost. You need to handle model loading, batching, failover, and prompt-caching yourself. You need to monitor VRAM usage and queue depth. You need to decide between vLLM, TensorRT-LLM, or a lighter framework depending on your throughput and latency requirements.
vLLM changed the equation
Before vLLM, local inference meant either accepting terrible throughput or building your own continuous-batching scheduler. vLLM's PagedAttention implementation meant we could serve multiple requests from the same KV cache without re-computing attention over prefixes. Our throughput on a single GPU roughly tripled compared to naive HuggingFace generation.
We run quantized GGUF versions of our models through vLLM's GGUF support. The quality loss from 4-bit quantization is negligible for our use cases — classification, extraction, and short-form generation — and the memory savings let us fit a 7B model comfortably on 8GB of VRAM while leaving room for KV cache growth.
Data sovereignty was the real driver
If cost were the only factor, we might have stayed with the managed API and optimized our prompt caching. The harder constraint was data sovereignty. Our customers in regulated industries need contractual assurance about where their data is processed and that it is not retained. With local inference, the answer is trivial: the data never leaves our VPC.
This constraint showed up in every vendor evaluation. Even providers with strong data processing agreements could not give us the level of control that bare-metal inference on our own hardware does. When the legal team asks "can you guarantee the data is not used to train their models", the answer with local inference is unambiguous.
The operational reality
Local inference is not a weekend project. We spent a month on load testing alone — finding the optimal max-model-len, tuning the KV cache block size, and building a queuing layer that gracefully degrades when the GPU is saturated. We added a GPU health endpoint to our orchestrator and a fallback to the managed API when the local model is unavailable.
The fallback matters more than I expected. Network blips, CUDA out-of-memory events, model hot-reload — there are more failure modes than I had budgeted for. But the system now runs with 99.7% availability on local inference, and the fallback covers the remaining fraction without the user ever noticing.
Would I recommend it?
If your bill is under a hundred dollars per month, no. The ops overhead is not worth it. If your bill is into the thousands and you have even one engineer comfortable with GPU tooling, yes — the return on investment is fast and the strategic benefits go beyond cost.
The tooling has matured enormously in the last year. What required a dedicated MLOPS team two years ago is now a few Docker Compose files and a reasonable vLLM configuration. The barrier to entry has dropped. I expect local inference to become the default for production workloads that process any volume of sensitive data.