Build vs. Buy for Private LLM Inference: What the Numbers and the Operations Actually Cost
At some point, if you're running AI features at any real volume, someone on your team asks the question: should we keep paying per token, or should we run this ourselves?
Most answers to that question are guesses dressed up as analysis — a spec sheet comparison, a back-of-envelope "GPUs are $X/hour, we'd need N of them." We didn't want a guess. Over six experiments (documented in The Metric That Kept Lying to Us), we measured actual cost, actual hardware efficiency, and actual operational overhead of self-hosting an open-weights model — and a few of the answers weren't what a spec sheet would have told us.
This piece pulls that data together specifically around the build-vs-buy decision: what it costs, what it takes to run, and when each side of that decision actually makes sense.
The two options, stated plainly
Buy means paying a per-token API rate to a provider — OpenAI, Anthropic, or similar — and never touching hardware. Someone else handles uptime, scaling, and GPU procurement; you pay a margin for that.
Build means provisioning your own GPU (on your own hardware or a cloud instance you control), running an inference server like Ollama or vLLM, and owning the operational overhead that comes with it — driver installs, capacity planning, uptime, and the occasional 3am page.
The honest version of this decision isn't "which is cheaper" — it's "cheaper under what conditions, and what are you actually signing up for operationally." Both halves matter, and most build-vs-buy content only covers the first.
What self-hosted cost actually looks like — and why the obvious number is wrong
If you self-host, the intuitive cost model is: GPU price × hours run. That number is available, and it's incomplete in a specific, measurable way.
GPU-only power draw isn't the real cost — the CPU picks up whatever the GPU doesn't do. When we split an 8B model's computation between GPU and CPU and only measured GPU power, energy-per-token rose the more work we put on the GPU. That's backwards. Once we measured the CPU's real power draw alongside the GPU (via the processor's own telemetry), the picture reversed: total energy-per-token fell steadily the more GPU offload we used — 9.11 down to 4.69 joules per token across five configurations. The CPU work doesn't disappear when it's not on the GPU; it just goes uncounted if you're not measuring it.
The lever that actually mattered wasn't hardware — it was concurrency. On the same cloud GPU deployment, raising the number of simultaneous requests being processed from 1 to 8 cut effective cost per token by roughly 4.8×:
| Concurrency | Cost per million tokens | GPU power draw |
|---|---|---|
| 1 | $4.11 | 71.7 W |
| 2 | $2.26 | 71.6 W |
| 4 | $1.59 | 71.8 W |
| 8 | $0.86 | 71.2 W |
Power draw barely moved. The GPU was mostly idle-but-"busy" at low concurrency — the classic signature of a workload waiting on memory rather than compute — and batching multiple requests let it do more real work per watt. This is a configuration change, not a hardware upgrade, and it's the single biggest cost lever we measured across the entire series.
Practical read for a build-vs-buy decision: if you're pricing out self-hosting based on a single-request benchmark, you're pricing out the most expensive way to run it. Any real cost estimate needs to account for the concurrency your actual traffic will run at — a self-hosted deployment serving one request at a time can be several times more expensive per token than the same hardware serving a realistic batch of simultaneous requests.
The hardware-selection trap
The instinct, when self-hosting, is to buy or provision the biggest GPU the budget allows. We tested that instinct directly by running the same model on a 4GB laptop-class GPU and a 24GB datacenter GPU (NVIDIA L4), at matched configurations.
The bigger card was 4–5× less energy-efficient per token than the smaller one, at the same settings. Two reasons, both directly measured: a datacenter GPU's idle power floor is simply higher (13.4W just sitting there, versus 3.9W for the laptop card), and at partial workload it was less utilized — under a quarter as busy as the smaller card doing the identical task, because its compute was too fast for that split to keep it fed.
The efficient point on that same 24GB card was full offload — the entire model loaded onto the GPU, nothing split to CPU. At that setting, its utilization jumped from under 4% to 96%, and it became the cheapest point tested per token, by a wide margin.
Practical read: a bigger GPU is not a safer default. If a model doesn't fully fit on the card you're considering, partial-fit is often the worst-efficiency zone — worse than not using the GPU heavily at all, and worse than a smaller card that fits the model entirely. Size the GPU to the model, not the model to whatever GPU is available.
What self-hosting actually takes to run — the part spec sheets don't show
The dollar figures above are real, but they're not the whole cost of "build." Provisioning a cloud GPU and running a real measurement campaign on it surfaced operational overhead that a pure cost model won't capture:
Spot/preemptible pricing failed three times in under two hours during one measurement campaign — once before work even began, once mid-run (killing that attempt outright), and once during cleanup. The deliberate fix was switching to on-demand pricing, at roughly 20–30¢ more per run, accepted specifically for reliability. That's a real tradeoff: the cheapest cloud GPU pricing tier is also the least reliable one for anything that needs to run to completion.
A genuine capacity stockout hit the target zone mid-campaign — the cloud provider's own error named a different zone with availability, and the work had to relocate there, still under the same quota. Capacity isn't guaranteed just because you're willing to pay for on-demand pricing.
Driver and inference-server setup was done manually, over an interactive SSH session, by design — no automated startup script. The reasoning: automated provisioning scripts hide failures in boot logs; a manual, visible setup surfaces problems (a bad driver install, a stalled model pull) where they can actually be caught and fixed. That's a deliberate engineering choice, and it means "build" includes a real person doing real hands-on setup work every time infrastructure gets provisioned — not a one-click deploy.
A GPU quota cap of one instance meant the entire measurement campaign shared a single resource slot — no parallel provisioning, no ability to spin up a second instance without a quota request. For anyone planning to self-host at meaningful scale, quota limits are a real planning constraint, not a footnote.
None of this shows up in a $/hour GPU pricing page. All of it is real time and real risk that "buy" simply doesn't carry — the provider absorbs it, and that absorption is part of what you're paying for in the per-token margin.
A framework, not a verdict
There's no single right answer here — the data supports a conditional one:
Buy (API) tends to make sense when:
- Traffic volume or concurrency is unpredictable, low, or bursty — you can't reliably keep a self-hosted deployment busy enough to hit the efficient operating point we measured.
- You don't have someone who can own driver installs, capacity planning, and the occasional preemption/stockout scramble.
- Time-to-ship matters more than marginal cost right now.
Build (self-host) tends to make sense when:
- Volume is high and steady enough to run at meaningful concurrency — the 4.8× cost improvement we measured depends on keeping the GPU genuinely busy, not idling most of the time.
- Data residency, latency, or compliance requirements make an external API a non-starter regardless of cost.
- You size hardware to the model (fits entirely, or deliberately goes fully-offloaded) rather than defaulting to "biggest GPU available."
- Someone owns the operational reality above — not just the Terraform apply, but the 3am page when a preemption kills a run mid-flight.
The mistake to avoid either way: comparing a per-token API rate against a single-request, GPU-only-power self-hosting estimate. That comparison flatters "build" with a number it can't actually hit in practice, and flatters "buy" by ignoring that a busy, well-configured self-hosted deployment is a genuinely different cost profile than an idle one.
Related reading
The numbers above are drawn from a six-experiment series — each finding has its own deep-dive:
- The Metric That Kept Lying to Us — the full synthesis
- One Config Change Cut Our Inference Cost 4.8× — the concurrency finding, in full
- Why We Didn't Upgrade to the Bigger GPU — the hardware-selection finding, in full
Where this leaves you
If you're facing this decision for a real workload and want the actual numbers run against your traffic pattern rather than a generic benchmark, that's the kind of analysis we do — book a call and we can walk through what build-vs-buy looks like for your specific volume, latency requirements, and hardware options.
The full experimental series — including the raw data and methodology behind every number above — is available in The Metric That Kept Lying to Us.