I have discussed running on-prem AI inference with a number of founders. Most consider it overkill to buy hardware for their own workloads. A few see it as a necessity. The disagreement becomes easier to understand once you separate the hope of a smaller bill from the need to control where the work happens.
For a new workload, I would usually start with an API. It lets you find out what the model can do before you take on the job of running it. Ownership becomes worth considering when you know what work you need done, which locally deployable model can do it, and what you gain by taking responsibility for the hardware.
The model question comes first. In the previous essay, models with the same benchmark score produced very different bills. A deployment comparison has to preserve the same discipline: measure the cost of work you can accept. Buying a machine does not make its cheaper tokens a substitute for a model that does the job better.
The cloud is a batching machine
A model API service gives you access to a serving operation whose job is to get as much useful work as possible from its hardware. Its customers supply the demand, and its software determines how efficiently that demand can be served. That combination is part of what you buy with an API.
Batching is one of the reasons this works. A serving system can process several requests together, sharing the cost of reading the model’s weights across them. With enough concurrent work, the same hardware produces more tokens overall, although each person may have to wait longer for theirs.
SemiAnalysis’s InferenceX measurements make the tradeoff concrete. Kimi K2.6 on B200 hardware delivers 485 total tokens per second per GPU when the target is 150 output tokens per second per user. Lower that target to 30 and throughput reaches 4,732. These are the best measured configurations at each speed, normalized per GPU.
The provider gets more work from the hardware it already pays for. How much speed you can give up depends on the work. An overnight processing job and an interactive conversation do not need the same thing.
30 tokens a second per user
4,732 tokens a second per GPU · $0.10 per million tokens
50 tokens a second per user
3,449 tokens a second per GPU · $0.14 per million tokens
75 tokens a second per user
2,265 tokens a second per GPU · $0.21 per million tokens
100 tokens a second per user
1,510 tokens a second per GPU · $0.32 per million tokens
150 tokens a second per user
485 tokens a second per GPU · $0.99 per million tokens
Caching gives the provider another way to avoid repeating work. If each request begins with the same instructions, tool definitions or document, the system can reuse the computation for that prefix. Anthropic’s prompt-caching documentation describes how this reuse becomes a lower input price. That lower price is the one an owned system has to compete with.
You can use batching and caching when you run models yourself. What you cannot assume is that your workload will keep the machines busy enough to reproduce a provider’s economics. Pooling demand from many customers gives an inference service more opportunities to fill its capacity. On your own hardware, the available work has to come from you.
Control over convenience
Cost is only part of the reason someone might choose to own the system. The nature of the business can make control worth paying for, even when an API would be cheaper.
The data cannot leave. A business may have data it will not send to an external model provider at all. Argonne National Laboratory described that requirement in its work on a local geospatial pipeline: control over model execution, no external API dependencies, and sensitive intelligence processed “without data leaving secure infrastructure”.
Other buyers can get the control they need through a service agreement. In July, Norsk Helsenett, which runs the Norwegian health sector’s network, announced a direct award to Anthropic for Claude Enterprise. Its stated reasons included a separate data processing agreement, control of where data is processed and better security mechanisms. For that buyer, more control was available through the contract.
Before buying hardware on privacy grounds, I would be precise about the boundary the data must stay within. Keeping it inside your building, inside a country and inside an agreed processing environment are different requirements. They need not lead to the same purchase.
The response cannot wait. Some work needs inference close to the device doing it. 1X, building the NEO humanoid, states the requirement directly: “Once models are trained, they need to run directly on the robot with minimal latency.” For that kind of system, the acceptable delay is part of the design. An attractive average response time is little comfort if the slow responses arrive too late.
The model needs to stay available. Once a workflow has been tested against a model, a forced change means work. Providers can retire models, as Anthropic’s deprecation notices document. Running open weights with a pinned runtime gives you control over when that configuration changes, whether the hardware is rented or owned. You take on maintenance, but you can plan the model change around your own system.
All of these are reasons to want more control. None makes a machine secure simply because you own it. A local network can restrict external connections, but access, patches, monitoring, backups and data transfers still need someone responsible for them. That responsibility is part of what you are buying.
Where ownership breaks even
Suppose the model can do the work and either deployment meets your requirements. Now the price comparison becomes useful. How busy would an owned machine have to be before it costs less than the service?
Start with the bill that arrives whether the machine is busy or not. An eight-B200 server starts at $400,328.50 at one reseller. Write it off over three years with no residual value and the hardware costs about $11,120 a month.
Electricity adds about $960. For this estimate I use Nvidia’s 14.3 kW maximum draw for a DGX B200 throughout a 730-hour month, at the June US industrial electricity rate. Together, hardware and electricity come to roughly $12,100 a month, or $2.07 per GPU hour.
This is a partial ownership cost. Colocation, staff, spares, networking and cooling overhead are left out, as is extra capacity to keep the service available through maintenance or failure. The calculation gives ownership the benefit of those omissions.
The amount of work that bill buys you depends on the model and the speed required. For Kimi K2.6, InferenceX reports 3,449 total tokens per second per B200 at 50 output tokens per second per user. Applying that rate to eight GPUs gives an assumed capacity of 2.4 billion tokens a day. If you used all of it, hardware and electricity would cost $0.17 per million total tokens.
That is an extrapolation from the benchmark, not a measurement of the quoted server. The benchmark also uses a single-turn request with 8,000 input and 1,000 output tokens. An agent repeatedly reading a long context may achieve a different rate. Before making a purchase, I would want the intended configuration measured on the actual work.
For the benchmark’s request shape, Kimi’s API prices work out to $1.29 per million total tokens. That combines $0.95 per million input and $4.00 per million output. The ownership estimate meets that bill at 13% of assumed capacity, about 308 million tokens a day. At that volume, either option costs roughly $12,100 a month before the omitted ownership costs.
But the API customer can cache inputs. If every input token receives Kimi’s cached rate of $0.16 per million, the blended API price falls to $0.59 and ownership breaks even at 28%. That is the fully cached case. How close your workload gets depends on how much of and how often the input tokens repeat.
Renting the GPUs is a third option. At Lambda’s listed $6.69 per B200 hour, the crossover is about 31%, provided you get the same throughput and keep the rented capacity productive during the hours you pay for. An always-available service with uneven traffic would have idle rental hours to account for too.
1 API list price
308M tokens a day · $12,077 a month · 13% of the node
2 API, every input token cached
677M tokens a day · $12,077 a month · 28% of the node
3 Renting at Lambda, hours as used
737M tokens a day · $12,077 a month · 31% of the node
Owning, flat
$12,077 a month at any volume up to 2.38B tokens a day
Here, utilization means the work delivered over the month as a share of that assumed token capacity. It is what lets the fixed bill be spread over more or less work. At 5%, the ownership estimate is $3.33 per million tokens. At 50%, it is $0.33. Those are scenarios to compare with your demand, not predictions of what your business will achieve.
The machine does not have to run at full capacity around the clock to cross the API price in this example. It does need enough work, at the required quality and speed, to pay for the capacity you committed to. If demand falls, the hardware still has to be paid for. If it outgrows the machine, you need more capacity or a longer queue.
That is why I would begin with the service and let the workload make the case for ownership. Measure what an accepted result costs, how much work arrives and how much of it can wait. If those measurements support the full cost of running the system, there is an economic case to examine. If the business requires local control regardless of price, the same measurements tell you what that control will cost.
Sources and measurement
Nothing here is a hardware benchmark run by Zero One Labs. Every input and calculation is in the dataset, generated by scripts/prepare-onprem-dataset.mjs.
Throughput. SemiAnalysis InferenceX, Kimi K2.6 on B200, published 24 March 2026, latest run 7 August. The batching figure takes the best configuration at each per-user speed. The cost calculation takes the 50 tokens a second point on the single-turn 8k input, 1k output workload.
Tokens. Blended prices and capacity count input and output together.
Server. One reseller’s listed starting price for an eight-GPU HGX B200 system, retrieved 8 September. 36 months of depreciation, no residual value.
Power. Nvidia’s stated maximum draw for a DGX B200, applied for all 730 hours of the month, at the EIA’s June 2026 US industrial average.
API. Kimi’s saved list prices, with no cached input or with every input token cached.
Rental. Lambda’s saved hourly rate at the same assumed throughput, every rented hour productive. Startup time and idle or reserve capacity would raise it.
Images. The batching and crossover figures are under /research/on-prem/, with SVG originals and a manifest recording the dataset hash.