GPU Economics for Data Teams

Believe it or not, LLMs don’t solve every problem.

Data teams have been buying CPU (and GPU) compute for thirty years and have developed reasonable intuitions about what it costs and how to use it. GPU compute, which data teams have suddenly had to buy a lot of since 2023, breaks most of those intuitions. The unit economics are different. The supply dynamics are different. The optimisation strategies are different. Worth working through, because the GPU bill is now a meaningful line item for any data team that’s touched AI in the last two years.

The Basic Price Shape

An NVIDIA H100 on AWS costs around $4-5 per hour on-demand. A B200 (the current generation) is higher again. A consumer-grade A10 or T4 is $1-2 an hour. A reserved H100 instance is lower than on-demand, but the reservation commitment is multi-year.

For comparison, a CPU instance with similar memory is $1-3 an hour. So a GPU instance is 2-3x a comparable CPU box. That sounds reasonable. The reality is more punishing once you factor in utilisation.

The Utilisation Problem

CPU compute can be near 100% utilised. A long-running batch job can saturate cores for hours. A web server can pin instances at high CPU all day. The unit economics work out because you’re paying for compute you’re actually using.

GPU compute, in most real-world ML workloads, runs at 20-40% utilisation. The math here is brutal: you pay full price for the GPU even when it’s idle, waiting for data to load, waiting for the next batch, waiting for the gradient sync. An H100 at 30% utilisation is effectively $15/hour of useful compute.

Where the time goes:

Data loading from disk or network. Especially bad if the data is in object storage.
CPU preprocessing pipelines that can’t keep up with the GPU.
Inter-GPU communication during distributed training.
Synchronisation barriers in multi-node setups.
Just waiting between jobs because the orchestration scheduler isn’t fast enough.

The discipline of measuring and improving GPU utilisation is what separates the teams burning money from the ones getting value.

The Inference vs Training Distinction

Inference and training have completely different cost shapes.

Training is bursty. You run for hours or days, then you stop. The economics are about throughput and time-to-result. You probably want the biggest, fastest GPU you can get, used as densely as possible for as short a time as possible.

Inference is steady. Requests come in continuously, often unpredictably. The economics are about latency, batching, and amortisation. You want enough GPU capacity to handle your peak request rate, but the smaller and cheaper GPUs often work fine. Quantisation matters. Batching matters.

Many teams over-provision inference GPUs because they were thinking like training. A 70B-parameter model can run on quantised inference at a fraction of its training cost. The same workload can be 10x cheaper on the right hardware with the right serving framework.

The Build vs Rent question

The biggest decision for serious GPU users is whether to use a hyperscaler (AWS, GCP, Azure), a specialist (CoreWeave, Lambda Labs, RunPod, Modal), or buy your own hardware.

Hyperscalers are convenient and expensive. The GPU price is high, the supply is tight (especially for H100s and B200s), the surrounding ecosystem is the deepest. Good for teams that need the cloud-native integration and are doing varied workloads.

Specialists are cheaper. CoreWeave, Lambda Labs, RunPod all offer the same GPUs at 30-50% lower hourly rates, with less polished orchestration but real cost savings. Good for teams whose workload is well-defined GPU-heavy compute that doesn’t need deep cloud integration.

Buying your own hardware is what large-scale ML shops eventually do. An H100 costs around $30K capital. Amortised over three years at high utilisation, the break-even vs cloud rental is typically 12-18 months. The catch: you take on the operational burden (power, cooling, networking, hardware failure handling, depreciation). For teams below a certain scale, this isn’t worth it. For teams above that scale, the savings are substantial.

The Optimisations that Pay Back

If you’re paying for GPUs, the optimisations that move the bill:

Measure utilisation. Nvidia’s nvtop, DCGM, or framework-level profilers (PyTorch Profiler). If you don’t know your utilisation, you can’t improve it.
Fix the data loader. The most common bottleneck. Profiling almost always shows the GPU is waiting for data. Faster storage, parallel loaders, prefetching, format choice (Parquet, WebDataset, MosaicML’s MDS).
Mixed precision. Train in bfloat16 or float16. Halves memory, often nearly doubles throughput. Free wins.
Gradient accumulation. Effective larger batch sizes without needing more memory.
Quantise inference. 4-bit or 8-bit quantisation of inference models. 2-4x throughput improvements common.
Batch inference. Group requests where latency permits. Serving frameworks like vLLM and TGI do this automatically.
Spot instances for training. Up to 70% cheaper than on-demand. Requires checkpoint discipline.

Token Cost vs GPU Cost. What you’re actually buying

The clearest way to understand GPU economics is to look at what you’re really paying for when you call an API. OpenAI and Anthropic don’t sell you GPUs — they sell you tokens, and the per-token price quietly bundles the GPU time, the model, the R&D, the availability guarantee, and the margin.

The current sticker prices (June 2026, per million tokens, input / output):

OpenAI — GPT-5 $1.25 / $10, GPT-5.4 $2.50 / $15, GPT-5.5 $5 / $30
Anthropic — Claude Haiku 4.5 $1 / $5, Claude Sonnet 4.6 $3 / $15, Claude Opus 4.8 $5 / $25

Batch processing halves those numbers, and prompt caching cuts repeated input by ~90% — which matters enormously for agents that resend the same system prompt thousands of times.

Now the other side of the ledger: the raw GPU cost. A rented H100 is roughly $2.50–$3.50/hour at a specialist provider (more at the hyperscalers). With vLLM, a strong open model like Llama 3.3 70B in FP8 serves a few thousand tokens per second once you batch it. Saturated, that pencils out to roughly $1–$3 per million output tokens; the smaller open models (≤20B) land at $0.10–$0.30. In other words, the marginal compute cost of generating a token is often 5–15x below the API sticker price. That gap is the provider’s model quality, availability and margin — and it’s why they have a business, not evidence of a rip-off.

So when does running it yourself actually win? Only when two things are true at once: you can use an open model, and you can keep the GPU busy. A single H100 running 24/7 costs about $2,160/month whether you push a billion tokens through it or a million. Spread across heavy, steady traffic it’s cheap per token; spread across spiky, low-volume traffic it’s catastrophic — the utilisation problem from earlier, wearing a new costume. As a rough rule, below a few hundred million tokens a month, per-token API billing wins because you aren’t paying for idle silicon. Above it — with an open model and real load — owning the serving stack starts to pay back.

The trap is comparing a frontier API price to a bare GPU rate and declaring the API overpriced. You’re not comparing like with like: you cannot rent GPT-5 or Claude by the hour. The honest comparison is frontier API against self-hosted open model, and the right answer is almost always a portfolio — frontier APIs for the hard, low-volume calls, self-hosted open models for the high-volume, well-defined ones.

Small Language Models

The biggest lever on the GPU bill isn’t a cheaper GPU or a smarter serving framework, it’s a smaller model. Small language models (SLMs), loosely 1–15B parameters, have quietly become good enough for a large share of real production work: classification, extraction, routing, summarisation, structured output, and tool-calling. Phi, Gemma, Llama 3.2 1B/3B, Qwen’s small tiers, Ministral, SmolLM are the frontier of “small” moves every few months.

Economically, an SLM collapses the entire cost shape. A 3B model quantised to 4-bit fits in a couple of gigabytes and runs at hundreds of tokens per second on a single cheap GPU — or on a CPU, a laptop, even a phone. You go from renting an 8×H100 node to running on hardware you already own. Cost per million tokens falls to cents, latency drops, and the data never leaves your environment.

How they actually get run:

Quantised local runtimes. llama.cpp and Ollama serve GGUF-quantised models on a CPU or a single consumer GPU. This is the default for laptops, edge boxes, and on-prem deployments.
On-device and edge. Phones and browsers run 1–3B models on NPUs or via WebGPU for private, offline inference at zero marginal cost.
Server-side batching. For throughput, the same vLLM/TGI stack that serves the big models serves a small one far more densely, on a fraction of the hardware.
Fine-tuning and distillation. The move that makes SLMs punch above their weight — distil a large model’s behaviour onto a small one for a narrow task, or LoRA-fine-tune on your own data. A tuned 3B model routinely beats a general 70B on the specific job you care about, at a hundredth of the cost.

The pattern that wins is routing: send the easy, high-volume requests to an SLM you run yourself, and escalate only the genuinely hard ones to a frontier API. Most teams discover that 80–90% of their traffic never needed the big model at all — and that realisation is worth more than any hardware optimisation on this list.

The Data Team Angle

For a data team that’s suddenly running ML or AI workloads, the lessons are:

GPU economics aren’t CPU economics with bigger numbers. The utilisation problem changes things.
Train and serve on different infrastructure. The cost shapes are different enough that the trade-offs differ.
Measure before optimising. Profiling first; investment second.
Specialist providers are often genuinely cheaper for well-defined workloads. The hyperscaler tax is very real though.
Quantisation and serving optimisations have larger leverage than picking newer hardware.

GPU compute is the most expensive thing on most data team budgets in 2026. It’s also the most opaque. The teams that get good at this will be the ones that take it seriously as an operational discipline, not as something the ML team will figure out.