The Real Cost-per-Token Math Nobody Publishes
Rate cards tell you an H100 costs $2.50/hr. They don't tell you about idle weeks, reserved commits, or the 23% of tokens you bill at FP32 because nobody updated the config. Here's the real equation.
Priya Raman
GPU Performance Engineering
The Real Cost-per-Token Math Nobody Publishes
Cloud GPU pricing is advertised per-hour. Teams pay per-token. The unit conversion is where the budget leaks.
The advertised number
A standard H100 spot listing might read:
NVIDIA H100 80GB — $2.49/hr
That's the vendor's number. It assumes the GPU is saturated 100% of the time, perfectly utilized, billed continuously.
The actual equation
effective_cost_per_1m_tokens = total_gpu_hours × hourly_rate / (tokens_generated_in_thousands)The two terms teams underestimate are:
- Total GPU hours — this includes idle spins waiting for the script to start, queue wait, debugging runs you forgot to kill, and the reserved commitment that didn't get released.
- Tokens generated — this is NOT prompt tokens. Many teams report the higher number (prompt + completion) and artificially deflate cost.
A worked example
- 1× H100 reserved node, billed 24/7 for a week: 168 hours × $2.49 = $418.32
- Real generation during that window: only 47 hours of active inference
- Tokens generated during active inference: 1.2M completion tokens
Naïve cost-per-1M: $418.32 / 1.2 = $348/M tokens Accountant's view (24/7 billing): the same — $348/M Reality (you could've used a 47-hour spot instance): the spot price escapes the 168-hour commitment entirely.
This is why "cost-per-token" reported by vendors using the active-inference denominator looks 3-4x better than the line item on your invoice.
Where the leakage comes from
- Precision drift: A model configured for FP16 inference gets re-pushed from CI as FP32 and nobody notices for a month. (Real event, real team.)
- Batch under-fill: You pad batch to 8 but your consumer traffic fills it to 4. Each token pays for a half-empty GPU.
- Cold-start queues: Each payload spins a new container; the warmup region is billed, not producing output.
- Reserved floor: A commit you bought for peak traffic sits at 30% utilization 22 hours a day.
How GPUOPs surfaces it
Every run records
effective_cost_per_1m_tokensgpu_hourly_rate_usdThe Vertical AI Agents then cross-reference your workload fingerprint (RAG, agentic, batch) against community benchmarks for the same hardware and produce a Golden Template — the config that minimizes cost-per-token for your specific shape.
The honest takeaway
Per-token cost is a function of three things, in order:
- Utilization (saturation × billed hours)
- Precision (FP16/BF16/INT8 where safe)
- Pricing (the rate card everyone obsesses over)
Most teams audit #3 every quarter and ignore #1 and #2. That's where the savings live.
Track effective cost-per-1M tokens automatically on every run. Submit yours to GPUOPs and let the AI optimizer close the gap.