How a FinTech Team Cut Inference Costs 41% Without Re-Training
A 12-engineer team serving a 13B-parameter chat assistant was paying $11k/mo on H100s. Without touching the model, they cut that to $6.5k. Here's what the benchmark runs told us.
Sofia Almeida
GPU Performance Engineering
How a FinTech Team Cut Inference Costs 41% Without Re-Training
The fastest way to reduce inference cost is usually not quantization and usually not distillation. Those are real wins, but they take weeks.
This team's win came from config changes — visible immediately in benchmark data, deployable in a single afternoon.
The setup
- Model: 13B parameter fine-tuned chat assistant
- Hardware: 4× H100 PCIe, reserved commit
- Workload: Customer-facing RAG with a 6K prompt, ~400 token completion
- Bill: ~$11,300/month
- P99 goal: 1.2 seconds
What the GPUOPs dashboard showed
We ran three weeks of representative traffic through the Sandbox Wizard and let the optimizer analyze the runs.
Three things jumped out:
- Batch under-fill at 47%: Peak traffic saturated the batch (8), but overnight traffic averaged batch-of-4. The GPU was sized for peak.
- FP32 fallback in one deployment slot: The CI pipeline had a stale config in one environment that hadn't been re-pushed since the precision move. ~23% of tokens were billed at FP32 cost.
- Idle reserved hours: The reserved H100s were pinned 24/7 but traffic had a clear 9-hour overnight trough where they ran at <10% utilization.
The plan
| Issue | Fix | Effort | |---|---|---| | FP32 fallback | Push the FP16 config from CI to the victim slot | 15 minutes | | Batch under-fill | Move overnight to a 2× H100 on-demand pool, reserve only for daytime | 1 sprint | | Idle hours | Run batch//async jobs (embeddings, eval) in the overnight trough | 1 week |
The results (six weeks later)
- Effective cost-per-1M tokens: $4.20 → $2.49 (-41%)
- P99 latency: 1.18s → 0.83s (slack restored against the SLA)
- GPU utilization: 41% → 88% (over the daily average)
- Monthly bill: $11,300 → $6,540
Why re-training wasn't the answer
The model was fine. The team was over-paying for under-utilized compute and serving precision they didn't need. Until you can see that, the temptation is always to reach for quantization — but quantization on an under-utilized node is solving the wrong half of the equation.
The takeaway
Measure before you optimize. Leaks that cost real dollars — precision drift, batch under-fill, reserved idle hours — are invisible from inside the codebase. They're only visible from the benchmark data.
The FinOps AI Agent in GPUOPs does this analysis continuously. Every run you submit feeds its baseline, so recommendations stay tied to your actual workload.