FP16 vs BF16: A Practical Throughput Guide for LLM Inference

When teams first move a model from FP32 to half precision, the most common question we hear is: "Aren't FP16 and BF16 the same? They're both 16 bits."

They're the same size, but they're not the same format — and on certain workloads the difference matters more than the 2x throughput either one buys you.

The 30-second refresher

FP16 has 5 exponent bits and 10 mantissa bits. Tight dynamic range (±65K), good precision.
BF16 has 8 exponent bits (same as FP32) and 7 mantissa bits. Wider dynamic range (~1e38), less precision per bit.

That exponent difference is the whole story.

Where FP16 silently breaks

For inference of short prompts with bounded logits, FP16 is fine — and on older A100/RTX stacks it's sometimes the only half-precision path with kernel support.

For training or long-sequence generation (anything with attention scores that can spike, or KL/softmax extremes), FP16 overflows are real and silent. You won't necessarily see NaNs; you'll see numbers drift, then a slow degradation that's easy to mistake for a learning-rate problem.

The benchmark we run

We benchmark the same model at FP16, BF16, and FP32 across three workload shapes:

RAG — short context, large output batch
Agentic — long context, chain-of-thought decode
Long-context summarization — 32K input, 256 output

What we measure:

throughput_tokens_sec

latency_p99_ms

, and

vram_used_gb

The pattern we see

On RAG and short-context workloads, FP16 and BF16 are within ~3% of each other on tokens/sec.
On agentic and long-context workloads, BF16 wins on stability AND ends up faster in practice because the FP16 runs need either loss scaling or fallback that tanks throughput.
VRAM is identical for both formats (half is half).

Practical recommendation

| Workload | Recommendation | |---|---| | Serving a fine-tuned model on a known prompt distribution | FP16 is safe if kernels are available | | Anything with attention score extremes or long decode loops | BF16 | | Mixed training + serving | BF16 everywhere | | INT8 not supported on your hardware | BF16 as the floor |

How to track it in GPUOPs

Create a single

Test Configuration

per model and submit runs at each precision with the workload_type tag set. The

Run Comparison

view will surface the deltas so you don't have to eyeball a spreadsheet.

If VRAM or P99 drifts between runs, the rule-based alerts fire automatically — which is exactly the silent-degradation scenario FP16 loves to hide in.

Start tracking precision sweeps against your own fleet by submitting your first benchmark run in GPUOPs.