NVIDIA's Blackwell Ultra (B200 Ultra) GPU has set a new record for large language model inference throughput, achieving over 10,000 tokens per second on 70-billion-parameter models in configurations published at the MLPerf Inference benchmark consortium. The result — roughly 2.8x the throughput of the previous-generation H100 on the same task — is accelerating procurement decisions at cloud hyperscalers and enterprise data center operators who had been waiting for the new generation before expanding their AI compute footprint.
The performance gains derive from several hardware changes: a larger HBM3E memory configuration (192GB per chip vs 80GB on H100), increased NVLink bandwidth for multi-GPU tensor parallelism, and a dedicated transformer engine with support for FP8 precision arithmetic that reduces memory bandwidth requirements for inference workloads without meaningful accuracy degradation.
For inference-focused workloads, the memory capacity increase is as significant as raw compute. Many modern frontier models at 70B parameters or larger require multi-GPU configurations to hold weights in GPU memory, with the inter-chip communication overhead reducing effective throughput. The B200 Ultra's larger per-chip memory allows smaller models to run on a single chip and larger models to distribute more efficiently, improving actual throughput versus theoretical peak numbers.
Cloud pricing for Blackwell Ultra instances has come in above H100 equivalents on a per-chip basis, but the throughput-per-dollar metric is favorable for sustained inference workloads. AWS, Google Cloud, and Azure have all launched B200 Ultra instances, with demand remaining constrained by NVIDIA's supply ramp.
The competitive picture is complicated by Google's TPU v5e and AMD's MI300X, both of which offer competitive inference performance for specific model architectures at different cost points. For standard transformer-architecture models, the NVIDIA ecosystem's maturity advantage — in terms of software optimization, tooling, and operator familiarity — continues to translate into production deployments even where alternative hardware achieves comparable benchmark numbers in controlled settings.