<p>The AI hardware narrative has centered on compute — GPUs, TPUs, NPUs, and the race to pack more FLOPS into each chip generation. But at the scale of modern large language model inference, a different constraint has become primary: memory bandwidth. You can have all the compute in the world; if you can't feed the model weights to the processor fast enough, the compute sits idle.</p>
<h2>The Bandwidth Wall</h2>
<p>A 70-billion-parameter model at BF16 precision requires 140GB of memory storage. Serving a single inference request means loading some fraction of those weights through memory bandwidth. With current HBM3E (highest memory bandwidth available in commercial AI accelerators), this creates throughput ceilings that compute scaling alone cannot push through.</p>
<p>NVIDIA's H100 offers 3.35 TB/s of HBM3E bandwidth. An H200 pushes this to 4.8 TB/s. HBM4, sampling to customers from SK Hynix and Samsung, targets 6.5-9.6 TB/s per stack — a 2x improvement over HBM3E at equivalent capacity.</p>
<h2>What HBM4 Changes</h2>
<p>At 2x bandwidth, inference throughput for memory-bound workloads scales proportionally. This means either serving 2x more users per GPU (halving cost per query), or serving users 2x faster at the same cost. For cloud AI providers where cost-per-token and latency are competitive differentiators, this is a meaningful generational improvement — larger than the compute improvements in the same generation.</p>
<h2>Supply and Timing</h2>
<p>HBM is a constrained supply chain. SK Hynix, Samsung, and Micron are the only producers. HBM4 production is expected to begin ramping in earnest in H2 2026, with volume availability through 2027. NVIDIA's next architecture (Rubin) and AMD's MI400 series are both designed around HBM4. As with every memory generation, initial allocations will go to the largest customers — demand is already oversubscribed.</p>