AI models are getting too big for the memory that feeds them. High-bandwidth flash, or HBF, is one proposed fix: it swaps the DRAM in high-bandwidth memory for NAND flash and promises up to 16 times the capacity at similar read bandwidth. The catch is latency. An HBF read takes nearly 20 microseconds, about 20 times longer than HBM, and serving model weights on demand from it can slow inference by as much as 12.8 times.
A paper from Georgia Tech, Nvidia Research and Stanford University argues that the gap can be closed in software and hardware rather than in the memory itself. The system, called SlipStream, uses a 40-megabyte buffer and a scheduling trick to hide most of the latency. In simulations across seven language models, it reached 92.4% of the performance of an ideal HBF system with HBM-like latency.
The work is a simulation study, not a shipping product. It has been accepted to HPCA 2027, a computer architecture conference, and it offers a preview of the engineering problem that HBF vendors will face as their memory reaches GPUs.
The memory wall
Large language models are limited by memory in two ways. They need enough capacity to hold their weights, and they need enough bandwidth to feed those weights to the compute units quickly. HBM, the stacked DRAM used on modern AI accelerators, is fast but expensive and capacity-constrained. A single HBM4 stack tops out at 64GB, and an accelerator can carry only a few stacks.
HBF takes a different approach. It keeps the stacked, near-compute package but replaces the DRAM dies with NAND flash, which is denser and cheaper per bit. Sandisk and SK hynix released the first HBF standard through the Open Compute Project in August 2026. It defines stacks of up to 512GB and bandwidth grades from roughly 0.4 to 3.0 terabytes per second, putting HBF between HBM and an SSD in the memory hierarchy.
That capacity matters for inference. The models in the SlipStream study range from 16GB to 800GB. A GPU with 80GB of HBM cannot hold the larger ones, so operators split them across many chips, which adds communication and cost. HBF could hold more of a model close to the GPU. The tradeoff is that flash is much slower to read than DRAM.
Why 20 microseconds is a problem
HBM responds in about one microsecond. HBF takes nearly 20. That difference is small in human terms and enormous for a GPU. If inference waits for each weight page to arrive from HBF, the paper found, performance drops by 10.8 times on average and by up to 12.8 times across the seven models.
The reason is a queueing rule called Little’s Law. To keep a memory channel busy at bandwidth B while each request takes latency L, the system needs B times L bytes in flight. Because HBF’s latency is 20 times higher, it needs roughly 20 times more outstanding data, about 40MB in the paper’s model. GPUs are not built for that. Their miss-status holding registers, the hardware slots that track outstanding memory requests, are sized for HBM, and even the combined shared memory on an A100 or H100 is only 18MB to 29MB.
Adding more tracking slots does not solve it. In the paper’s simulation, giving the GPU 20 times more request slots and queues improved on-demand HBF to just 14.6% of the ideal system, because the extra slots need more compute threads to generate demand. Conventional cache-line prefetchers have the same problem. They need 20 times more in-flight state to cover the latency, and that state is expensive.
Tensor-streaming designs take another route. H3, a prior system, streams data in virtual-address order rather than the order the kernel actually asks for it. That lets it prefetch aggressively, but it needs hundreds of megabytes of buffering to avoid running too far ahead. With a 40MB buffer, H3 reached only 10.6% of ideal HBF performance. Even with a 320MB buffer, it reached 43.1%, roughly a 2.3 times slowdown.
How SlipStream works
SlipStream treats prefetching as a producer-consumer pipeline. The producer is a staging warp, a small group of GPU threads whose only job is to request weight pages ahead of the compute warps. The consumers are the work warps that actually perform the matrix math.
On the software side, the compiler profiles the order in which a kernel will demand weight pages. For data-independent kernels, such as the matrix multiplications that dominate LLM inference, that order repeats and does not depend on the input. SlipStream inserts a staging warp that replays the order using non-blocking STAGE instructions.
The hardware side is where the key idea lives. A conventional prefetcher stops tracking a request when the data arrives. SlipStream stops tracking it only when the compute warps first ask for that page. Demand therefore sets the pace. If the work warps are running behind, the staging warp cannot run far ahead, and the buffer does not fill with pages that are not needed yet.
SlipStream also tracks 4KB pages instead of 64-byte cache lines, which cuts the number of tracking entries. It allocates buffer space only when a page actually returns from HBF, so the buffer size does not have to cover every request in flight. The result is a 40MB staging buffer that stays useful across models and latency assumptions.
The results
The researchers simulated an A100-class GPU with 80GB of HBM in the baseline and 1,280GB of HBF, 16 times the capacity, at equal aggregate read bandwidth. Each HBF page fetch carried the 20-microsecond latency. They then ran seven dense models and compared SlipStream with H3 and with an ideal HBF system that had HBM-like latency.
With a 40MB buffer, SlipStream reached 92.4% of ideal performance on average, with a range of 78.0% to 107.2% per model. H3 needed 320MB to reach 43.1%. SlipStream’s 16MB version reached 88.8%, within 4% of the 40MB result, and even an 8MB buffer reached 81.4%. The system also handled mixture-of-experts models, reaching 91.2% of ideal with 40MB.
Two large models, Llama-70B and Qwen2.5-72B, actually beat the ideal system by up to 7.2%. The paper attributes that to pacing. The ideal system releases a burst of loads at each kernel boundary, which saturates the L2 request slots and stalls later requests. SlipStream’s steady stream avoids that burst.
Smaller models do worse. Each weight-kernel boundary pays a fixed cold-start cost while the first pages arrive from HBF. That cost takes 18% to 22% of a layer’s time for 8B to 32B models, whose layers run for 353 to 773 microseconds. It takes only 4% of Llama-70B’s 1,391-microsecond layer. The researchers also modeled flash subarray contention and read retries, two sources of latency variance that vendors have not disclosed for HBF. Those effects lowered the 40MB result to 89.2% and 86.9% of ideal.
What it means
The paper’s contribution is not a faster flash chip. It is a demonstration that the latency penalty can be managed with a relatively small amount of on-chip memory if the software knows which pages are coming and the hardware lets demand control the prefetch distance. That is a different bet from waiting for HBF to become as fast as DRAM.
There are limits. The study is a cycle-level simulation of an A100-class GPU, not a measurement on HBF silicon. It assumes a 20-microsecond read latency and models latency variance with parameters that HBF vendors have not published. It focuses on read-only weight pages; a real system would also have to handle writes, refresh, power and cost. And 40MB of SRAM is still a scarce resource on a GPU, even if it is far less than H3’s 320MB.
Even so, the result lands at a useful moment. HBF hardware is moving from standards to samples. Sandisk has said it planned first samples in the second half of 2026 and expected devices with HBF in early 2027. If the capacity arrives before the software is ready, the memory will look slow. SlipStream is one attempt to make sure that does not happen.
For readers tracking GPU memory, the capacity gap is easy to see. Even a high-end workstation card such as Nvidia’s RTX PRO 5500 carries 84GB, which is small next to the 800GB models in the study. HBF is aimed at the data-center tier above that, where model size and batch size push HBM capacity to its limit.
Bottom line
The appeal of HBF is simple: more memory close to the GPU. The difficulty is that flash is slow, and a naive system loses more performance than the extra capacity is worth. SlipStream shows that a 40MB buffer and demand-paced staging can recover most of that loss in simulation. The next test is whether the same trick works on real HBF hardware, where latency varies and the vendors have not yet published the numbers.
Have a tip? Email [email protected] or use the contact form and choose Editorial. We do not promise anonymity.
Read next: Jensen Huang petition tops 180,000 as Musk backs Nvidia return; NVIDIA RTX PRO 5500: an 84GB RTX 5090 for $17,999; Claude picked a bombed Iranian school as a target in tests. Browse all Chips coverage.
Sources: arXiv; Semiconductor Engineering; SK hynix; Sandisk; Tom’s Hardware; Sandisk.
