TelcoNews US - Telecommunications news for ICT decision-makers
United States
NVIDIA rethinks AI memory to boost inference efficiency

NVIDIA rethinks AI memory to boost inference efficiency

Tue, 17th Nov 2026 (Today)
Joseph Gabriel Lagonsin
JOSEPH GABRIEL LAGONSIN News Editor

NVIDIA has outlined how it plans to ease memory pressure in AI systems by redesigning how data moves across GPUs, CPUs, storage and software. The approach is intended to improve the amount of inference work delivered per dollar spent.

The shift comes as larger AI models, longer context windows and more agent-style applications increase demand for high-bandwidth memory, or HBM, one of the scarcest and most expensive components in advanced AI hardware.

Rather than relying mainly on adding more HBM to each GPU, NVIDIA is focusing on managing memory across a broader hierarchy that includes CPU memory, local solid-state drives and shared storage. It argues that the constraint is not just memory capacity, but also how quickly data can move in and out of it.

NVIDIA said frontier models now contain trillions of parameters, while inference workloads also carry larger KV caches that store context for active requests. Agentic systems add to that burden because they can preserve context through repeated model and tool calls, in some cases spanning up to 3 million tokens.

Retrieval-augmented generation adds another layer of demand by adding enterprise information to prompts, increasing the amount of context that must be processed and retained during inference.

Memory hierarchy

NVIDIA says AI infrastructure operators should treat memory as a hierarchy rather than a single resource. HBM sits at the top as the fastest and most contested layer, while CPU memory and storage provide larger but slower pools of capacity at lower cost.

It describes a "memory wall" as the point at which compute resources can no longer be fed with data quickly enough, even if extra memory is installed. Beyond that point, additional HBM adds cost and power use without increasing useful output.

To address this, NVIDIA is trying to reduce the amount of data that must stay in HBM, shift less active data into cheaper tiers, and bring it back before it is needed again.

One part of that strategy is NVFP4, a 4-bit data format that reduces the space needed for model weights and, separately, KV cache data. NVIDIA said several recent models have been released with NVFP4 checkpoints, including systems from Black Forest Labs, Thinking Machines Lab, Reflection, Meta and StepFun, as well as NVIDIA's own Nemotron 3.5 Super VL.

The company also pointed to its NVL72 systems, where model weights can be distributed across up to 72 GPUs. That leaves each GPU holding a smaller share of the model, freeing more HBM for cache and concurrent users.

CPU and storage role

NVIDIA said its Vera CPU is designed to support this model by providing large LPDDR memory linked to Rubin GPUs through NVLink-C2C. That, it said, allows paused context to be stored nearby and brought back quickly when needed.

Its Dynamo inference software is designed to move inactive context out of GPU memory and into CMX context memory storage. NVIDIA said CMX is intended as a shared storage tier for temporary inference context, with lower cost per bit than HBM and fewer overheads than conventional enterprise storage.

NVIDIA listed storage partners for CMX including Cloudian, DDN, Dell Technologies, Everpure, HPE, IBM, NetApp, VAST and WEKA. It added that Dynamo's KV Cache Reuse feature manages cache across GPU memory, CPU memory and storage as a system-wide resource, allowing previously computed context to be reused.

It also said retrieval workloads can be handled with less pressure on HBM by keeping the knowledge index outside GPU memory. According to NVIDIA, Nemotron Retriever models create smaller embeddings, while the cuVS library allows the searchable index to be built and served across CPU and storage infrastructure.

Throughput claims

NVIDIA tied the architecture changes to performance and cost claims across platform generations. On the SemiAnalysis AgentX DeepSeek V4 Pro workload, it said, GB300 NVL72 delivered up to 10 times higher throughput per dollar than the H200 platform at 100 tokens per second per user, while HBM capacity per GPU roughly doubled.

It also said Vera Rubin NVL72 delivered up to 30 times higher throughput per dollar than GB300 NVL72 at 125 tokens per second per user, even though HBM capacity per GPU increased by about 3%.

NVIDIA is using those figures to argue that AI system economics are increasingly shaped by how effectively operators use each layer of memory and storage, rather than by raw HBM growth alone.

"For AI factory operators, the metric that matters is how many useful tokens the factory can deliver within its power budget while meeting user response-time needs," NVIDIA said.