NVIDIA took the Hot Chips 2026 stage to detail the Rubin GPU, its next data center accelerator in the Vera Rubin platform. Agentic AI is the recurring theme, and NVIDIA argues it changes not just the tensor core but the entire compute, networking, power, and serviceability stack around it. It feels like we know a lot about Vera, but let us see what NVIDIA has for Hot Chips.
This article is being written live from the talk, so please excuse any typos.
NVIDIA Rubin GPU at Hot Chips 2026
NVIDIA is talking about agentic AI is the most complex workload it has ever targeted. They did earlier in its Vera talk as well. It strings together observation, reasoning, and action across LLMs, CPUs, DPUs, orchestration, security, memory, and networking. That breadth is the reason the whole platform gets redesigned, not just the silicon.
Vera Rubin is presented as a full-stack AI factory platform spanning seven chips and five racks. NVIDIA folds in the Vera CPU, the Rubin GPU, BlueField-4, Spectrum-6, and even Groq LPUs, with storage, networking, tool call, and sandbox support layered on top. Agentic AI needs all of that surface area, from context and memory to the scale-out fabric.
This figure maps the Vera Rubin NVL72 rack against prior racks along a tokens-per-MW interactivity axis. Vera Rubin lands well above Blackwell NVL72 and far above the Hopper NVL8 generation (HGX H100/ H200 8-GPU) near the high-interactivity corner. We first saw the physical rack in partner booths at GTC 2026, and the architecture there aligns with the TPS-per-MW story this talk repeats. NVIDIA Groq LPX also extends this curve.
Benchmarking has to change for agentic inference. Traditional inference assumed predictable I/O and a single request with static hardware tuning, but agentic workloads bring tool calling, dynamic sequences, and multiple turns across end-to-end tasks. Input sequence lengths grow from the 1k to 32k range to 100k and 400k, which strains hardware tuned for short, fixed prompts. We have seen this at STH quite a bit.
NVIDIA points to a SemiAnalysis (Dylan’s) benchmark to quantify the payoff. Using DeepSeek-v4-PRO at 140K+ context on the AgentX workload, Vera Rubin NVL72 curves above GB300 NVL72 near the 60M total-token mark, and NVIDIA calls the uplifts 10x and up to 30x in tokens per MW as interactivity climbs.
Agentic AI also changes what NVIDIA optimizes for. Tokens per watt, time to first token, useful life, and mean time between interruptions (MTBI) all matter more than raw flops because the goal is to maximize revenue over time. TTFT and interactivity become first-class design targets. Basically, NVIDIA’s saying that it is not just having a fast system, but it is about maximizing the revenue of the platform.
Here is the Rubin GPU die floorplan paired with the headline factory-level specifications. Note these are 100MW AI Factory, not a single GPU. NVIDIA rates the part at 2 ZFLOPS for NVFP4 inference and 1.4 ZFLOPS for NVFP4 training, with 11 PB of HBM4 memory and 800 PB/s of memory bandwidth within a 100 MW AI factory footprint. As an aside, I guess Raja Koduri’s 2021 Zettascale was not far off! This design leans on NVLink 6 and NVLink-C2C beside a 5th gen x16 PCIe host interface, and NVIDIA notes the specs are at-scale figures using DSX with MaxLPS. I think we are not going into this too much today which is what I thought this talk was before it started.
Context grows with every agent turn and attention expands quadratically, so a long agentic session drives feed-forward and attention compute well past 400k tokens. NVIDIA says adding more raw math to the GPU is not enough.
For that, NVIDIA is going through the evolution from FP8 weights in 2023 through narrow-precision NVFP4 to Rubin’s adaptive sparsity in 2026, adding HW-managed mixed precision and dedicated silicon for low-precision operations. NVIDIA couples that with structured sparsity and accuracy-preserving compression for both training and inference.
Rubin’s adaptive sparsity makes the sparse path practical, using a 2:4 sparsity format that NVIDIA says is more general than prior NVFP4 sparsity. Sparsity applies throughout the transformer block, skipping near-zero values in the QKV, attention projection, and MLP feed-forward layers. NVIDIA claims that most models require no changes or fine-tuning and that it deploys as an opt-in for the inference runtime via a flag.
Sparsifying attention is where the token throughput comes from. Rubin retains only the tokens that matter, which NVIDIA says makes the downstream SoftMax and BMM2 operations about 2x faster. This sparse attention path preserves dense cores for dense work and routes the QKV matrices through a dedicated sparsification stage.
NVIDIA also claims the sparse path preserves visual quality out of the box. This hardware-generated demonstration pairs a Qwen-Image BF16 dense output with the NVFP4 sparse result of the same cinematic portrait prompt. Side by side, the two renders look essentially identical, but this is just one handpicked data point.
A hardware measurement backs the visual demonstration. Accuracy criteria vary by benchmark and reflect the percentage of evaluated items meeting each benchmark’s success criteria. NVIDIA shows the sparse results tracking dense across the evaluated benchmarks. The Qwen3.5-395B-A17B is one we have used a lot at STH (albeit running on a Mac Studio M3 Ultra 512GB because of the memory requirements.)
Distributed inference needs lower-latency GPU-to-GPU sync, and Rubin reworks how that happens. This figure contrasts Blackwell’s traditional MEMBAR transfer, which polls and updates an atomic flag, against Rubin’s counted-write-based sync. Replacing the memory barrier with a counted write lowers NVLink latency and helps throughput at higher interactivity.
NVLink scale-up is the other big lever for tokens per second. Sixth-generation NVLink ties a 72-GPU scale-up domain with 3.6 TB/s of all-to-all bandwidth per GPU, and NVIDIA cites 4x and 3x factors plus 10x lower latency and 130 TFLOPS of in-network compute compared with Ethernet scale-up. This NVL72 rack relies on the NVLink 6 switch tray to maintain coherence across the entire domain.
Around the silicon, NVIDIA is pushing the third-generation MGX open platform for the rack. More than 80 MGX partners span millions of square feet of factory floor across 350+ sites in 30 countries. This rack leans on 45C liquid cooling, 800 VDC, no retimers, hot-swappable, cable-free trays, and a copper scale-up fabric designed to improve throughput per watt, time to first token, and MTBI. The broader Vera Rubin platform was laid out earlier in 2026. Still, this is interesting given that 800 VDC racks were said to be delayed earlier this year. At STH, there is a reason we covered Liquid-Cooling a TE Connectivity Busbar and More from the Wiwynn Booth a few months ago because we were hearing this was happening. Also, on stage they just said that a hot tub is usually 39C while the liquid cooling here is using 45C liquid.
NVIDIA’s Vera Rubin compute tray has no cables and no fans, which the company says improves time to first token and mean time between interruptions versus the GB200 NVL72 compute tray beside it. Removing the cables and fans from the tray simplifies both build and service at rack scale. We saw with the IBM z17 Mainframe that they go into replacing power cables with PCB for reliability, so what NVIDIA is doing makes a lot of sense.
NVIDIA’s loop diagram shows warm-water cooling that removes the chiller and wasted water from the picture while preserving full performance. Raising the facility supply lets more of the power budget go to tokens rather than heat rejection. 45C is one that helps because it increases the temperature differential between the loop and the ambient, which often means a chiller is not needed. Sorry all, struggling to keep up with this one.
Power is another place NVIDIA reclaims headroom. This figure compares a rack power profile with and without smoothing, showing the ramp-up peak, steady state, and ramp-down tail side by side. NVIDIA argues that smoothing reduces idle-time overshoot, so more GPUs fit under a provisioned power cap. The black cylinders in the bottom right of this diagram help do this smoothing. It is neat to get to see this since NVIDIA has been talking about the feature for some time.
For LLM training on a Vera Rubin NVL72 rack, NVIDIA reports a 13% peak power reduction and improved grid compliance. Combined with other system-level power work, NVIDIA expects up to 40% more GPUs per provisioned watt.
Now it sounds like we are getting to MTBI. NVIDIA’s second-generation RAS engine, or RIST, runs zero-downtime GPU health checks while the workload continues to run, with advanced SRAM ECC and an enhanced NVLink hot-swap service. First-generation health checks required the entire node to be offline for hours, so the change directly affects goodput and predictive maintenance.
Across the talk, NVIDIA framed the Rubin GPU less as a single chip and more as one layer in a factory that spans computing, networking, power, and serviceability.
Final Words
Rubin GPU is a reminder that NVIDIA is now selling efficiency and total cost of ownership as much as raw flops. Instead of just a hundreds-of-watts GPU, or 1-3kW, NVIDIA is now using a 100MW AI Factory as a unit. Maybe the real message here is twofold. It feels a bit like NVIDIA is doing this to differentiate its offering versus AMD Helios (up later in this Hot Chips block). Also, importantly, it means that if a company has an AI accelerator that looks cool, NVIDIA is basically saying that the unit is a data center-scale, reliable system, not a single fast accelerator or even a fast system.
