Next up, d-Matrix is presenting its Raptor 3D-DRAM accelerator for generative inference at Hot Chips 2026. The company has made waves, and we have covered it before, including the d-Matrix Corsair In-Memory Computing for AI Inference at Hot Chips 2025. We also found they were doing networking in The New d-Matrix JetStream 400G Ethernet Card for Data Center Scale AI Inference. Let us see what they have going on this year.
This is being done live, so please excuse typos.
d-Matrix Raptor 3D-DRAM Accelerator for Generative Inference at Hot Chips 2026
Model weights keep growing, and the KV cache scales with context length multiplied by batch size. So 64 users at 1M context can mean roughly 935 GB of KV cache. Weights and cache together create a problem that is both a capacity problem and a bandwidth problem, and both sides keep growing.
SRAM meets the bandwidth target, but only on a tiny scale. A Corsair SRAM accelerator card pair reaches roughly 300 TB/s at about 1 ns latency, yet holds only about 4 GB. A 6T SRAM cell is around 10 times larger than a DRAM cell, and leakage runs to tens of watts at GB scale. This makes SRAM suitable for a draft model in speculative decoding, not for holding frontier model weights. That seems to be what NVIDIA is using Groq for as an example.
HBM solves the capacity half but struggles on bandwidth. Pin speed and I/O width per base die improve slowly, and the number of stacks is limited by available package beachfront, roughly 8-16 stacks per package. d-Matrix cites a practical bandwidth ceiling around 20 TB/s for HBM4 packages such as the NVIDIA Vera Rubin and AMD Instinct MI455.
Bandwidth that high carries a power price. At 2.4 pJ/bit, pushing 100 TB/s through HBM eats about 1.92 kW before any fabric traffic is counted. Packages today lack both the beachfront and the power budget to reach SRAM-class bandwidth with HBM.
d-Matrix’s answer is to stack compute directly on top of DRAM dies. Stacking creates a thermal challenge because hundreds of watts must escape through TSVs in a temperature-sensitive DRAM stack, plus a power-delivery challenge from IR drop. d-Matrix says a 1-Hi logic-on-top stack at no more than 0.5 W/mm2 can be liquid cooled and keep DRAM under 100 C.
3D DRAM lands between the two extremes on an energy ladder. On-die SRAM costs roughly 50 fJ, while 2.5D HBM4 systems run in the 2.5 to 5 pJ range when chip-level energy is included. Vertical 3D IO comes in at around 0.3 to 0.4 pJ, about 10 times lower than HBM, because it is a PHY-less millimeter-scale path rather than a centimeter-scale interposer route. Fewer stacked layers than HBM also means a larger die and better yield.
d-Matrix is now mapping that view of technologies onto how LLM inference workloads behave. Prefill processes many prompt tokens in parallel and is compute-throughput-bound, whereas decode produces one token at a time and is typically memory-bandwidth-bound. Attention can flip to compute-bound with high GQA and speculative decoding, and MoE stays memory-bound even at modest batch sizes. Decode is the phase that wants huge bandwidth. If you saw our NVIDIA GB10 or AMD Strix Halo coverage, memory bandwidth is the big challenge with those types of systems.
Since decode dominates wall-clock runtime, the memory-bound portion matters most. d-Matrix highlights that most inference time is spent in the decode phase, so improving decode bandwidth improves overall inference performance.
At 32GB per card, with 4-bit weights and an 8-bit KV cache, d-Matrix sizes to fit in one rack. A 72-card scale-up can host a frontier model such as Kimi K3 at 1M context. Disaggregation and multi-rack extend beyond a single Raptor rack.
Building the system around this memory is a co-design exercise across the memory subsystem, data movement fabric, and workload mapping.
Now d-Matrix is showing its topology using the full mesh package and discussing its communication protocol.
d-Matrix’s specific implementation is called Raptor. A TSMC N4 logic die sits on top of a 3D DRAM die using 36 um face-to-face stacking, a process d-Matrix describes as proven, low-cost, high-volume, and high-yield.
Turning the dies into a working system exposes a broad set of integration challenges. d-Matrix is highlighting four here.
Those four problems are not independent. d-Matrix walks through three entangled challenges in bank mapping, I/O power, and thermal reliability, noting that a solution to any one constrains the design space of the other two.
Each tensor engine needs a 128B flit per access, and with 32B delivered per column access from 32B banks, that works out to needing 4 banks per channel. d-Matrix’s die has 840 banks, 768 after 72 spares, spread across 256 channels for just 3 banks per channel. This flit does not divide evenly across what is available.
With 3 banks per channel, a single access returns 96B, so delivering a 128B flit takes two accesses and fetches 192B, wasting about 33 percent of bandwidth near 33 TB/s. Column staggering could pack flits but needs a 192B shifting buffer and complicates timing and verification.
Stream blocking reclaims that waste. d-Matrix shares one partial 32B access across three flits, so 4 accesses at 96B feed 3 flits at 128B, with 384B in, matching 384B out. Overfetch drops to zero, every column access is used, and no shifting network is required.
Moving 100 TB/s at 0.37 pJ/bit works out to 296 W just for I/O, and conventional DBI could save 20 percent. HBM gets away with DBI because its multi-cycle bursts let the PHY see the full burst, but d-Matrix’s single-cycle 256-bit 3D-DRAM link has no burst and no sideband pin to signal the inversion choice.
Stream flipping delivers that 20 percent without the pin. Each flit is compared to the previous one and inverted when needed, cutting toggles to near zero with a single metadata bit per flit carried alongside ECC. d-Matrix puts the overhead at 0.8 percent with no PHY change.
Heat poses the third challenge at a 105C junction temperature. Yield matters because 840 banks mean even a 1 percent fault rate threatens whole channels, and discarding bonded dies is uneconomical. Disabling a faulty bank narrows its channel, and a single weak channel can throttle the entire slice, while higher temperatures drive more soft errors. Retention drops from 32 ms at 85C to 4 ms at 105C, demanding 8 times more refresh while ECC and scrub must keep up. This is why Hot Chips is so neat, because we are getting to this level of nerd detail.
To combat all of this d-Matrix interleaves ECC and DBI bits into the last columns of each subarray, pairing commodity ECC with a Reed-Solomon code. Reading 16 to 32 times fewer rows means the 8-times-more-frequent refresh costs only about 1.37 percent, holding bandwidth near 100 TB/s with under 1.4 percent loss.
Bank chaining keeps channels symmetric when banks fail. Two physical mux levels let level one skip the first fault and level two skip the second, so the chiplet’s 72 spare banks absorb faults anywhere on the die. Spares backfill and channels stay uniform with negligible routing cost.
Now d-Matrix has what looks like a silicon-area comparison against HBM4 and NVIDIA Rubin R200. Raptor posts about 32.6 GB/s per mm2 compared with roughly 1.5 GB/s for the HBM parts, around 20 times the bandwidth per square millimeter, and 2.96 mW per GB/s against 40 mW, a 13.5x improvement. That is pretty neat.
Raptor’s result claims roughly 1,000 tokens per second per user, serving a 3-trillion-parameter class model with 1M context. d-Matrix’s thesis is that moderate per-card memory paired with high bandwidth wins in low-latency inference. It would be neat to see this actually running. Hopefully we can do that one day.
That is pretty neat assuming it works, but it would have been nice to hear a bit more on the drawbacks.
Final Words
Raptor is an architectural bet that bundles bandwidth and capacity into a single package for inference. d-Matrix’s comparison figures against HBM4 and Rubin R200 point to a real density and power-per-GB/s advantage. If 3D DRAM closes the memory bandwidth gap, it reshapes how accelerator vendors weigh capacity against bandwidth in AI systems. Whenever someone has a solution like this that seems somewhat adjacent to the current architecture, my first question is usually “why is everyone not using this?” You would have to imagine companies like NVIDIA have someone somewhere who would have thought about stacking DRAM on top of logic. This one is really interesting, and hopefully we can see it running soon.
