Waymo just did an awesome keynote with video examples. If you are interested in this, then it is certainly worth getting a Hot Chips 2026 virtual ticket just to see the keynote. Now, we have the company’s Sensor Fusion Processor, a purpose-built silicon effort built around the compute at the center of its autonomous vehicles. Also, this is a HUGE presentation, so I am going to do my best to cover this, but it is a lot to cover.
This is being covered live from the presentation, so please excuse any typos.
Waymo Sensor Fusion Processor at Hot Chips 2026
Waymo starts with the system view of the vehicle. Compute sits at the center, surrounded by the lidar, camera, radar, and microphone sensor arrays, which is exactly where the Sensor Fusion Processor must aggregate every stream in the car.
Today’s focus is the sensor fusion encoder inside Waymo’s foundation-model stack. Camera, lidar, and radar embeddings all land on this encoder, and that stage is what this silicon was built to run. The driving VLM is more for the higher-latency-tolerant tasks.
Waymo wants in-house sensor processing co-designed around its own sensor fusion models, with latency measured from first pixel in to embedding out at low batch sizes. Something else that Waymo said in the keynote is that they can have vehicles sitting in the Phoenix summer sun that have liquid cooling, but with 60C liquid which is very different than in the data center.
Headline specifications are compact for the class. This part fits in a 45 mm by 45 mm FCBGA package with 208 mm2 of silicon on an N5 process, pairing in-package LPDDR5x memory at 273 GB/s with a PCIe Gen5 x8 host interface, 25G Ethernet, and a TDP of less than 75W.
Major subsystems split between Waymo-designed and proven third-party IP. Waymo’s own block handles the carTPU, ISP, codecs, and MIPI ingest path with scratchpad memory, while two GPUs, the on-chip interconnect, memory, PCIe and Ethernet controllers, PHYs, secure boot, and peripherals come from third parties. Notably, Waymo does not have DSPs in this. They said it is partly due to the required programming skills.
An evolution timeline explains why Waymo built its own silicon. Waymo tracks its AI from deep conv networks in 2013 through transformers and large datasets to a foundation-model stack driving perception, planning, and world modeling by 2024.
Compute throughput relies on integer and higher-precision. Waymo quotes 160 TOPS for INT8 by INT8 and 80 TFLOPS for FP16, with 64 MB of on-chip SRAM, an 8 MB register file, and 160 GB/s of external memory bandwidth. Waymo says it is focused on higher precision to preserve fidelity.
This architecture overview shows a control block paired with the core carTPU array. Each control PE field carries a dual-core RISC-V CPU with vector extensions, while the core array packs 16 processing elements and 2 cores.
Waymo’s carTPU cores split across three backbones: camera, radar, and fusion. This fusion path sits latency-critical while camera and radar backbones run at maximum concurrency.
Dataflow principles keep the hardware lean. Waymo relies on deterministic mega-instructions with static shapes, running single-thread execution over roughly a thousand cycles with no caching or branches, while FIFOs and memory with semaphores handle ordering and backpressure across compute and DMA threads.
Each processing element keeps a highly banked 2 MB SRAM sized so activations and context stay resident to feed GEMVs. Ring and mesh DMA divide weight streaming from cross-PE collectives, and a 4x compute slice sustains INT8 GEMV per cycle with sparse gather and scatter fusion. I am struggling to keep up with this presentation. It is fast.
A flexible GEMV path shapes itself to each compute primitive. Waymo morphs the operation for maximum spatial utilization, which matters for keeping the tensor cores busy across very different layer shapes.
Software follows a compiler-first philosophy. Ahead-of-time compilation builds a mega-kernel for the whole model and shards it to fit in SRAM, leaving the compiler to manage partitioning, global memory traffic, race conditions, and numerical accuracy.
Waymo argues that achieved performance matters more than raw TOPS here. This figure plots continuous optimization as relative speedup against initial latency, showing co-design and measured efficiency beating headline throughput.
Camera pipeline work starts with MIPI ingest. Raw sensor data is moved to DRAM while a grayscale alignment pyramid is built for later fusion, with sensor error handling performed at the same stage.
An HDR ISP extends that path. Raw frames are converted from RGB to YUV with temporal denoising, then a still-frame image codec writes a YUV pyramid back to DRAM.
Pipelining keeps the camera flow moving. MIPI ingest outputs progress as the ISP advances as far as it can, and the YUV pyramid is produced with sub-frame latency roughly a few hundred image lines behind sensor read-out.
Patch extraction follows in the camera path. This stage pulls arbitrarily scaled regions of interest from the YUV pyramid, converts color space, and applies synthetic HDR exposure so models only see the windows they need.
This flow view ties the ISP blocks together. Raw sensor streams flow through image signal processing into RGB, then on to pyramid generation with an inline lossy codec compressing the pyramid before it reaches DRAM. A lot of this Waymo seems to be doing itself.
Temporal denoising compares a single-frame result against multi-frame processing. Multi-frame temporal denoising trades latency for cleaner images, which matters at night and in low-contrast settings for downstream fusion.
Waymo builds its own demosaic rather than using a standard image signal processor block. Traditional demosaic can introduce zipper and color artifacts, while the in-house version preserves color accuracy and edge detail.
HDR sensing targets the hard real-world cases. Headlight glare at night and the transition out of a tunnel are exactly the scenes an autonomous car must read correctly, and both drive the dynamic range requirements.
Radar and lidar share the same ingestion front end. MIPI ingest moves raw data into DRAM and applies sensor error handling before deeper processing.
A GPU stage handles the heavy sensor math. Point cloud processing, waveform processing with FFT cubes, and GPGPU programming models run on the GPUs, separate from the carTPU inference cores.
Some radar and lidar data takes an Ethernet path directly to the host. That offloads traffic from the GPU and carTPU so the sensor math and the model inference are not competing for the same bandwidth.
Typical radar processing is built on multi-dimensional FFTs. Range, Doppler, and azimuth projections come out of this stage, feeding the point and feature representations that later fuse with camera data.
Example lidar projections show the representations the pipeline produces. A perspective view and a top-down view, alongside range, intensity, and elongation maps, give a sense of the output forms available for fusion.
Waymo closes the sensor fusion talk by positioning the carTPU as the first purpose-built ASIC for autonomous driving at scale. It describes the chip as co-designed from the ground up with best-in-class efficiency, and notes that riders can experience it in sixth-generation vehicles.
This is, of course, just one chip being used at Waymo.
Final Words
Waymo has moved beyond off-the-shelf autonomous driving accelerators to silicon co-designed around its own sensor fusion models. This processor puts the company in control of the entire sensor-to-embedding path, and per Waymo it is already running in a live fleet.
Honestly, there are probably quite a few misses in this one. This was a really fast presentation.
