NVIDIA Jetson AGX Thor Developer Kit Specs
At the heart of the kit is the NVIDIA Jetson T5000 which has 2070 FP4 (sparse) TFLOPs. Figure that sparse is twice that of dense, and that is still very impressive. There is also a 14-core Arm Neoverse V3AE processor. 128GB of LPDDR5X memory provides 273GB/s of memory bandwidth.
Here is the spec table with a bit more.
Something that NVIDIA does, that we wish other companies did more of, is to specify a wide power band. One can use this chip at 40W for more power-constrained environments and then 130W for when you need more power. The Jetson AGX Orin 64GB kit is 15-60W. While we get a lot more performance and I/O, it is pushing to a higher power band which is important for battery operated devices.
NVIDIA Jetpack 7
If you have never used a NVIDIA Jetson device, you might assume that the process is that you boot to an arm64 Ubunutu image, install that, then add NVIDIA packages. Instead, you install NVIDIA Jetpack which is like a custom NVIDIA version of Ubuntu.
Jetpack brings together not just the OS, but a lot of the software needed to get these embedded platforms online and operating. Instead of just looking at a screenshot of the Ubuntu desktop with a NVIDIA background, the above shows a bit more about what Jetpack covers. In the embedded space, this bundling of an OS and tools is a lot more common than on the server side these days.
NVIDIA Jetson AGX Thor T5000 Block Diagram
NVIDIA has this block diagram for the Jetson T5000 modules:
This is a neat look at the modules in the new kits.
Checking out the NVIDIA Jetson AGX Thor
Here is our 14-core Arm Neoverse V3AE CPU and its lscpu output:
Here is the nvidia-smi output of the Thor. You will notice that even with CUDA Toolkit version 13, and the 580 driver needed to support the new CUDA version, there are still a lot of things reporting as N/A. At least we are getting something.
Taking a quick look at the topology, we see the 14 core CPU with 1MB of L2 cache and 64KB of both the L1d and L1i caches. We also see the 128GB of shared LPDDR5X and then the other devices.
Here is a quick core 2 core latency view of the Arm cores:
Those net mgbe interfaces utilize the nvethernet driver and a specific 6.8.12-tegra driver version so they seem to be using a different stack than NVIDIA’s popular ConnectX lines.
Here is the Realtek RTL8126 5GbE NIC in ethtool where you can see it also supports lower speeds like 2.5GbE and 1GbE.
The SSD we found was a WD/ SanDisk SN5000S. We reviewed the WD SN5000 earlier this year.
Next, let us talk a bit about performance.
Can it run larger LLMs (70B+ params) and if yes – how fast?
Yes – I think the number we are not publishing for llama 3.3 70b is in the low 10s of tokens/ second. We did not get to run this through a standard process to average out runs.
I’d love to see this compared to DGX Spark. On paper, this should have more GPU compute at around the same price.
Agreed jtl. I thought we were going to do GB10 earlier this month. We will get that to you when we can.
@jtl The DGX Spark seems to go for about double the Thor’s price. The only offering I found was >$6000. $5800 on alibaba.
What is the fan noise like? That huge cooling system looks like it might be pretty quiet. More of a pleasant swishing sound like Rosie from the Jetsons rather than the roar of my Dyson?