At SC24, we learned something really neat. The NVIDIA BlueField-3 has a special version, and we are not talking about the BlueField-3 SuperNIC. Instead, there is a self-hosted version for primarily storage applications.
NVIDIA BlueField-3 Self-Hosted Version Necessary
At SC24, there were a few new storage servers with a twist. Instead of sporting an x86 or traditional Arm CPU, these servers had a NVIDIA BlueField-3 DPU installed as the host processor. Some of them included not just storage, but through the magic of a PCIe switch, were able to host a GPU alongside the NIC and storage.
It turns out that NVIDIA has “SH” or self-hosted version of the BlueField-3 DPU. These are able to expose the PCIe root and then have other PCIe devices like NVMe SSDs and GPUs attached. That is a bit strange since NVIDIA has been using the BlueField-3 DPU downstream ports for storage within servers. The best way I have had it explained is that it is for the gold finger ports.
One of the big benefits of the BlueField-3 generation is its significantly higher memory bandwidth since BlueField-2 was a single-channel design. That BlueField-2 actually had less memory bandwidth than the BlueField-1 as a result. With the BlueField-3 generation, we get faster everything making it a big upgrade. With its own 16-core Arm A78 infrastructure, it has a capable host processor for applications that do not have high compute requirements like storage. The BlueField-3 has become NVIDIA’s lower-power network-attached CPU solution as well.
Final Words
While the standard NVIDIA BlueField-3 is the B3220, the self-hosted models add “SH” and are thus B3220SH. It is interesting that NVIDIA has distinct cards here given the onboard PCIe switch they are using. The self-hosted version is in the NVIDIA docs, but it is something that not many folks talk about in the industry unless they happen to be building a storage appliance that uses the cards.
Hopefully this is a fun one for our readers.
I like the bottom right port with just the label “Black Cable”
BF2500 also provided a root complex. In the STH BF2 ZFS article (with the JBOF backplane) it was very vague about the configuration and mentioned using BF2 516A parts. I always assumed it was possible Nvidia provided firmware or some enablement for root complex (turning 516A into 516B).
I have some BF1 boards and they are useless – no way to get software for it, or a BSP (board support package) to build an own distro – despite having much more capability than a regular desktop system. I really hate NV for producing new and new hw, and dropping support for previous generations so quickly. That is a brutally forced short update cycle, not seen anywhere else. Same goes with Tegra chips – they never achieved complete support of announced features before getting EOLed.
@Daniel: Nvidia has always been a bad company to be stuck with unless you are either big enough(I suspect that Nintendo’s Tegras did mostly end up working as intended and will probably continue to do so for life of product) or what you wanted to do happened to align with what they wanted to sell. Awkward, because they are sometimes quite good at a given thing; but they always held it close enough to their chest that if they lost interest your luck dried up rapidly.
Probably even worse now that ‘AI’ has hit; since the shiny new business area is so big and so high-value relative to the legacy stuff that they can probably make especially dramatic arguments about the cost of supporting legacy and non-core products.
Beyond just Nvidia, though, ‘DPU’ in general seems like an area ripe for hardware abandonment. A lot more going on than more typical NICs, so anywhere between “there’s a fairly punchy embedded ARM device just burning watts here” and “there’s a dangerously outdated Linux appliance grafted into my server’s brainstem” if nobody is keeping an eye on the firmware (not just the old “oh, yeah, XYZ 10-40GB NIC pulls are crazy cheap vs. the ones based on the other chipset because RoCEv2 is basically a lie never try to enable it unless you like crashes” type abandoned NIC problems); and since so many of the offerings are either relatively early days or very closely tied to some specific hyperscaler’s pet project there’s not necessarily a lot of support hammered out (whether in the form of a relatively generic Linux BSP or a project more specifically aimed at making hardware agnostic management easier like SONiC or OpenBMC) if a vendor has lost interest in a device or you aren’t the intended customer.
I’ll be interested to see whether that gets smoothed out over time, if it’s mostly just a maturity problem; or whether the people who actually buy these things are basically OK with it because adding some powerful offload capabilities and smooshing much of the management plane responsibilities of the BMC into the system’s biggest NIC is reducing the amount of complexity they have to deal with and the ability of the secondary market and small scale operators to deal with it is not their problem.
It will also be interesting to see how ‘silicon root of trust’ appearing on all the spec sheets ends up shaking out. I can understand why people want that; but, depending on implementation, that could either be some relatively light-touch measured boot stuff(or heavier touch all-signed secure boot stuff that gets wiped during decommissioning or ships in customer provisioning mode); or it could involve a lot of expensive hardware aging like an Android handset.
@Daniel
They just have UEFI on them. You should be able to install a regular distribution w/o a separate BSP.
BlueField-1 never was really widely available. Was more of a trial balloon rather than the deployed product that BF2 onwards are. The latter has easily publicly available software.
@never_released – The BF2500 (516B / DPU controller) firmware and software is also not available. Only the endpoint cards have software available, which is unfortunate.