Yesterday, we published our MikroTik CRS812-8DS-2DQ-2DDQ-RM Review, but one of the big questions is what to pair with it in terms of optical modules. Of course, our recommendation is to use copper if you can. If you must go optical, then we found some Intel Silicon Photonics modules that have worked really well for us, and better than some of the other optical modules that we have tried. The plus side is that they are currently plentiful and cheap. The negative side is that there is a non-zero chance you end up with very rough optical modules. Still, we thought folks might like this one.
If you are in a buy-first-and-read-later mode, here is an affiliate link to the QSFP56-DD DR4 modules we purchased and recommend on eBay. Here are the longer-range 2km DR4+ modules that work fine (affiliate link), but we will discuss why we would use those as more of a backup rather than a primary. We have ~50 of these (working) modules at this point and have been running them for between 30 days and many months.
These Are the Super QSFP56-DD 400G DR4 Intel Silicon Photonics Optics
The modules we have been buying are QSFP56-DD 400G DR4 optics. These come from Intel’s old Silicon Photonics business that used to serve hyper-scale customers like Facebook-Meta.
As a quick aside, the first time STH covered an Intel 400G DR4 QSFP56-DD optic was at Intel Silicon Photonics Update at Interconnect Day in 2019. 2019 was when 100G optics were still a big deal, so a 400G optic was very exciting indeed.
Now though, you can find these 400G-DR4 and 400G-DR4+ optics inexpensively on the second hand market. To make this easier on folks, the yellow tab ones are the 500m reach DR4 optics and the blue tab modules are the 2km reach 400G-DR4+ optical modules.
The DR4 may not seem exciting, but remember in a QSFP56-DD optic, there are eight 56G PAM4 or 50Gbps lanes which give us our 400G speeds. Normally, on something like a SR8 optical module, that means eight lanes with one transmit and one receive channel which go on different fibers. So the SR8 optical module requires 16 fibers, hence why it needs MPO/MTP-16. The DR4 optical modules not only go further, but they only use four channels meaning we can re-use our MPO/MTP-12 cabling (eight of the twelve fibers are used.)
This may not sound like a big deal, but it means that the modules need an internal gearbox to take eight electrical channels and transmit them over four optical channels.
While we do not need silicon photonics, and we do not need 500m or 2km reach of these DR4 optical modules, we have standardized on the DR4 modules in the lab because it makes connecting with other components we need to get to our big Keysight CyPerf load generator easier. One of the reasons we have been doing our series is that at 400G and 800G speeds, the physical connectivity is no longer trivial. Many folks are finding it difficult to align the physical connectivity when building smaller AI clusters.
Now these DR4 optics work not just with other DR4 optics we have like the OSFP modules we have in the lab, but also the DR4 and DR4+ optics will connect to each other. For the MikroTik CRS812 DDQ, there was a huge difference between the DR4 and DR4+ optics.
Check the Max Power field in the two modules. That 3.5W for the yellow tabbed 400G-DR4 modules versus the 14W for the blue tabbed 400G-DR4+ modules may not seem like a huge difference, but in the MikroTik, and other switches it can. First, two of them mean significantly more power on a lower power switch like the MikroTik CRS812 DDQ. Second, the fans on the MikroTik added over 8dba when we were blasting 800Gbps of bi-directional traffic through the switch.
Between the power and the noise aspects, I would strongly urge folks to consider the yellow tabbed 500m modules over the 2km ones. The Cisco InnoLight SiPho modules used slightly less power at 13.5W, but those also caused the switches fans to spin faster. All of them were able to handle 400Gbps bi-directional speeds, but in a switch like that it is better to save power. We will note that in any case, the power and noise go up quite a bit with optical modules versus copper DACs, but if you need to span a longer distance, that is the price you pay.
Final Words
These are nowhere near as exciting as the Intel Silicon Photonics 100GbE QSFP28 500m Optics for $6 from a few years ago. Those used less expensive cables and came in much better condition.
On the other hand, 400GbE is more costly today. A SR8 optic like the FS 400Gbase-SR8 400GbE QSFP-DD Optical Transceiver we reviewed requires 16 fiber MTP-16 cabling but costs $220 or so. Even low-cost DR4 optics new often cost $520-$750 or so. Getting a $500-750 module used for $29 feels like a decent deal. We have seen the blue tabbed 400G-DR4+ modules for $69 or less. If you need 1.5km more reach, $40 is going to be a small cost.
Something I want to warn folks on is that we have purchased dozens of these modules and we probably had about 1 in 8 that came in rough conditions with dust in the connector (do not expect covers on them), broken latches, dinged connector PCBs, or other abnormalities. Even Jabil did not buy the Intel Pluggable Silicon Photonics Business you would still be on your own for self warrantying. Luckily, these modules were known for being reliable, so if you need 4 and buy 5 or 6, to self-warranty you are probably going to be OK. That is one big drawback of this “deal.”
If you want to pick these up, here is an affiliate link to the ebay listing we bought from for the 500m units we would recommend. Here is an affiliate link to the DR4+ 2km optics that we probably would not recommend unless you needed the extra reach.
Thanks for being honest that not all are working since they’re listed as C condition. I’m still buying a few because hell if 1 in 10 works I’m saving money.
What good are cheap modules if there are no cheap NICs ?
Patrick has had an article about some “cheap CX-7” but not a peep about where he got them.
Or is this something in the making – massive CX-7 price drop for mere mortals ?
Couple more questions:
1. Is there or will there be affordable 100/200/400 Gbe module that uses just one fiber pair ?
2. How is one with homelab supposed to deal with that many-fiber fiberoptics ? Is there an affordable source that manufactures cables/connectors in custom lengths ? Or are premade standard lengths the only option ?
@NotReally Me
1) Affordable being subjective but yes, 100 Gbit pricing is set to come down this generation and next. Most of that is going to be pushed by SFP112 arriving on switches. A fiber pair is going to be the more common implementation of that. This also means that single pair modules for older QSFP28 ports are going to become more common place but will carry a premium as they need some internal logic to merge four 28 Gbit SerDes into one 112 Gbit SerDes for a single 112 Gbit optical PHY.
Faster speeds like 200 and 400 Gbit are going to remain expensive as they’re going to remain multilane technologies whose simplest implementation is multiple pairs. Single pair optics do exist but you maybe looking at WDM. This is still multilane but each lane gets its own wave length of light where each frequency is then merged onto a single cable for transport. Expensive optics and transceivers are required to do this on top of higher cabling requirements. There are other ways of doing these high speeds over a single pair but this is the most common and even at their outrageous prices, the more economical solution.
2) Pre-made cables are generally abundant. I’m lucky to have a Microcenter close by and they stock various SM and MM single pair cables of various lengths in the same aisle as you’d expect to find copper CAT wiring. The one thing they do not carry is bulk fiber.
However, there is a great local electronics store that various cable running companies in the areas use to purchase bulk fiber and CAT cabling. One of the services they offer is to make custom length fiber cables. Inexpensive for a single pair configuration and it doesn’t really matter what ends (LC, SC, ST etc.). Thus they’ve pulled through on the odd job that’d normally require a standard length but different ends. I have not checked if my local store will do MPO style which requires multiple pairs to function but I can order standard premade lengths through them. If you live in any decent sized metro area, there are probably a couple of them local to you worth checking out. If not, some will ship to you.
Is there an inexpensive fiber fusion/termination kit you recommend. Of course I am not going to terminate at 8 pair cable, but I want to run some fiber in old house and terminate at the wall box. It would be nice to have LC to the wall.