Qotom Q20332G9-S10 External Hardware Overview
The front of many fanless systems we review is generally a USB port, maybe audio connectivity, and a power button. This has a power button and a USB Type-C port, but then things diverge.
First, we get a VGA port. This uses an ASPEED AST2400 (no iKVM that we can find) for video. That uses relatively ancient GPU IP so do not expect hardware offloading for media encoding or even the ability to run OpenCL compute benchmarks with Geekbench. This is more server video output than consumer mini PC.
There is a SIM card slot that we did not get to test, but it is there and labeled. Perhaps the most interesting, however, is the SFF-8087 port. SFF-8087 is the internal connector used for connecting motherboards and SAS controllers/ HBAs to storage and was popular up until the SAS2 generation. Normally, for an external interface, we would expect SFF-8088, but that is what we have. In our key lessons learn, we will connect this to a 4-bay QNAP JBOD as the connector carries four SATA III 6Gbps lanes.
On the subject of being interesting, we have a lot of I/O. We have two USB ports and a serial console port on the back, along with the 12V power input. The real star is the networking. We have five 2.5GbE ports based on the Intel i225 NIC series, not the i226, which is frankly a bummer.
There are then four SFP+ 10GbE ports. These ports are Intel X553-based and driven off of the Intel Atom C3758 SoC.
The system is also fanless. So the top is the heatsink, and it does get very warm.
On the bottom, we added the rubber feet, but we do not see vents. Instead, there are mounting points for a VESA mount and a SSD, but to get to the SSD mount, we will need to get inside the system.
With that, let us get inside the system.
Thanks STH for actually doing real home-lab content. This is a nice system because it’s a real CPU with real NICs and fanless. I still want to see the ms-01 review, but I’m thinking this is a better value. You’ve got the makings of a 3 host Proxmox VE cluster with this for under $1000 and 100W. If each has that DAS you’ve even got enough to do Ceph.
Here is a question, If I have a Xeon-D 1540, is it worth replacing it with this? Currently running it as my firewall, but experiencing odd latency I can’t put my finger on with my knowledge of Opnsense.
I’m curious to know if the C3758R version 4 lane M.2 slots? The R version has 20 PCIE 3 lanes instead of the 16 PCIE 3 lanes on the C3758.
Hoping you guys can publish a review of it configured as a router or firewall. It’s no great shakes in the CPU department, nobody will buy it to compile C++ code…
Some of these Aspeed chips on Board, but no Mgmt-Software on it, very curios!
Maybe you can test some switching and routing performance of this board, would be nice.
It’s got the same CPU as the Netgate 8200 Max that’ll do 18Gbps firewall iperf3 in pfSense
Here’s the speed using the C3758R version in the Netgate 8200 Max specs:
L3 Forwarding
IPERF3 Traffic: 18.60 Gbps
IMIX Traffic: 11.76 Gbps
Firewall (10k ACLs)
IPERF3 Traffic: 18.55 Gbps
IMIX Traffic: 5.1 Gbps
IPsec VPN (AES-GCM-128 w/QAT)
IPERF3 Traffic: 3.24 Gbps
IMIX Traffic: 810 Mbps
That’s why you want QA.
These aren’t new processors. They’re used in so many branded firewalls and other appliances I’m shocked there’s questions on the network perf.
7 year old Atom C3758 = FAIL
Can that ancient 14nm chip route even one 10Gbps connection let alone **four**?
@Patrick – can you confirm if the system actually supports ECC?
although the “barebone” 18w idle is high, and 35w with 3 SSDs suggest some issues… perhaps some bad support in power saving features by the bios? …. unless of course the issue is with drives used – can you perhaps re-test this with some “known-to-be power-saving” drives?
(I’m interested, since I’m in search of possibilities for low idle power home server)
Uh, Carl, your question was literally answered by AldiK in the comment before yours.
It’s a cool system. I’d buy 2 barebones and use them for silent HA cluster.
Stanos I looked at their Supermicro A2SDi-8C+-HLN4F motherboard review with the same chip and it’s really close 19.1W and 32.5W https://www.servethehome.com/supermicro-a2sdi-8c_-hln4f-review/4/
Perhaps that’s what the C3758 uses?
@Patrick did you test the the SFP+ ports with any linux distribution using kernel 6.*, including Proxmox? They work in FreeBSD based systems (*sense).
I have a similar system and the intel x553 SFP+ does not work in Proxmox 8.* reported in Bug 5103 – https://bugzilla.proxmox.com/show_bug.cgi?id=5103
Whether the networking capabilities of an Atom-based firewall appliance are well known or not, I’d prefer that a technical review of such an appliance verify the expected performance and include the results in the report. For that matter, how fast it takes to compile a Linux kernel seems as relevant as using the firewall appliance to warm a cup of coffee. Even so, I found the review and hardware interesting. Thanks!
Would it be possible to introduce a testing methodology for firewalls and networking?
10 Gb ports driven by Intel X553. Last I looked Linux kernel 6.xx did not support the X553. Has this changed? If not this seems to be a rather severe limitation of the unit for any of a variety of new releases. At first glance this looks like a good candidate for an inexpensive experimental Ceph network. But if the 10G ports cannot be used . . .
If that had something like an N305… i’d be all over it.
I was also excited to use these systems as routers. Unfortunately, 4 of them failed over years, after approx 1 years of 24×7 use consistently.
I am still in favor of fanless systems, but am staying away from Qotom specifically as unreliable.
Unless you want to have a spare server fully configured on standby ready to replace a failed system, those are not reliable.
As a former Netgate staffer I found the 8200 price and rack requirement off-putting.
This week as I needed to purchase a 10G firewall for an event kit I considered the 8200 (despite the experiences I had at the company) and I absolutely could not justify the $1300 price tag under any circumstances and purchased this specific firewall on Monday for the kit.
I am looking forward to getting it on my test bench in a week or two.
Netgate needs to up their game – the 8200 price tag is way too high, but then again all their hardware prices are very high.
Does anyone know if this will run VMware?
@Curious: I don’t see a reason it could not run it. It might not be the current release, though… so you should check the VMware HCL against the known parts of this system to be sure.
buckm – the C3758R is x2 bifurcation as well, so you do not get more lanes.
SteveP – Just ran iperf3 between the C3758 and C3758R both running Proxmox VE 8.1.3 and they are passing traffic at 9-10Gbps. See https://forums.servethehome.com/index.php?threads/the-everything-fanless-home-server-firewall-router-and-nas-appliance.42718/post-406041
This would be an interesting router if it had working IPMI. I just can’t bother with a server that does not have IPMI. Not worth the time and effort to hook up a monitor, keyboard, etc. when something goes wrong and I need the console.
This isn’t bad, but as others have pointed out, the lack of iKVM isn’t particularly great.
Furthermore, that 3052 m.2 slot underneath the SSDs is a horrible idea. High-end 5G modules can easily dissipate enough heat to require a heatsink and having to install them underneath the otehr m.2 drives makes that impossible.
Hey, would the USB-c allow a USB-c DAS?
Such as https://www.qnap.com/en/product/tl-d800c
Great box – sad the QTOMO currently does not ship to Germany. Probably some misconfiguration in their Aliexpress store.
@George Salambassi – there is no USB-C on this device
Sadly I cant order this unit in germany
@George Salambassi – oh I see the SS+ on the front now… ehh. Probably?
Patrick – It is good news about Proxmox (8.1.3 maybe typo?). This is using the 6.5 kernel. Ubuntu LTS is still using 6.2, which at least originally had the problem. It looks like the X553 problem must have gotten fixed in the newer kernel. I don’t know if Canonical has backported the fix since I have avoided the X553 for this reason. I appreciate your having done the test.
Can you do a test where you put all 5 2.5g ports into a l2 bridge and then test this as if it was a switch?
I’m sure it can’t do full line rate on all ports, but how much can it do?
If I had this, I’d use it as a firewall and seriously consider just connecting 3 or 4 devices directly to it and not bothering with a separate switch. With 1 wan link, one access point, a 10g nas and 3 wired hosts there would still be one free port.
ECC works?
The motherboard has 2 SATA ports but only one SATA SSD was connected for the article. Is there internal mounting space for a second SATA SSD?
@Tim – I just got mine and, no. Just the single 2.5 screw holes but you could certainly bodge it for SSDs.
@Robert, true, the lack of IPMI or iKVM is not great, but at IPMI cost 5-9W (depending on implementation), but a console port is also not bad, you can do connect it to a RPi Zero W and have remote administration.
@Robert, missed the AST2400, then it is strange that they have not added the IPMI features, maybe a bios bug.
Also wondering if ECC SODIMMs are confirmed working, would be ideal for a proxmox host with ZFS
It also doesn’t work with alpine linux 3.19 no igxb (pretends to be there but no link ever)
ECC so dimms work, but the 10gbps thing is a MESS. I can’t reccomend these if 10gbps is your desire. I tried three different brands of twinax cables sfp+ that work fine with my ciscos, and other (brocade etc) they REFUSE to connect to ‘switches’ which include a zxytel XGS1210-12 AND an aruba 1930 24G Class4 PoE 4SFP/SFP+ 195W Switch JL683A.
The really weird thing is if I ‘cross over’ the sfp+ cable to its two interfaces it comes up and works fine. It just won’t connect to anything else 10gbps that I have with multiple twinax sfp+ cables (including the expensive cisco ones)
So if you want this for 10gbps its a bit of wash. Its kind of sad that the reviewers didn’t do a better job of explaining how tempermental the sfp+ ports are. I read TONS of people having problems with this (the authors claim to not have had any problems) myself included (the problems part).
I even used the ‘same’ version of proxmox and couldn’t really ever get things working. I might try freebsd just to see if its working. Why can’t they tell us what SFP+ modules work with the 2#$$@#%#$^%#$% thing? Or even better yet which twinax cables actually work?
Mostly its a nice PC, just don’t count on 10gbps working unless you are willing to spend tons of time on it. I was sad, since I wanted it for a nas, but its kind of useless as that. It will make a nice vm box, I just will have to live without the 10gbps until someone kindly shows me the right cable.
So if you want the 10gbps modules to talk to themselves, the 10gps works great, otherwise IMHO 10gbps is not a reason to buy this box. I tried MULTIPLE operating systems, and multiple cables with no ability to get this working.
oh and they take an eon to reboot.
I just installed truenas core and it seems the BSD drivers dont’ hate the thing. The linux ones are a disaster so alas all things linux are out unless you are a scholar and a gentleperson.
Its working with my 10gbps switch (the aruba) and the cisco twinax cable. Seems to be a dorky linux driver than. Sadness.
Final update: it seems like twin Ax is a very bad way to use 10G. This thing seems to prefer Intel SFP plus cards. They’re not too expensive on eBay. I could not recommend using TwinaX when you have multi-vendor switches.
Thomas Munn: FS.com sells DACs where they allow you to pick the vendor code for each end.
On an unrelated note, my unit came with a 12V 7.5A PSU from Channel Well Technology. CWT has designed premium ATX PSUs for consumer brands like Corsair in the past, so I’d expect it to be pretty decent, not the type of generic PSUs we often see included with these mini-PCs. It’s also UL rated.
Dear,
what is de max amount of Memmory and storage for the Everything Home Server.
Does the Everything Home Server support ProxMox or VMWare ESXi
We need for small environments a small server solution.
Many thanks and best regards,
Wim Kobben, Erasmus IT Services
Tried multiple 10GTek SFP+ and SFP28 DACs into a Mikrotik CRS310-8G+2S+IN, CRS326-24G-2S+RM and Zyxel XS1930-10 and no dice. I think it’s probably the Linux drivers, but if you can’t run newer Linux Kernels it’s pretty worthless IMO.
Some updates on my continued travails with this box: I was running it in my server closet (which does admittedly get quite hot) and the passive cooling fails massively. The system completely froze and wouldn’t restart until I unplugged it and let it ‘cool down’. Seems OK in ambient nice 23C 70’sF but in anything approaching 95F (about 30 something C) it sadly dies. To answer the other person’s question 64GB max, with ECC. I simply cannot recommend this system based upon my own testing and attempts to make it usable in a reliable fashion. The ‘fix’ is to manually updated the drivers for proxmox (which require using DKMS and all kinds of unpleasant kernel things) and ‘hopefully’ it works. You also need to use the INTEL drivers (from intel’s out of tree distro) since some genius in the ‘in tree’ kernel COMPLETELY removed the autonegotiation function of the driver!!!!
Final say on this: I got all my fancy SFP+’s only to have the unit refuse to boot (aka power on) anymore. I am sending it back to amazon. STAY AWAY.
The C3758R version sold out quick! Luckily, I got my order in. This one’s going to become my new gateway! I don’t think this one will be a heavy lifter, as the atom only has E-cores, but it’ll be nice to offload some light tasks like pihole and maybe Home Assistant.
I suggest enabling the s3 – power up, after power loss
And regarding colsole
I suggest setting it to VT100, 115200 and the same with opnsense.
Then a simple RJ-45 cisco console cable to usb-c will work in the bios.
regards
Svemd
@Svend Gundestrup Did you have to do anything special to get the console port working? I don’t get any output from it with a Cisco console cable using minicom, but it’s possible I’m doing something wrong. :)
@Svend Same question – I’m using `minicom –device /dev/ttyUSB0 -b 115200 -t vt100` and the console is dead preboot out of the box (i only pulled out the drive and that’s it).
I just picked up the C3758 (non-R) and a Noctua 5V USB powered fan to keep it cool. Picked up some large rubber band to keep it attached to the top grill.
I’m currently planning on installing OpenSense on it. I was going to go with Pfsense, but decided at the last min to change course.
I’m going to use all the ports on day one once I set it up. I’m also planning on getting the Minisforum MX-01 to have both of these units next to each other, connected together via the SFP+ ports.
After reading everything here and what i’ve seen written around online, it appears going with an Intel branded SFP+ adapter makes the most sense. Any suggestions beyond that? Any other alternatives?
So I’m not sure what all the handwringing about SFP+ DACs is. I’ve just loaded Proxmox 8.1.4 on my new rackmount unit and I’ve tested it in my company’s lab with a 10Gtek CAB-10SFP-P2M (my go-to cable for mixed-vendor applications). I’ve tested it plugged into our Mikrotik CRS326-24G-2S+ (first pic) and Unifi USW-PRO-48-POE (second pic), iperfing to our TrueNAS scale server (single Intel X722+same DAC, also plugged into the Pro-48). https://imgur.com/a/X90QcUX
It’s perfectly fine. No boot hangups either with the DAC plugged in.
I’m going to redact the above comment for reasons that should be obvious when you look at the iperf results ♂️ More testing required, will be back.
Alright so I’m back. Even after trying a bunch of different options in Proxmox 8.1.4 to enable the X553s (see this thread: https://www.servethehome.com/intel-x553-networking-and-proxmox-ve-8-1-3/), nothing worked. So I nuked it and went back to 7.4X (completely stock) and wouldn’t you know it, it works like it’s supposed to now. https://imgur.com/a/xnySI4N This leaves no question in my mind about the DAC or switches being the problem – the new driver tree for 8.X’s kernel is absolutely the culprit here.
Got mine today. Loaded Proxmox 8.1.4 in UEFI boot (systemd-boot) and updated to kernel 6.5.11-7pve. Can confirm the X553s will NOT link. I do get a syslog note saying: ixgbe 0000:0x:00.0 eno#: detected SFP+: #, but no link/communication.
The QNAP is sold out EVERYWHERE! I called the manufacture and they said their was no lead up time on the box. dude, SO many people are getting this. any alternatives that use that SFF cable?
I absolutely love my Terra master is a bit pricier but it works!
message to everyone who wanted to buy :
i live in china and they had some models are back in stock, receiving mine soon :)
try to msg them on a aliexpress to get yours
version C3758R