Introducing Project TinyMiniMicro Home Lab Revolution

28
Project MiniMicro Cover Introduction 800x450 Cover
Project MiniMicro Cover Introduction 800x450 Cover

Project TinyMiniMicro is something I have wanted to look at for some time. There are a lot of home/ SMB server users out there who need large amounts of storage. Others simply need a server to run a WiFi AP controller, a VoIP solution, and a few other services/ development VMs. While there are some that need massive compute and memory footprints in single machines, there are others that prefer building smaller clusters. Project TinyMiniMicro is not for everyone, but many of our users are going to be wowed by the results. TinyMiniMicro comes from the fact that Lenovo, HP, and Dell are the primary manufacturers of these nodes and those are the tags they use respectively to indicate they are small form factor workstations.

Video Version

Since this has been a major undertaking at STH, we have a discussion of the units both in article format as well as a video version:

Of course, we have a bit more detail below that is harder to get all of the detail into a video.

What and Why Tiny/ Mini/ Micro PCs?

There are a lot of technologies out there that one would want several physical nodes. For example, one may want Proxmox VE deployed across more nodes for HA style setups. One option is to use nested virtualization, but sometimes this is not what one would want. Further, sometimes you may have a big node, and want two smaller nodes simply for offload purposes when the main node needs to go down for service.

This may seem crazy to use these PCs as servers, but maybe it is not. Chick-Fil-A came out and stated that they are using Intel NUC platforms for in-store Kubernetes clusters. You can read about that here. Intel NUCs are very hip and very popular, but they are also quite costly for what they deliver. The major PC OEMs each have platforms that follow a similar style, huge volumes, great quality, and excellent support. Frankly, while NUCs are trendy, TinyMiniMicro is the segment that drives volume and has vendor diversity.

Project TinyMiniMicro Cover Old
Project TinyMiniMicro Cover Old

My thought on the nodes is that basically:

  • There are a ton of these corporate mini/ micro desktops
  • Customers for them do regular upgrade/ refresh cycles plus there are events such as bankruptcies that lead to a constant supply on secondary markets
  • They are likely to be sold by the large vendors in most countries due to their lower cost
  • The build quality of these devices is fairly good and they are easy to service
  • Allegedly, some even come with active onsite warranties. New units can be purchased with onsite warranties

When we look at cluster builds with the Xeon E-2200 series, such as with the HPE ProLiant MicroServer Gen10 Plus, something we noticed was that we either wanted more low power compute nodes, or we wanted smaller and lower cost nodes without 3.5″ bay support.

Project TinyMiniMicro HPE ProLiant MicroServer Gen10 Plus And TMM Nodes
Project TinyMiniMicro HPE ProLiant MicroServer Gen10 Plus And TMM Nodes

There is a lot to like here, but let us get to the obvious question: why not a Raspberry Pi 4 now that they are 8GB capable?

How Raspberry Pi 4 8GB Spawned this Project

For anyone that is thinking of building a small cluster of nodes, the Raspberry Pi ecosystem is awesome. They tend to be fairly inexpensive, small, and low power which makes integration extremely easy. When I heard that the Raspberry Pi 4 8GB model was launched, I immediately went to go order some. Part of that was seeing how it would impact our AoA Analysis Marvell ThunderX2 Equals 190 Raspberry Pi 4.

I ordered two kits from CanaKit, one was a barebones starter, one had a bit more functionality. Although the RPi 4 is advertised as a $75 device, the starter kit with the 8GB RPi 4 was around $140 before tax and shipping. I know you can get the components less expensively, but I like supporting the ecosystem and frankly having the components picked for me was worth a few extra dollars.

Raspberry Pi Canakit Starter Kit
Raspberry Pi Canakit Starter Kit

These units took a long time to arrive. When they did, I had kits of parts, not completed systems. So for just over $300, I had 8 low-power cores, 16GB of memory, in two nodes that I had to assemble and then had to set up power, add networking for, and so forth.

While I was waiting for them to arrive, I had the thought that perhaps I was struggling down a mentally inept path for what I was trying to accomplish. That is when I started looking at the TinyMiniMicro (TMM) nodes. What I found was an interesting comparison.

Here are some notes on the comparison:

  • Raspberry Pi 4 8GB models may be $75, but once one buys all of the accessories to make a single node, they end up being $120-150.
  • Some of the TinyMiniMicro units I have been buying in the $250 range have at least 4 threads and 8GB or 16GB of memory (upgradeable.) Usually, I have been getting 256GB SSDs with the units that are higher quality than the 32GB SD cards for the RPi’s.
  • Expansion on TMM units usually allows a 2.5″ and a m.2 SSD
  • While RPi’s are way better for IoT and such, as a lab, where power is a concern, but perhaps not the only concern, a 16GB/ 256GB x86 device at $250 is not that much of a worse value proposition than an 8GB/ 32GB (SD Card) RPi 4
  • They are designed to be silent except under heavy loads
  • They tend to use lower-power processors
  • They are small and stackable
  • A negative is that WiFi is optional where it is a core feature of the RPi 4

The more I looked into it, the more buying a TinyMiniMicro PC made sense. These had many features similar to the Raspberry Pi 4 8GB but effectively traded power consumption and a bit more size for a massively more robust platform.

By the time the Raspberry Pi 4 8GB kits arrived, I already had bought the first batch of TinyMiniMicro nodes and started iterating.

Project TinyMiniMicro Nodes

The first nodes I purchased were HP EliteDesk 705 G3 Mini units with the AMD Pro A6-8570E chips. These had 16GB of memory each and 256GB SSDs for around $225 after tax and shipping. Those are not great processors by any means. My thought was that I am usually more RAM than CPU limited, so it would be fine. This made me realize a few things. First, these TMM PCs have great build quality. The chassis on these inexpensive nodes has a single thumbscrew to get into and it is made mostly of metal giving it a surprisingly good quality feel. The units were small. The RAM used standard SODIMMs so that could be easily varied. There was room for a 2.5″ SSD, M.2 SSD, and even an extra M.2 slot for WiFi. The CPUs are actually socketed but HP has a nice cooling solution that runs fairly quiet. It even had multiple display outputs which makes sense as a corporate desktop.

Project TinyMiniMicro HP EliteDesk 705 G3 Rear IO Customization
Project TinyMiniMicro HP EliteDesk 705 G3 Rear IO Customization

There was a lot to like, and then the killer feature Windows 10 Pro. These corporate desktops have their Windows licenses tied to the motherboard. As a result, this $225 device has a Windows 10 Pro license which is worth $100 alone. With Windows 10 Pro one can join these nodes to a domain but also get remote desktop capabilities among other features. While some may scoff at Windows, Microsoft is doing an awesome job providing features such as WSL2 so you can run popular Linux distributions alongside Windows 10. That means you can have a Windows desktop that also is a lab node. Or if you are running Linux on it today, you can convert it into a Windows web browsing machine in the future.

Project TinyMiniMicro Windows
Project TinyMiniMicro Windows

While the cost is around $70-100 more than a RPi 4 8GB, you get a lot more machine for your money and in some cases more than two RPi 4 8GB models for less.

The processors on those nodes were anemic, so I then found two HP EliteDesk 705 G3 Mini nodes with AMD Pro A10-8770E CPUs for only around $200 each. 8GB SODIMMs sell for around $27-30 each so if you get an 8GB unit with 1x 8GB SODIMM, it is relatively inexpensive to get to 16GB.

Project TinyMiniMicro Dell Micro And HP Mini Interior Drives
Project TinyMiniMicro Dell Micro And HP Mini Interior Drives

Installing Linux on these machines was straightforward (although sometimes requires disabling security features meant to prevent this in corporate scenarios.) Within about an hour, I already had Kubernetes installed.

HP EliteDesk 705 G3 Kubernetes
HP EliteDesk 705 G3 Kubernetes

At that point, I had a 5-node cluster and realized, this was absolutely awesome which is how Project TinyMiniMicro was born.

Initial Nodes In Our Population

We have been working with about 15 different TinyMiniMicro nodes and 18 of the machines in total for more than a month. Here is the quick list:

Lenovo Tiny

  • Lenovo ThinkCentre M710q Tiny
  • Lenovo ThinkCentre M715q Tiny
  • Note STH reviewed the M720q Tiny previously
Project TinyMiniMicro Lenovo ThinkCentre M715q Tiny Front
Project TinyMiniMicro Lenovo ThinkCentre M715q Tiny Front

HP Mini

  • EliteDesk 800 G2 Mini
  • EliteDesk 705 G3 Mini (AMD A6)
  • EliteDesk 705 G3 Mini (AMD A10)
  • EliteDesk 800 G3 Mini
  • EliteDesk 705 G4 Mini
  • EliteDesk 800 G4 Mini
  • ProDesk 600 G4 Mini
  • ProDesk 400 G4 Mini (was being sold as an EliteDesk 800 G4)
  • ProDesk 405 G5 Mini
Project TinyMiniMicro HP EliteDesk Prodesk Front
Project TinyMiniMicro HP EliteDesk Prodesk Front

Dell Micro

  • OptiPlex 7070 Micro
  • OptiPlex 7060 Micro
  • OptiPlex 5050 Micro
  • OptiPlex 3050 Micro
Project TinyMiniMicro Dell OptiPlex 7070 Micro Front
Project TinyMiniMicro Dell OptiPlex 7070 Micro Front

All of these nodes are a similar size, use external power bricks usually shared with their respective company’s notebook lines. There are usually internal spots for 2.5″ and NVMe SSDs as well. Almost all of them also have the ability to customize the display outputs. All have DisplayPort, some have a mix of HDMI and VGA as well. All of our test units fall in the 10W or so idle to 64-68W maximum power consumption range even while spanning 2-6 cores. Performance ranges from somewhere in the Atom C3000 range to the lower-middle of the Xeon E-2200 range.

Project TinyMiniMicro Lenovo ThinkCentre M715q Tiny Rear
Project TinyMiniMicro Lenovo ThinkCentre M715q Tiny Rear

Having this many different units makes it abundantly clear that this is a highly competitive segment where the companies are investing heavily in design and features. All have excellent documentation. Most consumer-level and server-focused SFF cases would not be competitive in this market.

Management Details

While typical servers have built-in IPMI and Redfish management, along with features such as BMCs, some of the TinyMiniMicro PCs have additional management features.

AMD AMC Management Console
AMD AMC Management Console

Many of these nodes have Intel vPro and AMD DASH. Both let you do remote power on/ off. Intel vPro is probably the more robust solution. Neither is as good as a standard server BMC, however, there are some benefits, namely lower idle power consumption.

Intel AMD VPro IKVM Ubuntu Setup
Intel AMD VPro IKVM Ubuntu Setup

The management situation is one we are going to look into more deeply with the nodes. Depending on the chipset used, and the markets that the particular units can be exported to, one may or may not have full Intel AMT/ vPro features.

Project TinyMiniMicro Intel Core I5 VPro
Project TinyMiniMicro Intel Core I5 VPro

Final Words

If you purchase one of these units, especially second-hand, you may end up with a great deal, but there can be disasters as well. For example, one of the HP models arrived and it was a similar yet different model. Some were missing parts as well. Others had very interesting default BIOS settings.

By getting a relatively large population, we were able to see many of the configuration differences and purchasing pitfalls beyond just what one may see on a single listing. We have had forum users working with these nodes for years, so we decided to purchase a large set of nodes (and they are still arriving) so we can share in-depth insights into each model along with some of the general trends and potential pitfalls of the space.

Perhaps the most important aspect of this project is that these nodes can be purchased new and deployed similar to how Chick-fil-a does with the Intel NUCs, with the added benefit of getting on-site support to further bolster warranty and remote service and support. While we are discussing lower-cost options here, there are many organizations that can deploy these edge compute solutions easily since they already purchase the Lenovo, Dell, and HP units with volume discounts. The fluidity of being able to use the nodes as desktops or servers is something that the Raspberry Pi lab folks often touted as an advantage.

This was just a quick piece to introduce the new project now that we have over a month of testing this large population. Expect more to come on what we have learned from testing all of these units for our Project TinyMiniMicro home lab revolution.

28 COMMENTS

  1. For those looking to purchase these kinds of nodes the few things to look for are the type of CPU, the memory configuration and storage. In a perfect situation you’d buy a used one with a single stick of 16Gb RAM and a 120/128/256Gb M.2 disk, nice to haves are the Intel T series CPUs like a 6500T.

    Best ebay/second hand keywords are USFF + 6500T + SSD. Typical price of the most desired configuration is right around 200USD

    You can also find NGFF network cards to add a secondary NIC you these nodes, but it may require some grinder action to cut an opening in the case.

  2. I’d love one for routing purposes but I’ve never seen one with two nics. Do they exist? Closest I’ve come is some vague references to an addon card for certain lenovo models.

  3. @patrick there are a few companies out there that make m.2 network adapters you could use although depending on the exact adapter and case design it might take some time with a dremel to mount it properly

  4. Patrick, The high end NUCs, Fit-PC Fitlet, or Supermicro E300-9A are the only comparable systems with multiples NICs that I’ve found. None of them are as readily available as refurbed Micro PCs though.

  5. I have used small style PC chassis for years as routers. I build my own PC from various hardware selections and a certain small ITX (0.5 cu.ft. maybe) PC chassis (with 1x 80mm and 2x 60mm fans) I can find only at Fry’s. Yes, I could look for a pre-built computer with multiple NICs, but that generally means getting something in a proprietary form factor in a NUC-sized chassis with ZERO expansion possibilities (no slots or extra SATA ports or extra headers) and RealTek NICs (UGH!). Ok, I prefer Intel NICs for long-term always-on usage scenarios while RealTeks have consistently given me trouble and reduced features (relative to Intel) over the years.

    I have a personal stash of older Supermicro motherboards with older Intel Atom “in order” processors. Maybe that makes a bad router in your opinion, but you would be wrong. If you don’t overload your router with “oohh ggee SHINY” software it works incredibly well, like 1 to 2 percent CPU utilization at most; 1 to 2 GB of RAM usage out of a maximum of 4 GB; 2 onboard Intel GB NICs (tested to GB speeds in a LAN environment); VGA port; and good Linux OS support. Expansion? It has 1 slot for a PCIe x4 NIC; I have those with Intel chips. A total of 6 SATA ports onboard. But you want faster, like NMvE or M.2 or whatever? Get real and realistic about your expectations; it’s supposed to be a router not a Swiss Army knife or a gaming platform.

    Seriously, I am a big fan of tiny and silent computing projects. The critical factor is keeping your expectations under close control. That means you might have multiple devices where other people would pile all their stuff into 1 chassis.

  6. No dual NICS for tinyminimicro’s. for that you have to go SFF, They tend to be a bit cheaper though, so if you can “tolerate” the larger format, you do get access to both 3,5″ drivespace (+m.2+2,5″ usually). Powerconsumption will rise though as well.

  7. @Patrick I modded a Lenovo M900 Tiny to add a secondary NIC (iocrest SI-MPE24043 and iocrest M2F8111 both worked fine). From cursory research they all use the same realtek chipset so unless you need the combined speed you may be better off with VLAN tagging on your switch as it was a real pain in the ass.

  8. Patrick, certain Lenovo Tiny’s have a PCIe 3.0 x8 slot that you can install an Intel i350-T4 into.
    I’m currently using an M720q as a pfSense firewall with this NIC. You can purchase certain Tinys with these pre-installed or purchase them as an option. I recommend purchasing the kit (4XC0R41416) from Lenovo as it will include the PCIe riser as well as the bracket to mount into the Tiny. The kit also includes the SFF & Tower brackets too.

    Disclosure: I’m a tech enthusiast that is a Technical Client Advisor who works for Lenovo.

  9. Hey Bob,

    Any idea where to get that kit 4XC0R41416 you mentioned for a reasonable rate? $300 is awful steep as that’s nearly the cost of an entire M720Q on the used market…

  10. Tim,

    I bought two of them on sale a few months back to replace one’s that I’d bought cheaply off of eBay years ago. (Thanks to STH for showing me that these cheap ones were knockoffs, it was painted, not etched “DELTA” chips on the ones from eBay.)

    Sadly, new, retail i350-T4’s are not inexpensive. But what you’re really looking for is the Tiny mounting bracket and the PCIe riser. I looked for about a year before buying these, in service parts, etc. I tried rigging it into another Tiny (P320 Tiny ThinkStation by removing the P620 Quadro graphics card) but it wasn’t solid enough to make me comfortable in making it my firewall as it did not have the retention bracket.

  11. I have an M900 Tiny running as a router with a second Intel NIC in the form of an M2 e-key card from Commell in Taiwan (model M2-210).

    Commell support were kind enough to send one to Australia when I contacted them; couldn’t find Intel M2 NICs anywhere else at the time.

  12. All of those mini-desktops require an external power brick. You should always have it in the picture for size/weight and level of cable awkwardness. The OEMs can market them and create nice looking pictures which never show the power brick, but you shouldn’t.

  13. Unless the focus is on setting up Ceph storage or a similar type of redundancy, not all the computers in a home lab need separate power supplies, cases and disks. In particular, it is possible to scale out a cluster of quad-core 8GB Raspberry Pi computers using a beefy 5V power supply and network boot at a price much closer to $75 per node.

    In my opinion, a lot of things (including Docker and Kubernetes) can be done using “diskless” computers with iSCSI network block devices served from a good NAS. Avoiding SD cards also allows for easy setup and maintenance because all that’s needed in that case is a way to remotely reboot the machines. This can be done either with POE or by running wires from the GPIO of a designated controller Pi to the run enable pins of the others. The latter is not such as mess as it sounds.

    On the other hand the TinyMiniMicro project sounds close in power efficiency and includes the possibility of running any of the BSDs and, as mentioned, the Windows operating system. Buying those little x86 computers used was an option I hadn’t considered but sounds very reasonable given typical desktop refresh cycles and when they first appeared. I’m quite interested to hear about how to enable remote management on these machines.

  14. I’m running a All Flash VSAN cluster on 4* HP G4 prodesk 400’s 64GB, 256GB NVMe, 1TB SATA SSD.

    Networking for vsan is via qnap 5GB USB nic (set to 2.5GB)

    Connected to the Wifi slot is extension cable to pcie X1 riser to run I350-t4 nic fro each.

    The wifi slot hase 2 X PCIE3X1 connectors so you could also run GPU external for ML/AI etc. so may get around to building my own dual riser board for them.

  15. Router with a single NIC works if you have a VLAN capable switch with a spare port for your modem.

  16. I bought a pair of HP EliteDesk 800 G5 Mini to refresh my home lab, one with an i7-9700K, the other with an i9-9900K, both with 64GB RAM and dual NVMe M.2 SSDs. Very nice and fast machines, the only drawbacks being the lack of 10G Ethernet and Xeon/ECC. Also, in markets like the UK you cannot skip the Windows tax, even on a build-to-order configuration.

    My use cases are programming so I don’t care all that much about GPUs.

  17. Patrick (and others) – What does your software stack look like? Ubuntu here it looks like, but what “bare metal” installs are you doing to get a managed cluster environment? I’ve been using Proxmox/Portainer for awhile, but very green when it comes to K8s.

  18. @Mark – A router with a single physical NIC and using VLANs to separate the different networks is called “router on a stick”. That design is fine if ALL of your networks are internal and under your control, but remember the bandwidth limitation of using a single physical port.

    If your router has to perform any security duties, like firewall the Wild Internet from your home network, then don’t use a “router on a stick” design.

    Some VLAN switches are still vulnerable to “VLAN hopping” attacks. In the “router on a stick” design the VLAN switch is the most critical point in the design; it has to work properly (isolate ALL packets, including b-cast and m-cast, from each VLAN from every other VLAN) and function in a secure manner, or you have no security.

  19. Could I use a bunch of these to render 4k60 videos? I’m using Adobe Premiere Pro CC 2014 currently with a ton of filters for the videos. 10 minutes of video takes 6 hours to render. Let me know please and thanks!

  20. @Jamie Adobe Premiere unfortunately has no support for multi-node rendering to create a render farm. As a result you’re better off with a single high performing machine (such as using a Ryzen 3950X, ThreadRipper 3960x or Intel HEDT like a 10980XE) as each node would only take a single video to render.

  21. I would really like to consider HP ProDesk 600 G2 for a ceph / kubernetes setup, the only thing that is missing is ECC Ram. I read about the CPUs that are used in these devices. May ECC Ram work although it is not supported?

  22. Thanks for the great article. I got 4 Lenovo M93p (i5-4570t) for 1 euro in my company. I have a lot of fun between Proxmox and old console emulators 🙂

  23. What is the maximum disk height for each of the shown models?
    Does a 15mm 2,5″ drive fit in all of those?

  24. Sleepy:
    > I have a personal stash of older Supermicro motherboards with older Intel Atom “in order” processors.

    > Seriously, I am a big fan of tiny and silent computing projects. The critical factor is keeping your expectations under close control.

    Me too. I like older SM Atom boards precisely because they are quiet. For me this is most important in a home server environment, where expectations are low.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.