The Project TinyMiniMicro form factor represents an excellent opportunity for home-labbers and power-conscious small businesses alike to do something special. In an extremely small footprint, these devices can perform many tasks that once required 1U servers to accomplish. One of my favorite tech purchases ever was a Netgate SG-5100 firewall. That device is fast. It allows me to saturate my 400/30 connection with traffic shaping, full IPS/IDS, DNS filtering, and even more services. Doing all that, I never hit over 50% CPU utilization. The purchase was made as a future investment, with gigabit internet in mind.
When Patrick first announced Project TinyMiniMicro, one of the first things that came to my mind was pfSense. This build will be based on the low-cost HP EliteDesk 705 G3 Mini. For under $200, can I build a router that comes close to an $800 one? Of course, this may not be the solution for everyone. You may require an active support contract with a robust SLA for your firewall, but it is certainly worth exploring for some.
Some Considerations Before Proceeding
One of the first considerations to keep in mind is that the HP EliteDesk 705 G3 I will be using only has a 1-gigabit ethernet port. It is a Broadcom NIC, not an Intel one. Historically speaking, Intel drivers on FreeBSD (which pfSense is based on) has had better driver support. However, the bge driver-based Broadcom NICs are well-enough supported at this point to proceed. Still, we only have 1 NIC port which his challenging.
That does not mean we cannot use this platform for pfSense, however. Since pfSense natively supports VLANs, we can use the HP EliteDesk 705 G3 in conjunction with a managed switch to accomplish our goal. VLANs allow us to have multiple, isolated, networks inside of a single switch. More importantly, it allows us to bring multiple networks (WAN/LAN) to our pfSense router over a single cable. We are limited to 1-gigabit of overall throughput, though, and we will not be able to build a LAG for redundancy.
We need a managed switch to accomplish the goal. Since managed switches allow us to tag VLANs on an interface, this is a requirement. In order to accommodate this need, we worked with the STH community and got a managed 24-port Brocade ICX6450 up and running. A special thanks goes to community member fohdeesha for putting all of the information we needed in one place.
You do not need an enterprise-class switch to do this, however. Any small-business smart switch, like the Netgear GS110EMX we reviewed, will do.
A look at our Network topology
In our How to build a lab series, we took a look at my network. In order to begin the testing process for this pfSense deployment, we are going to install it on the test bench. Exactly how that will look can be best described in the below diagram:
pfSense will use VLAN 99 as the WAN, and VLAN 991 as the LAN. For those following along at home, instead of plugging your managed switch into another switch as we have here, all you would have to do is plug your ISP modem unto a port untagged on VLAN 99. In that configuration, no other devices on your network should be on that VLAN. All client-side access will take place over ports untagged on VLAN 991.
Setting Up pfSense
Setting pfSense up to do VLANs is actually very straight forward. if you need help getting started with the installation, please read Netgate’s installation guide. On your first boot after installation, pfSense will go through an initial setup wizard. The very first question it will ask you is Do VLANs need to be set up first?
Your answer should be Y. It will then ask you to select your interface. On our HP EliteDesk 705 G3 it was bge0, so we typed that in. It will then ask you to specify the VLAN tag, enter 99. It will ask again, enter 991. Then a third time, you should enter nothing.
Next enter the WAN Interface name, type bge0.99. Then it will ask for the LAN Interface name, type bge0.991. It will ask you if you want to proceed, press Y. It will then put you on the pfSense management screen. We set our LAN interface’s IP address by pressing 2 in the menu. Ours is set to 10.99.20.1/24. After following the prompts for changing that, it asked us if we wanted to enable DHCP. We said yes.
After that, we can go over to a PC connected to our network and log into the web interface. The default username is admin and the default password is pfsense. It will walk you through another setup wizard to do things like choosing your DNS servers, time zones, etc. Once you have gone through those prompts, you will land on the Dashboard page.
Initial Testing Results
pfSense has many plugins that you can install. To do so, you go to System on the top menu bar and then click Package Manager. We installed the iperf3 plugin to see if it could properly route at gigabit in its default configuration. The HP pfSense router will act as the client, and my SG-5100 firewall will act as the server.
With the results in, it can do gigabit without breaking much of a sweat. CPU utilization was hovering around 35%. As we found in the Project TMM review, the HP EliteDesk 705 G3 Mini’s AMD Pro CPUs are not particularly fast which means this is an excellent result.
What about traffic shaping and IDS/IPS? For traffic shaping, we are going to use fq_pie, which is derived from the Common Applications Kept Enhanced (CAKE) project. The CAKE project was founded to combat the phenomenon known as Buffer Bloat.
They define buffer bloat as “the undesirable latency that comes from a router or other network equipment buffering too much data”. This increase in latency comes into fruition as you utilize higher percentages of your internet connection. For that reason we setup Traffic Shaper Queues on our own network. Lawrence System’s has a walk-through guide on setting this up.
For IDS/IPS we use Suricata. Intrusion Detection/Prevention systems are a security feature in modern firewalls. They use deep-packet inspection to snoop on every packet coming into, and going out of, your network. Their mission is to find exploits and malware and alert you to their presence, and optionally, stop them in their tracks.
Setting up Suricata is very complex and requires its own article. If you are interested in this topic, I can refer you to Lawrence System’s YouTube page again to get you going. One thing worth noting is that the bce-based NIC that is running in the HP 705 G3 does not support inline Suricata blocking, and you have to use the legacy method. This creates some additional CPU overhead. Ironically, Suricata out-of-the-box was blocking iperf and I needed to adjust the ruleset to allow my testing.
With traffic shaping and Suricata turned on we are still getting gigabit speeds, though we are down a few megabits. The CPU utilization only went up a few percent.
While I did not run the HP EliteDesk 705 G3 through a battery of tests, I was able to run some basic testing to show its viability as a firewall. Something to additionally consider is VPN performance. Since the AMD A10-8770E CPU in my unit has AES-NI crypto acceleration, It should perform fairly well in that category.
While I am not drawing a like-for-like comparison between this unit and the likes of the Netgate SG-5100, it is worth looking at it as a benchmark. Since pfSense is a software firewall, CPU performance is very important. In that arena, the A10-8770E in the HP EliteDesk 705 G3 out-performs the Atom C3558 in our Linux kernel compile test.
CPU performance is not everything for this application, however. Power consumption is something worth considering. The HP EliteDesk 705 G3 pulled about 10 watts idle in our testing. These numbers are right on target, with the SG-5100 rated 7 watts idle. Of course, the EliteDesk can scale to much higher power numbers so the Netgate unit is more power-friendly.
On the flip side, being limited to only 1 interface has its drawbacks. Having to purchase a managed switch is an added cost. Having no ability to build a LAG is unfortunate. It can also be a show stopper not to have a dedicated interface for a high-performance device, or to have a dedicated interface for a DMZ. The Netgate SG-5100 has all of these features.
However, these HP units are selling for around the same price point the Netgate SG-1100 is selling at. With that said, if the added comfort of using the vendor’s own hardware to run their software is nice to have. If that weighs heavily on your decision making, the SG-1100 is also a great firewall. If you buy hardware from Netgate, you also have the option to purchase a TAC contract. You do not have that option with third-party hardware.
Are you guys running a spreadsheet with comps, features and stats?
Very nice. I’ve been thinking about using pfSense in this kind of “router-on-a-stick” setup with only one NIC.
Tozmo, we are going to do some comparisons as the hardware side of Project TinyMiniMicro progresses. One item we are trying to do a lot of is document some of the key specs/ options for each platform so people can find them. It turns out we often spend a lot of time compiling just those pages. We may test one configuration but there are a lot of options for each.
Stuart, this guide applies to other platforms as well, it was just using Project TinyMiniMicro since those machines are low-cost and high-quality.
Could we use a USB ethernet adapter as an interface for WAN?
My impression is a home lab is seldom space constrained the way production deployments in a data center are where you pay by the inch. Are these tiny machines really more frugal with power than a similarly provisioned used micro ATX tower, or are they just cheaper to purchase used?
Christos, that should be possible although generally there are lower quality NICs in external cards.
Eric, there are a lot of people constrained by space. A great example is that more than half of the world’s population lives in cities and that will likely grow to more than two thirds. When you want to have 3-5 nodes, this form factor has huge benefits.
Advocating a “router on a stick” design in a respected public forum is CRINGE-WORTHY journalism.
Any security professional with many years of practical experience will tell you that “router on a stick” designs are best left in a lab and used only for purposes of experimentation.
Using a “router on a stick” design in a production network with a VLAN-enabled switch exposes your network to security flaw called “VLAN hopping”. All it takes is a flaw in the switch firmware, or a misconfigured VLAN setup and your network goes from “a warm-fuzzy degree of security” to “no security at all”.
Just sharing my 25+ years of networking experience that included 10 years working with network security and around other security professionals.
/shaking my head/ at STH
The bge driver is stable enough, but not all Broadcom NICs support jumbo frames. Can be annoying for some use cases, such as ‘baby jumbo frames’ (MTU 1508) for PPPoE (RFC4638).
USB ethernet adapters are mostly (though I don’t know else) Realtek under the hood, so it should be avoided if possible.
Sleepy I thought the entire Project TMM was for labs? So you’re saying this is bad because it’s showing this as a lab router but saying its “best left in a lab”.
I don’t think anyone would use this if they had a real production network that they cared about VLAN hopping, right? You’re talking a few bucks more to get a dedicated NIC system.
Why not just use usb3 to gigabit ethernet adapter??? I’ve done this with an old zotac ad6 model before I built a mini itx AMD A-10 the a 4 port gb nic.
Sleepy, if your switch is vulnerable to vlan hopping then you have a problem regardless of whether you are using router on a stick.
And in your 25+ years of experience have you ever known a successful attack that actually used vlan hopping in the way you are describing?
It’s the classic example of a threat that you can point to as being sub optimal, and bound to make certain people moan like hell, but in practice is hard to exploit.
In any case, I would point out that from my experience trying to use pfsense on a couple of projects over the past year it isn’t really the leading solution it perhaps was. On one of the projects I ended up using openwrt, which routed faster on identical hardware and had a more flexible failover solution, and on another I’m using nethserver which has proven very easy to setup and use, giving me the solution I wanted in a fraction of the time required on pfsense.
Both benefit from using Linux which has better driver support than freebsd, and better routing performance.
I doubt I would try pfsense first on my next project to be honest.
I dont get it….
why dont you run pfsense directly on your proxmox env as virtual machine insteadt of using some old limited iron that sips a lot of ekstra energy.
Brian – I used to run pfSense virtualized, and for years. There were some great features such as being able to roll-back snapshots if something went wrong. We still have a few virtualized instances, however, having a physical appliance is just easier. Personally, I can say we did this and now have mostly migrated back to physical nodes.
I think we have somewhere between 35 to 50 instances running on hyperv at different sites / customers. cant imagine going back to standalone appliances on pfsense. it is running smooth and is very stable. And as you mentioned we really love the abillity to rollback if a someone messes everything up or an update fails. Can you tell a bit about, why you think its easier?
Thanks for a great article!
But why not do it with an under $100 budget? 🙂
Cheap TP-Link routers that support OpenWRT can be used as a smart switch. For example TP-Link WR1043ND can go as low as $20-$30 new, and it has a very stable Gigabit switch.
HP EliteDesk can be replaced with a second hand Dell Wyse using OpenWRT or pfSense. Those small network terminals are starting from $60-$70 and its CPU can easily handle 500Mbps connection with SQM or even QoS, while using only a few Watts of power. Just make sure it has a Gigabit LAN. The downside for those needing it is that most of those terminals do not support AES-NI, in case you need OpenVPN. But for home networks it can easily get replaced by Wireguard, that is less CPU intensive.
An alternative with those power hungry could be any Lenovo ThinkCentre or any other Tiny PC, that are starting from ~$50 for an old box. Some of those ThinkCentre PCs support an additional PCI-e low profile slots, so you can skip using a switch by adding another LAN card with multiple ports (4 ports), that cost like $25.
The idea is basically the same as yours, but even cheaper.
I’ve had limited success with the WAN to VLAN trick. Some managed switches insist on blasting LLDP an other advertisement frames on all interfaces (eg. Unifi).
ISPs like Xfinity bind their DHCP to the first MAC address they see, which doesn’t give your pfSense a chance to get an IP. I’ve had to run a temporary 50ft patch cable to have my router be the first to be bound, before switching to the VLAN.
Sail Internet has a strict “1 customer, 1 MAC address” policy and shuts down the link if they see any extra frames with a different MAC address.
It would be good to warn users of this caveat. Even better would be recommendations on equipment and configs to make the WAN VLAN into a true “virtual wire”?
you can run a another network card on this via its mini-pcie slot, what I did is buy a solderless (the one that you can screw in the wire) DB9 male serial port, align the pinouts that can match with cisco’s rj45 rollover serial cable to match with the rj45 pinouts. after that, I have a dual NIC setup on it. the only downside for this is there’s no fancy lights, but for me those are not necessary anyway and the activity ligth on the ONU (which I plug this thing in) is more than enough to see a network activity light indicator for the WAN.
Why isn’t this referenced in the guides/pfsense section of this site? Way to bury the lead guys!
I’m loving the tiny mini micro series and looking forward to seeing what you do with them all. You can actually put a normal server network card in one of these tiny computers. I’m currently running pfsense on a dell 3020m and a have a normal 4 port card that fits inside. You just need to get a cable from the WiFi card slot to an x4 pcie (and cut a whole in the back to access the ports).
For HP mini PC series, is there a Flex IO card that provides an additional LAN port?
E.g. will the HP 1-Port 1GbE Flex IO NIC (3TQ26AA) be compatible with Elitedesk mini PC ?
Is this kind of setup possible with a layer 2 switch. It’s managed but not “fully managed”. My understanding is that because it’s an L2 switch, I won’t be able to route between nodes on different VLANs. Is there a workaround for that?