Ubiquiti EdgeRouter ER-12P 12-port Router Review

11
ER-12P Front
ER-12P Front

The Ubiquiti EdgeRouter 12P (ER-12P) is a big brother of the Ubiquiti ER-6P device we reviewed earlier. This unit is based on the same platform but offers an additional onboard Ethernet switch chip. We wanted to take a look at what Ubiquiti is doing with this device and whether or not we are getting better value as a result.

Ubiquiti EdgeRouter ER-12P Hardware Overview

The device measures 268.1 x 136.5 x 31.1 mm (10.55 x 5.37 x 1.22″), 700 g (1.54 lb), and has a maximum power consumption of 20W (not including PoE). Ubiquiti went with an all-metal, clean, and minimalistic design, consistent with the rest of the ER- line. The ER-12P is a fanless unit. To help with cooling, most of the surfaces of this device are mesh. On the front of the device, we find ten (10) RJ-45 Gigabit Ethernet ports with support for 24V 2-pair/ 24V 4-pair passive PoE output, two (2) SFP port, serial console, reset button, a USB 3.0 port is reserved for future use, and status LEDs for device and ports.

ER-12P Front
Ubiquiti EdgeRouter ER-12P Front

Please note that “Passive” PoE is not part of 802.3af or 802.3at standards. While the ER-12P will work with most Ubiquiti PoE devices, we encourage users to read the ER-12P QSG to get a better idea of the power budget, requirements, and limitations for PoE devices connected to ER-12P.

At the back, one can see a 24V DC input and ground point.

ER-12P back
Ubiquiti EdgeRouter ER-12P back

At the bottom of the device, we find integrated keyholes for wall mounting and (4) rubber feet.

The ER-12P can be rack-mounted with an optional kit. And at $15 the price is reasonable, but it would have been nice if Ubiquiti included this.

Just like the EdgeRouter 6P, the 12P version is based on a Marvell (Cavium) CN7130 SoC containing a 1Ghz MIPS64 quad-core CPU. It has a variety of connectivity options including a packet processing engine, QSGMII, RXAUI, SATA3.0, USB 3.0, and PCIe Gen 2, although not all of these interfaces are utilized by ER-12P. This Marvell platform has started on the carrier-grade market and as time passed, made its way to consumer and SOHO markets.

CN71xx
Marvell Octeon III CN71xx Block Diagram

The Marvell CN7130 SoC in this device is coupled with 1Gb of DDR3 RAM and 4GB eMMC storage.

Unlike the ER-6P, the ER-12P version has a switch chip connected to the first eight Ethernet ports (LAN). Ubiquiti does not disclose technical details in the datasheet, but according to reports from the community, it is a Qualcomm Atheros QCA8511 or similar. The advantages of such a design include low latency and line-rate throughput between clients on LAN ports configured for L2/L2+ switching. On the other hand, combined throughput for clients connected to LAN port configured for L3 operation will be limited by the interface between switch chip and SoC. While schematics and/or block diagrams are not publicly available, we would be surprised if Ubiquiti does not utilize one of the SerDes/QSGMII blocks (up to 5Gbit/s). It is also worth mentioning that hardware offloading for IPv4 forwarding is not available (or does not work) for communication between LAN ports. At this point it’s not clear if this is a hardware limitation or an issue with firmware, there is nothing in the release notes that specifically calls out the issue with LAN ports, but it does mention “[Offloading] – VLAN traffic is not being offloaded on ER-12,” which may be related.

EdgeRouter ER-12P Management

EdgeOS is the default firmware for EdgeRouter X, which we briefly covered in the EdgeRouter X piece. At the time of this review, the latest release of 2.x version of EdgeOS (2.0.9-hotfix.1) has improved IPv4 forwarding performance with and without HW offloading. Taking into account that the 1.x version has not been updated for almost a year and may miss critical security fixes, we believe it’s time to switch to 2.x. The UI for 2.x firmware remains almost the same with some cosmetic changes, like the addition of a button to check for firmware updates in the top right corner, or a button to perform a factory reset in the system menu.

EdgeMAX ER 12P
EdgeMAX ER 12P

The CLI also remains the same; we can get access to a packet processor engine to get information about the status of various modules, per-packet and per-flow statistics, and the ability to manipulate parameters. We can also turn on/off these modules and adjust the size of the lookup tables.

Next, let us look at the performance.

Performance evaluation

Test Bench Setup

Our testing bench is based on a Cisco T-Rex project which in turn is based on the DPDK framework which we are going to cover in future articles and consists of:

Host Dell Precision 7920
CPU (2) x Gold 6134 CPUs 16 cores/32 threads x 3.19 GHz
RAM 128GB: 8*16GB DDR4-2133P
Host OS VMware ESXi 7.0
Guest Debian 10.
* STF mode: 6 vCPUs 32GB RAM (low latency)
* ASTF mode: 12 vCPUs 64GB RAM (low latency, 2 vNUMA nodes)
T-Rex version v2.86
Network Intel X710 in PCI Passthrough mode

You can read our Dell Precision T7920 Dual Intel Xeon Workstation Review for more on the platform itself.

T-Rex modes of operation

For our tests, we primarily use stateful (STF) and advanced stateful (ASTF) modes of operation. You can get a full overview of each mode on the T-Rex home page. In a nutshell, STF scales up to Mpps per core, and the number of flows can easily go up to 100k and above. In this mode, T-Rex emulates stateful traffic, and while it allows one to verify some devices under test (DUTs) with NAT enabled, it does not implement a TCP/IP stack, and as a result, will not handle any TCP/IP errors properly and will not work with DUTs that terminate traffic. For the same reason, we allow a rather large percent of packets to be dropped (up to 1%) by DUT, understanding that in a real-world scenario, the TCP/IP stack on DUT or client/server-side may be able to handle the error gracefully (I.E transparent for end-user). Primary metrics collected with this mode are %of packets lost and latency. We will continue to use this mode to evaluate the basic (raw) routing capacity of DUT, but for NAT/VPN/DPI we will switch to ASTF.

ASTF mode has TCP/IP implementation based on the BSD stack. While it is more demanding in terms of resources and CPU power, in addition to raw performance, it allows us to evaluate the quality of services by inspecting the number of packets that were re-transmuted, out-of-order, duplicated, etc., allowing us to review a wider range of DUTs. Last but not least ASTF comes with a Python API, which helps a lot with automation.

Using Case Driven Benchmarks

While synthetic benchmarks are good for marketing, and when used properly give a high-level overview of device potential, they do not make it easier to evaluate the performance of the device for a particular use or compare performance across devices due to different boundary conditions. Such boundary conditions may result in a significant error in the final numbers.

T-Rex gives us the freedom to define any workflow we like, or even create one based on real traffic captured from a production system. In order to see how the device under test performs in a more realistic scenario, we will use the SFR profile. This profile includes a combination of traffic templates such are:

  • http_get / http_post / https
  • mail-related traffic flows
  • SIP
  • DNS
  • and etc.

Below we can find a graphical representation of SFR profile:

SFR Profile
SFR Profile

EdgeOS v2.x.x

There were several reasons why some users stayed on the v1 version of firmware for Cavium based on ER devices in the past. To name a few, early versions of v2 firmware had a number of stability issues with GUI and performance degradation compared to v1. Early versions of v2 firmware did include the following statement in the release notes

Known issues:
Performance - Throughput degradation by 5-10% when comparing with v1.10.9 
  firmware with older kernel

Things have changed since our ER-6P review; in the latest version (2.0.9/2.0.9-hotfix.1) of firmware, the “Throughput degradation” warning disappeared and firmware appears to be more stable. What is more important, the v1 version had no updates for over a year, and it may miss important security fixes. For these reasons, we encourage users who are still on the v1 version to re-evaluate the latest firmware.

ER-12P Routing performance (STF)

While we are not going to test the ER-12P with the v1 version of firmware for the reasons stated above, we want to give our readers some reference points to set the right expectations about potential performance degradation when switching to the v2 version of the firmware. The ER-12P is based on the same platform as the ER-6P, so we will use it as a reference point to spot-check routing performance with a multi-WAN scenario. To recap, our STF routing performance test ER-6P with firmware version v1.10.11 had virtually 0 packets lost at speeds up to 400Mbps and less than 0.003% up to an upper limit of our tests at 2.3Gbps.ER-12p 4ports RTE Packet Drop

ER-12p 4ports RTE Packet DropThe ER-12P with firmware version 2.0.9-hotfix.1 in the same scenario has virtually 0 packets lost up 300Mbps after 300Mbps percent of packet lost slowly raises reaching 0.8% at 2.3Gbps, which is still acceptable.

ER-12P Routing performance (ASTF)

Note: We would like to highlight that percent of packets dropped should not be compared between STF and ASTF modes.

The test is executed with the firewall disabled and hardware offloading enabled for IPv4 forwarding and VLAN. We use two (2) pairs of ports connected directly to our bench. Each pair gets its own pool of clients(255) and servers (64k). The client-side of a pair is connected, to LAN while the server-side is connected to one of the WAN ports.

In this test, we will be looking for maximum combined throughput (based on SFR profile) the DUT can handle without losing data, which for ASTF we define as a combined packet drop below 0.1%. The ER-12P has no problem handling this scenario with 0 packets lost up to ~1.2Gbps and reaching a maximum of 0.03% at 2.3Gbps

Another point we are curious about is a statement in the release notes claiming to fix the following issue:

 Routing - Added Ethernet driver patch from Cavium that fixes packet reordering 
            with 4.x kernel

For reference, a previous version of the firmware had the following statement under known issues in the release notes:

Offloading - On Cavium-based routers (ER, ER-Pro, ER-Lite, ER-PoE, ER-4, 
  ER-6P, ER-12, ER-Infinity) small percentage of packets are randomly 
  reordered. This issue was fixed in v1.10.0 firmware but it reappeared 
  since v2.0.0 because of new ethernet driver.

So let us look how many out-of-order (OOO) packets we get:

ER 12P RTE OOO Client
ER 12P RTE OOO Client

Looking at the results, I personally do not believe the issue is resolved, or that it is a small number. With that said, it is important to mention that out-of-order packets by themselves are not a problem, they just make the network stack (in case of TCP) or Application level (in case of UDP) work a little bit harder, but users with more or less recent devices are unlikely to notice anything.

ER-12P NAT performance (ASTF)

The setup described in the previous test is modified to add masquerading rules based on outgoing interface (WAN) for each pair of ports. With NAT enabled, we hit our limit of 0.1% at ~2Gbps (combined throughput).

ER 12P 4P NAT ASTF SFR Pkt_drop
ER 12P 4P NAT ASTF SFR Pkt_drop

And checking the % of OOO packets, we see that it’s still quite high.

ER 12P 4P NAT OOO Client
ER 12P 4P NAT OOO Client

Ubiquiti ER-12P Power Consumption

We saw an average idle power consumption for the device around  ~10w. During the test execution, power consumption was ~13w.

Final Words

Just like the ER-6P, we think the Ubiquiti EdgeRouter 12p (ER-12P) can be an excellent choice for most home and SOHO users who are not locked to a particular ecosystem. The solution is built on a hardware platform that is still in use by big names for entry-level enterprise devices. Here, the ER-12P delivers excellent results and we expect it to be able to reliably handle multi-WAN connections up to 2Gbps based on our testing.

PoE support is limited to “Passive mode” and the power budget is quite small. Last year when we reviewed the ER-6P, we praised Ubiquiti for this feature, as it would allow users to connect some low-power Access Points, cameras, and phones directly. However, as of today, it would not be sufficient to power most recent APs even within the Ubiquiti ecosystem, so the value of “Passive POE” is diminishing very fast. New WiFi 6E APs are using even more power so this is a trend we see continuing.

Special consideration should be made towards which version of the firmware to use, especially taking into account known VLAN-related issues in the latest firmware, which could hurt the ER-12P primary use case. We hope it will be resolved with the next update. We also recommend that anyone who is still using v1.10.x firmware on devices with some public services to re-evaluate v2.0.x firmware due to a number of security fixes added through the year.

Support for license-free cloud or on-premises management through UNMS could potentially turn into significant TCO savings as well. At $299 list price ($249 street price) this makes this device extremely competitive.

11 COMMENTS

  1. What is Ubiquiti’s reputation at this point for fixing firmware issues in reasonable time?

    There seemed to be an awful lot of mentions of the software being in a somewhat unnerving state at present; I’d just want to know if they are the sort to release early but fix responsibly; or if buying one of these could mean being in beta right up to the point of obsolescence.

  2. The only way I’ve had any reliability with their products at this point is sticking to old firmware. The edgerouter has been better in my experience, but the DHCP unifi issue us an utter joke. They say it’s “fixed” but it’s not and hasn’t been for a looking time.

  3. I ended up opting for the 12, not the 12P. The passive only PoE is not that useful anymore but with a 12 and an X you can theoretically link them in a loop (Port 9->0, 5->0), and set them up with VRRP using the loop connections for the heartbeat, and backup power for each other… For various reasons that didn’t work out for me (mostly the 12 and the X are now separated by several miles). The only problem I’ve had with the 12 (other than configuring it to work the way I want, which is quite complex) is that it didn’t seem to want to do a LAG group within a VLAN aware switch. Some posts on the forums suggested that worked, but it did very nasty things to my network when I tried. I really wanted that to work because I have a lot of cross VLAN traffic that goes through the LAN side of the router.

  4. FYI, if the tests were run with hardware offload enabled, I would strongly urge you to rerun the tests with hardware offload disabled.

    The following thread on the latest firmware release includes comments about potential memory leaks likely being caused by hardware offloading.

    https://community.ui.com/releases/EdgeMAX-EdgeRouter-Firmware-v2-0-9-hotfix-1-security-update-2-0-9-hotfix-1/e44ba49b-b46a-47c9-a6c9-09b7483af954

    We use EdgeRouters in our WISP and have found that disabling hardware offload significantly improves stability.

  5. Software development has been a real disgrace. Functions do not work properly ad advertised because of bad software. Support is really bad. Lots of unsolved bugs.
    Firmware v.1.x is stable, but old, and not patched for over a year. Also some accelerations have not been repaired for YEARS.
    Please read user forums before considering purchasing ANY Ubiquiti product. They really have a BIG issue to fix in the organisation, but it is NOT happening…

  6. As an MSP with hundreds of UBNT switches and AP’s out there: Look elsewhere. UI has devolved into a hot mess in the last 18 months. They keep trying for “enterprise class” but no IT professional would use them in anything but edge level devices, where they are totally fine.

  7. I’ll never buy anything from Ubiquiti again unless the entire C-level suite is replaced, along with anyone else that was involved in the hack cover-up attempt. Getting hacked is bad enough, but lying about it and trying to cover it up is inexcusable.

  8. To fix the OOO packets, there is a tunable to make only one core process received packets that should fix the issue with very little performance drop, at least last I checked.

    With that being said, and as others have said, the software is a mess. At one time I had a few of these in my lab and since have turned them all off due to missing features (vrf), broken features (tons of stuff) and their inability to fix major problems over years. If I was looking at this class of hardware for my business, I’d seriously look at FRR on Ubuntu or opnsense instead.

  9. STH, Thanks for the Comprehensive review.
    From the replies/comments, I’m sorry to hear that UI has continued to slide and has not kept up.
    So maybe UI can refocus and instead of trying to be everything to everybody, can … “Do one thing and Do it very well,”
    Their reputation can be built back maybe, “One step at a time”. We customers/users are sometimes hard to win back.

  10. I think they did … they’re focusing on UniFI (it’s much more lucrative compared to business of selling ER- devices). But to be fair to ER- family, I would like to mention that it’s an open design based on Debian, which offers flexibility other player are not offering. The core of the system (hardware, and OS) is quite stable, and security issues are fixed on regular basis. Advanced feature (IDS, DPI, integration with cloud etc) may have issues, so one should carefully evaluate if a device would work for a specific usecase.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.