Ethernet and console cables finally arrived, so it was a busy and sweaty morning of cable routing for the lab. Still need to hook up power, move my Home Production stack over w/ the US-16-XG, and figure out how I’m going to lay out the 10GbE stuff and a few other miscellaneous items.
Was adjusting some firewall rules and verifying them from the outside, and discovered that an unexpected port was being allowed, but only on the secondary WAN connection.
This particular setup uses dual WANs in a failover configuration, but has a ton of DNAT rules to allow inbound traffic on the secondary WAN because port-forward can only apply to one wan-interface. Here’s some relevant CLI:
set firewall group port-group web port 80
set firewall group port-group web port 443
set firewall group port-group web port 8080
set service nat rule 100 description wan2-web
set service nat rule 100 destination group address-group ADDRv4_eth7
set service nat rule 100 destination group port-group web
set service nat rule 100 inbound-interface eth7
set service nat rule 100 inside-address address 10.10.10.10
set service nat rule 100 inside-address port 8080
What I was seeing was that an inbound request to 443 was being allowed from anywhere, despite having firewall rules restricting access to certain IP ranges. Upon further examination I discovered that the request was being sent to 8080 on the inside.
And I immediately suspected my error: DNAT rules are processed before Firewall rules. Turning up logging on the FW and DNAT rules quickly confirmed that was the case.
Whoops.
My other mistake here is that I probably shouldn’t be using inside-address port in my DNAT rules. The guide I followed to originally configure these DNAT rules suggested that they were necessary, but that is not the case and translating port numbers is definitely not the behavior I was looking for.
There’s a whole lot of cabling to be done, and I’m still waiting on a switch, a router, and a serial terminal server… but the lab is coming to life!
Have you ever wondered why some Ubiquiti products use a C5 cord instead of the more typical C14?
That’s why.
UPS dropped off some rack shelves and an ES-48-Lite today, which means the only thing holding me up from starting to get this lab racked up and online is time and rack screws.
A kind soul on the Ubiquiti Discord offered me a good deal on an ERLite-3, pair of ERPro-8, and a ES-16-XG so the total amount of Ubiquiti gear available for labbing is going to be fairly impressive!
On hand I’ve got:
Waiting to be delivered:
The end goal here is to be able to run mock deployments with a pair of every class of EdgeRouter and build out complex internetworking without ever having to physically touch the boxes.
It has been confirmed to me that the ER-X / ER-X-SFP have only one 1 Gb/s link between the SoC and switch. Since every packet that enters the SoC through that link will have to exit the same path, 1 Gb/s is the maximum aggregate throughput.
I’ve not been able to garner any interest in why bi-directional testing take a substantial performance hit. I may try some earlier firmware releases in the future but for now I’m moving on from this subject.
Here’s the scenario: Dual WANs configured for failover load-balancing. Firewall / DNAT rules in place allowing either interface to be used for incoming connections, and Dynamic DNS is configured on eth6 for wan1.domain.com and eth7 for wan2.domain.com.
Problem: Want to have active-wan.domain.com resolve to whichever WAN is active. Can’t use web-check on the eth6 / eth7 interfaces because the load-balancing policies apply to traffic originating from the router so wan1 and wan2 would always be set the active interface’s IP.
Solution: At first I thought it would be necessary to create a transition-script for the load-balancing policy to update active-wan outside of ddclient during a transition, but I realized that I was over thinking the problem.
What I ended up doing was creating an additional DDNS entry on a separate interface for domain.com which uses a web-check.
set service dns dynamic interface eth5 web dyndns set service dns dynamic interface eth5 service custom-lb host-name active-wan.domain.com set service dns dynamic interface eth5 service custom-lb login user set service dns dynamic interface eth5 service custom-lb options zone=domain.com set service dns dynamic interface eth5 service custom-lb password pass set service dns dynamic interface eth5 service custom-lb protocol cloudflare set service dns dynamic interface eth5 service custom-lb server www.cloudflare.com
This exploits the load-balancing of the router’s traffic to discover the correct IP for active-wan. During a failover transition, ddclient will automatically detect that the IP has changed and update active-wan — no transition-script is necessary.
Note: Do not use web-check for weighted load-balancing. It will constantly flap between WAN IPs.
Every day on the forums, Reddits, and chatrooms, I see people struggling to set up an EdgeRouter for the first time. The trend has really exploded since the $49 ER-X hit the scene, and generally they’re having a bad time at it. They ran one of the setup wizards and got the basic Internets working, but now they need to make that shiny new EdgeRouter do the things their old router did and are completely lost.
It has a GUI, it must work just like my old ASUS / Linksys / Dewalt!
In some ways the GUI is great. You can bootstrap an EdgeRouter from pretty much anything with an Ethernet port and web browser, and there’s even a little button to bring up the CLI.
But the GUI was literally an afterthought. It’s not meant to, and never will, provide everything that an average home user needs in an easily digestible manner. Consumer routers don’t expose their users to Masquerade and Destination NAT rules or ask for RADIUS servers or countless other things an EdgeRouter has in the GUI.
And these poor folks search the Internets for help. Find some old guide that’s thoroughly obsolete and expects them to be familiar with the CLI or hand-editing config.boot with vi, and they can’t make it work, and they get frustrated, and they call the product crap because it wasn’t meant for the likes of them.
Fact is, they probably should have stuck with their ASUS. Or, gasp, maybe bought a USG.
Now, if you want to learn Networking Engineering as a skill or trade, an EdgeRouter is a fine piece of equipment to start with. But don’t just rip-and-replace your functioning home router, screwing up the whole family’s Netflix in the process.
Point your router’s DMZ mode at your shiny new EdgeRouter.
Put your own PC or home server behind the EdgeRouter.
Take some time to learn and understand how to do all the things you need a router to do.
When you’ve figured it all out, then you can replace your home router. Then you can think about installing one at your parents’ or girlfriend’s house. But don’t make anyone else suffer through your steep learning curve. Don’t make yourself suffer!
See Preliminary Observations of the ER-X for the story leading up to now. I retrieved the ER-X-SFP but after giving it some more thought I concluded that the SFP port shouldn’t provide better throughput because it has to be hanging off the switch.
But I decided to test away anyways. Here’s a baseline iperf run of my test setup:
About what I’d expect from the hardware I’m using: 908 Mb/s uni-directional, 1,465 Mb/s aggregate bi-directional.
Here is the ER-X, with eth0 WAN and switch0 / eth1LAN:
The bi-directional result of 765 Mb/s is one of its better runs, high 600s to low 700s was its general range.
Now, here’s the ER-X-SFP in the same config:
It takes a hit, tho it’s results were consistently in the high 700s — a tiny bit better than the ER-X.
Now let’s try it again using an RJ45 SFP in eth5 as WAN:
929Mb/s bi-directional! And consistent! So it performs better… and I’ve no idea why… but clearly the platform is limited to 1Gb/s aggregate throughput.
As a sanity check I ran the same test against an ERLite-3 and was able to get > 1,700 Mb/s using multiple threads. Using multiple threads against the ER-X did not affect the results.
Both ER-X and ER-X-SFP on v1.9.1.1, configured using the Basic Setup Wizard for single LAN, with set system offload hwnat enable and port forwards for 5001, 5201, and 5202 TCP & UDP (iperf / iperf3).
I’m going to work up a thorough review soon, but preliminary testing confirms what others have seen: Bi-directional performance of the ER-X is sub-1Gb/s aggregate and a fair bit lower than uni-directional performance. I’m seeing over 900 Mb/s uni and low 700 Mb/s for bi-directional NAT’d traffic on v1.9.1.1.
I’ll do more testing with routed non-NAT traffic next week.
I’m surprised more attention hasn’t been brought to this given how often the ER-X is promoted as a cheap router for Google Fiber and other Gigabit FTTH offerings. The reality of the ER-X is that it’s more like a 350/350 router than Gigabit.
In trying to understand why this is, I came across a blog entry speculating that one of the 1Gb/s interfaces from the SoC is reserved for the SFP on the ER-X-SFP. I’ll be swapping one of these ER-X’s for the ER-X-SFP at my girlfriend’s home to test if there’s a performance improvement using eth0 and eth5 on the ER-X-SFP vs eth0 and eth1 on the ER-X.
Another item of note: Pass-through PoE does not require a 24v PSU! The included 12v PSU is only 0.5A and not suited to powering anything else — it will, but expect stability problems under load, or perhaps a fire — but any 9v-26v PSU that supplies more watts will work. An upgraded 12v PSU will generally be a few bucks cheaper than 24v.