The Real MGP

UniFi Management Gateway Pro, that is. Who comes up with these names?

UMG-frontUMG-back

Freshly announced, ahead of being available for Early Access purchase, we have essentially a UDM Pro… minus the switch, minus the HDD bay, half the RAM… and no local controllers! Adopt it to your Cloud Key, cloud-provider, or self-hosted UniFi install.

It also has a built-in UniFi Smart Power Plug. I can understand why — reboot your Cable / DSL “modem” if the Internet goes down — but it’s just such an odd thing to integrate into what is otherwise a plain router.

I’m happy to see this, tho disappointed that it’s not in more of an ER-4 form-factor with an optional rack mount kit. Hopefully this is a sign that a smaller-but-equally-capable desktop unit will come eventually.

I’m also hoping that this is a sign that in the future it will be possible to disable all local controllers on the UDM / UDM Pro.

Update: They’ve tweaked the original post, now it’s “UniFi Managed Gateway Pro” and the UMG Pro will be the first of the “UniFi Managed Gateway Product Line.”

Update #2: Renamed again, UXG-Pro. If Ubiquiti is anything, it’s consistently inconsistent. 

UniFi Internet Security DNS Filtering

Today I was prompted to figure out what exactly the DNS Filters settings in UniFi Internet Security are doing.

unifi-dns-filtering

The main thing that happens is that the DNS queries for the associated VLAN are forwarded to the cleanbrowsing.org public resolver for the chosen filtering category. This is accomplished by creating a new dnsfilter network interface, binding another instance of dnsmasq to it, and using NAT to redirect DNS queries from the associated VLAN.

The implementation is slick from a technical perspective but I have a couple major problems with it.

Firstly, cleanbrowsing.org supports DNS-over-TLS, DNS-over-HTTPS, and DNSCrypt, but Ubiquiti has chosen to forward the queries unencrypted.

Secondly, if you’re using a local DNS server, say Active Directory or PiHole, those NAT rules will prevent the local DNS from functioning when queries are coming from or destined to a filtered VLAN. This might be fixable with some config.gateway.json or scripting hackery on the USG, I have not dug into that, but UDM users are hosed since there is no json replacement nor a way to automatically run a script.

Thirdly, Ubiquiti could have licensed the cleanbrowsing.org RPZs, converted them to dnsmasq format for local blacklisting ala PiHole, and eliminated the need to forward DNS to a 3rd-party and interfere with cross-VLAN DNS.

I cannot recommend using this DNS filtering because it will cause issues if your DNS implementation isn’t “UniFi Default” and the filtering options are minimal. My personal preference for home use is a PiHole in conjunction with OpenDNS Home.

For an office, Cisco Umbrella is the way to go. Do not underestimate the value of having a commercial relationship with a security provider. I <3 PiHole, but if you try to figure out why it blocks, say, zombo.com, and how you’d get it removed, you will immediately recognize that “free” security lists aren’t worth more than you’re paying for it.

Update: Also want to point out that the dnsfilter interfaces use IPs in the 203.0.113.0/24 subnet. That is TESTNET-3 in RFC 5737. It’s unlikely this will ever cause a problem… but over the long haul, every misuse of an IPv4 subnet ends up causing a problem. The loopback range 127.0.0.0/8 or link-local 169.254.0.0/16 would have been more appropriate choices. 

Isolating UniFi Devices

I’ve been lax in firealling my VLANs at home, but with the recent controversy over UniFi devices phoning home without consent, this has taken on renewed importance. I’m also taking on a new tenant in my detached apartment and would like to keep all their stuff segregated from mine.

Fortunately it’s all pretty easy.

For my network, I’m keeping my Cloud Key and Pi-Hole on the main LAN. I have additional Corporate VLANs created for Management, Cameras, and the Apartment.

In the Firewall rules, for WAN_OUT I’ve created two rules to Drop all traffic from the Management and Cameras networks. They cannot reach the Internet at all.

To allow and deny particular cross-VLAN traffic, the first step is to create a group of all the Private IP address ranges:

  • 10.0.0.0/8
  • 192.168.0.0/16
  • 172.16.0.0/12
  • 100.64.0.0/10 (this is CGNAT, might be a bad idea if your ISP uses them)

Back in the Rules area, LAN_IN needs a series of rules:

  • Allow (Management | Cameras) networks access to the CloudKey.
  • Allow (Management | Cameras | Apartment) networks access to the Pi-Hole.
  • Drop (Management | Cameras | Apartment) networks access to the Private IP ranges group.

The Allow rules must be before the Deny rules for each Network.

The gotcha with denying devices access to the Internet is that they cannot directly obtain firmware updates. For UniFi Networking products this can be worked-around by having the UniFi Controller cache the firmware prior to upgrading — see Settings -> Maintenance -> Firmware.

I’m not sure whether Protect can distribute firmware updates to the Cameras. Guess I’ll find out the next time there’s an update available. Once my UniFi Protect NVR arrives I will place that in the Cameras VLAN so that traffic doesn’t have to cross the router and figure out the WAN_OUT / LAN_IN firewall rules needed to keep it happy.

UniFi Protect NVR

A bigger NVR for UniFi Protect finally hit the Early Access store early this morning… and was sold out by afternoon. Four LFF/SFF bays, support for RAID1 and RAID5, 1x GbE and 1x 10GbE SFP+ ports, and a port for the UniFi SmartPower Redundant Power System (also in EA). No details on the CPU or RAM but I’d presume it’s the same platform as the Cloud Key Gen2 Plus.

At $299 in EA it seems like a decent enough value.

I’ve ordered one, as I’m just finally starting to put Protect to real use in my backyard and the 5TB max drive size on the CKG2+ will soon be a problem. Hopefully I’ll find time to get the rest of my G3 Flex cameras up before the New Year.

unifi-protect-screenshot

UDM Pro thoughts

UDM ProI finally broke down and added the UDM Pro to my Ubiquiti router lab. Here are the specs as provided in the Early Access store:

  • 8-Port gigabit switch with 10G SFP+ port
  • Dual WAN ports for redundancy and load balancing: 10G SFP+ and 1G RJ-45
  • Bluetooth connectivity for easy setup via UniFi app
  • Scalable UniFi Network Controller with advanced management capabilities
  • UniFi Protect video surveillance NVR with 3.5″ (or 2.5″) HDD support
  • Enterprise-class IPS/IDS and DPI capabilities
  • 1 x 1.3″ Touchscreen display for quick status information
  • Powered by fast 1.7 GHz quad-core processor

Not mentioned is that the UDM / UDM Pro use a new OS that is not derived from EdgeOS / VyOS / Vyatta. This is why I’d held off on buying one, when the UDM was first made available to Early Access back in March it was FAR from having feature-parity with the EdgeOS-based USGs. The present state of UbiOS is much closer to production-ready (by UniFi standards).

From the perspective of the USG Pro, this is a pretty serious upgrade in performance with a very minor bump in the expected MSRP. 10Gb/s inter-VLAN routing and WAN. Supposedly can hit 5Gb/s of IPS/IDS throughput — I’m underwhelmed by Ubiquiti slapping a pretty face over open source Suricata with publicly-available lists, but I seem to be a minority.

It’s also quiet. With it sitting on my desk and the LCD showing the fans at 50%, I struggle to hear it. The USG Pro, USG-XG-8, and their EdgeRouter siblings are not quiet-space-friendly.

I feel like they’ve missed the boat in a couple of areas with regards to Protect.

One, those LAN ports should have POE. As a router, those ports are of marginal value — where a $379 router is justified, it’s going to be attached to a larger switch. But as an NVR with a larger storage capacity than the Cloudkey Gen2 Plus, having 8 PoE ports would cover many deployment scenarios and would be quite valuable (12-16 ports would be better).

Two, it should have more drive bays. 2x LFF would have been nice. More LAN ports w/ PoE and 4x SFF bays could be better.

Three, it needs a USB host port for offloading footage directly to removable storage. As things stand now, pulling footage out of Protect is a pile of suck, but presumably it will get better, and being able to push footage directly to removable storage would be a great feature.

Ubiquiti has teased a larger, 4x LFF device. It’s not clear if that will be a NAS that Protect can use, or if it will run Protect directly, and they haven’t shown the back yet so we don’t know if it has PoE switch ports to act as a more traditional NVR appliance.

DNS Changes in UniFi 5.11.50

It’s not DNS.
There’s no way it’s DNS.

It was DNS.

Took a nasty hit yesterday from a change made in UniFi 5.11.50:

If you have any ‘service dns forwarding options’ configuration defined in config.gateway.json, it will overwrite the provisioning of statically defined name servers, leaving you with no DNS. Either remove the ‘service dns forwarding options’ portion of config.gateway.json, or add additional ‘options’ lines defining name servers, such as ‘server=1.1.1.1’, ‘server=8.8.8.8’, etc.

I’ve long used some config.gateway.json tweaks to add conditional forwarders for my Active Directory zones because Ubiquiti still hasn’t seen fit to put that functionality in the GUI.

So after upgrading to 5.11.50, the first full re-provision cycle of my USG Pro ended up with no catchall forwarder. The interesting bit is how this manifested itself. My client devices are pointed directly at a local PiHole so that its metrics don’t show everything as coming from the router, so they all kept humming.

What broke was failover load-balancing. By default, the USG’s WAN health check pings a target… by DNS name. So the USG thought both my WANs had failed, and due to the vagaries of how the routing tables are managed, that ends up sending all traffic out WAN2. Not a desirable condition when using WAN2 costs me money (LTE) and is significantly slower than WAN1.

This led to a couple hours of cursing at how badly failover load-balancing is messed up on Ubiquiti’s routers, but in the end, as it is so often, the problem was DNS.

More interesting is that this problem was brought about by some behind-the-scenes changes in how UniFi is managing DNS for the gateway device. Prior to this version of UniFi, the USG defaults to putting the WAN DNS value in /etc/resolv.conf. This is not ideal for two reasons:

  1. No DNS caching takes place for things running on the router itself. If you have IPS or load-balancing enabled this can result in an excessive number of DNS queries originating from the router, and creates a ton of noise if you have DNS metrics through PiHole, OpenDNS, etc.
  2. DNS servers are always queried one-by-one in the order they appear. This may or may not be desirable, but when the first DNS server is unavailable it will result in slow DNS responses.

I’ve worked around this on my USGs by configuring the WAN DNS servers manually and making the first entry 127.0.0.1. So the router will query the local dnsmasq first, and dnsmasq itself will ignore that entry when selecting an upstream DNS to forward to. dnsmasq is also “sticky” by default in choosing an upstream DNS server — it won’t keep sending queries to the first resolv.conf entry if it’s non-responsive.

What UniFi 5.11.50 has changed is that now they’re automatically placing just a 127.0.0.1 entry in resolv.conf and have moved the rest of the DNS configuration into /etc/dnsmasq.conf.

Another change they’ve made is setting all-servers. This causes dnsmasq to query all upstream DNS servers simultaneously and whichever answers first wins.

This is great, in that the USG’s DNS configuration is now made sensible by default. And it’s not great, in that if you have config.gateway.json changes to service​ dns forwarding options they will override what UniFi is doing to dnsmasq.conf and you will need to make adjustments to provide the DNS forwarders yourself.

And beware if you’re configuring multiple DNS servers with differing views of DNS! This has always been a bad idea but all-servers can make it much worse to troubleshoot.

Left the Discord, Permanently

I checked out of the Ubiquiti Discord for months around the time of my move, and when I came back everything had changed. More Channels. More Rules. More Mods holding everyone else to higher standards than themselves. And… the same old cliquish behind-the-scenes behaviors.

Basically a shitty sub-Reddit in chat form.

I tried to focus on the good and ignore the parts I didn’t like, but… ultimately I realized that I wasn’t getting anything out of my participation in the community beyond frustration.

So I said Adios.

USG-XG-8 is Dead

Ubiquiti has finally admitted that the USG-XG-8 is dead. The big problem with the USG-XG-8, aside from UBNT taking a solid year to product decent firmware, is that UniFi just doesn’t play in that league. It’s forgivable that the USG and USG Pro sacrifice features for simplicity because they’re cheap and UniFi is awesome. The USG-XG-8 is not cheap and that makes it harder to ignore all the things it couldn’t do. Especially when the EdgeRouter version can do those things at a much lower price.

Good riddance!

My Favorite Black Friday Deal

I always get myself the best “Christmas” presents. I know me so well. This year, it’s a couple of Arcade1up cabinets from Walmart for $249/ea.

img_0423

In my early 20s I got into collecting arcade cabinets for a minute. A 29″ Neo-Geo MVS 4-slot and mint 4-player Gauntlet were the highlights of my collection, but of course, what I really wanted was a Pac-Man cabinet. I was just never willing to pay the price for one that was in presentable condition.

Eventually I had to give up the collection. I’ve always wanted to get back into it, but… they’re just so big, and heavy, and difficult to move up and down stairs without several helpers.

Spotting the Pac-Man cabinet at Walmart literally made my Christmas. Even tho it was only Black Friday.

These Arcade1up cabinets are just 4′ tall and a mere 65lbs. Easy to shuffle around and I can man-handle them up and down the stairs all by myself. Assembly takes about 40 minutes with just a screwdriver. All the bags of parts are labelled so there’s no guesswork as to which type of screw gets used where and it comes with a bag of spares.

Obviously it’s not as solid as a 300lb cabinet made of 3/4-inch birch or MDF, but the construction is good enough for home use. I’ve no concerns that they’re going to fall apart.

I’ll be keeping the Pac-Man cabinet as-is for now, but I’ve already ordered the parts to convert the Street Fighter cab to a RetroPie MAME setup — basically it just needs an LCD controller board and a USB encoder for the controls.

And I suspect I might pick up another one or two…

Ring and Retry

I have a Ring Doorbell Pro on my front door which has always been problematic. At first I could get it to join the WiFi but then it would error — turns out it was trying to use an outside DNS server and I had blocked clients from using any DNS but mine.

When I replaced my temporary AmpliFi setup with UniFi, I couldn’t get it to find my SSID at all. I literally held an AP directly in front of it and it would find several neighbors WiFi but not mine.

img_3400

I read somewhere that sometimes a Ring will get confused seeing several APs broadcasting the same SSID, so I decided to give it its own AP and SSID on 2.4GHz-only. I put it in the attic slightly offset from being directly above the door. This has mostly worked ok, except that it takes a long time to re-connect after the AP reboots from firmware updates.

Today I had cause to look at my AP retry rates…

Screenshot 2018-11-07 at 12.23.20 PM

And the Ring’s AP retry rate is just ridiculously bad. I crawled into the attic and shoved the AP all the way into the soffit. Gained 6dBm but the retry rate didn’t budge. Changed from Channel 1 to Channel 11, no difference.

Then I had the thought that, since initially installing this stuff, I’ve put a great deal of effort into tuning the power levels and minRSSI values to get devices to use the right AP instead of clinging to a poor signal. Let’s try turning off an AP in the attic I don’t really need anymore, bring the one I’ve been using for the Ring back into broadcasting my normal SSID on 2.4GHz and 5GHz, and trying joining the Ring to it.

And it joined right up! On 5GHz. The signal is decent and the retry rates have dropped to a more reasonable level. Huzzah!