DNS Changes in UniFi 5.11.50

It’s not DNS.
There’s no way it’s DNS.

It was DNS.

Took a nasty hit yesterday from a change made in UniFi 5.11.50:

If you have any ‘service dns forwarding options’ configuration defined in config.gateway.json, it will overwrite the provisioning of statically defined name servers, leaving you with no DNS. Either remove the ‘service dns forwarding options’ portion of config.gateway.json, or add additional ‘options’ lines defining name servers, such as ‘server=1.1.1.1’, ‘server=8.8.8.8’, etc.

I’ve long used some config.gateway.json tweaks to add conditional forwarders for my Active Directory zones because Ubiquiti still hasn’t seen fit to put that functionality in the GUI.

So after upgrading to 5.11.50, the first full re-provision cycle of my USG Pro ended up with no catchall forwarder. The interesting bit is how this manifested itself. My client devices are pointed directly at a local PiHole so that its metrics don’t show everything as coming from the router, so they all kept humming.

What broke was failover load-balancing. By default, the USG’s WAN health check pings a target… by DNS name. So the USG thought both my WANs had failed, and due to the vagaries of how the routing tables are managed, that ends up sending all traffic out WAN2. Not a desirable condition when using WAN2 costs me money (LTE) and is significantly slower than WAN1.

This led to a couple hours of cursing at how badly failover load-balancing is messed up on Ubiquiti’s routers, but in the end, as it is so often, the problem was DNS.

More interesting is that this problem was brought about by some behind-the-scenes changes in how UniFi is managing DNS for the gateway device. Prior to this version of UniFi, the USG defaults to putting the WAN DNS value in /etc/resolv.conf. This is not ideal for two reasons:

  1. No DNS caching takes place for things running on the router itself. If you have IPS or load-balancing enabled this can result in an excessive number of DNS queries originating from the router, and creates a ton of noise if you have DNS metrics through PiHole, OpenDNS, etc.
  2. DNS servers are always queried one-by-one in the order they appear. This may or may not be desirable, but when the first DNS server is unavailable it will result in slow DNS responses.

I’ve worked around this on my USGs by configuring the WAN DNS servers manually and making the first entry 127.0.0.1. So the router will query the local dnsmasq first, and dnsmasq itself will ignore that entry when selecting an upstream DNS to forward to. dnsmasq is also “sticky” by default in choosing an upstream DNS server — it won’t keep sending queries to the first resolv.conf entry if it’s non-responsive.

What UniFi 5.11.50 has changed is that now they’re automatically placing just a 127.0.0.1 entry in resolv.conf and have moved the rest of the DNS configuration into /etc/dnsmasq.conf.

Another change they’ve made is setting all-servers. This causes dnsmasq to query all upstream DNS servers simultaneously and whichever answers first wins.

This is great, in that the USG’s DNS configuration is now made sensible by default. And it’s not great, in that if you have config.gateway.json changes to service​ dns forwarding options they will override what UniFi is doing to dnsmasq.conf and you will need to make adjustments to provide the DNS forwarders yourself.

And beware if you’re configuring multiple DNS servers with differing views of DNS! This has always been a bad idea but all-servers can make it much worse to troubleshoot.

Left the Discord, Permanently

I checked out of the Ubiquiti Discord for months around the time of my move, and when I came back everything had changed. More Channels. More Rules. More Mods holding everyone else to higher standards than themselves. And… the same old cliquish behind-the-scenes behaviors.

Basically a shitty sub-Reddit in chat form.

I tried to focus on the good and ignore the parts I didn’t like, but… ultimately I realized that I wasn’t getting anything out of my participation in the community beyond frustration.

So I said Adios.

USG-XG-8 is Dead

Ubiquiti has finally admitted that the USG-XG-8 is dead. The big problem with the USG-XG-8, aside from UBNT taking a solid year to product decent firmware, is that UniFi just doesn’t play in that league. It’s forgivable that the USG and USG Pro sacrifice features for simplicity because they’re cheap and UniFi is awesome. The USG-XG-8 is not cheap and that makes it harder to ignore all the things it couldn’t do. Especially when the EdgeRouter version can do those things at a much lower price.

Good riddance!

My Favorite Black Friday Deal

I always get myself the best “Christmas” presents. I know me so well. This year, it’s a couple of Arcade1up cabinets from Walmart for $249/ea.

img_0423

In my early 20s I got into collecting arcade cabinets for a minute. A 29″ Neo-Geo MVS 4-slot and mint 4-player Gauntlet were the highlights of my collection, but of course, what I really wanted was a Pac-Man cabinet. I was just never willing to pay the price for one that was in presentable condition.

Eventually I had to give up the collection. I’ve always wanted to get back into it, but… they’re just so big, and heavy, and difficult to move up and down stairs without several helpers.

Spotting the Pac-Man cabinet at Walmart literally made my Christmas. Even tho it was only Black Friday.

These Arcade1up cabinets are just 4′ tall and a mere 65lbs. Easy to shuffle around and I can man-handle them up and down the stairs all by myself. Assembly takes about 40 minutes with just a screwdriver. All the bags of parts are labelled so there’s no guesswork as to which type of screw gets used where and it comes with a bag of spares.

Obviously it’s not as solid as a 300lb cabinet made of 3/4-inch birch or MDF, but the construction is good enough for home use. I’ve no concerns that they’re going to fall apart.

I’ll be keeping the Pac-Man cabinet as-is for now, but I’ve already ordered the parts to convert the Street Fighter cab to a RetroPie MAME setup — basically it just needs an LCD controller board and a USB encoder for the controls.

And I suspect I might pick up another one or two…

Ring and Retry

I have a Ring Doorbell Pro on my front door which has always been problematic. At first I could get it to join the WiFi but then it would error — turns out it was trying to use an outside DNS server and I had blocked clients from using any DNS but mine.

When I replaced my temporary AmpliFi setup with UniFi, I couldn’t get it to find my SSID at all. I literally held an AP directly in front of it and it would find several neighbors WiFi but not mine.

img_3400

I read somewhere that sometimes a Ring will get confused seeing several APs broadcasting the same SSID, so I decided to give it its own AP and SSID on 2.4GHz-only. I put it in the attic slightly offset from being directly above the door. This has mostly worked ok, except that it takes a long time to re-connect after the AP reboots from firmware updates.

Today I had cause to look at my AP retry rates…

Screenshot 2018-11-07 at 12.23.20 PM

And the Ring’s AP retry rate is just ridiculously bad. I crawled into the attic and shoved the AP all the way into the soffit. Gained 6dBm but the retry rate didn’t budge. Changed from Channel 1 to Channel 11, no difference.

Then I had the thought that, since initially installing this stuff, I’ve put a great deal of effort into tuning the power levels and minRSSI values to get devices to use the right AP instead of clinging to a poor signal. Let’s try turning off an AP in the attic I don’t really need anymore, bring the one I’ve been using for the Ring back into broadcasting my normal SSID on 2.4GHz and 5GHz, and trying joining the Ring to it.

And it joined right up! On 5GHz. The signal is decent and the retry rates have dropped to a more reasonable level. Huzzah!

Blasting WiFi across the street

I have a lot of front yard to maintain.

img_0160

UMA-D_Front_Angle

It would be nice to have good WiFi signal while mowing all this lawn. There’s an AP in the attic above the front door but the signal doesn’t reach all that far, maybe 30-40′ out. I needed something with a bit more oomf and the UAP-AC-M + UMA-D antenna combination sounded like the perfect solution.

If you haven’t heard, the UMA-D is a tiny miracle antenna: dual-band, 15dBi, 45-degrees on 5GHz and 90-degrees on 2.4GHz, for $99. It transforms the otherwise unimpressive UAP-AC-M into a directional WiFi blaster that will send its signal hundreds of feet downrange in open terrain.

As an initial test, I placed the combo in the bonus room knee wall space:

img_0157

Blasting through my roof I was getting about 180Mb/s of download speeds to my iPhone XS… from across the street! That’s 140-ish feet away.

Of course, that wasn’t good enough for me, so I found a pre-existing hole to run an Ethernet cable to and mounted it outside the garage.

img_0214

The improvement is incredible.

img_0269

That’s from my phone. 140 feet away.

If you need to blast a WiFi signal far away outdoors, the UAP-AC-M + UMA-D are a powerful and affordable solution.

60GHz Point-to-Multipoint Backhaul

This past weekend I finally had everything in place to deploy my Mikrotik 60GHz gear to backhaul the WiFi being installed in my pool house and detached apartment. There are a few reasons for choosing the 60GHZ equipment over using wireless uplinks within UniFi or running AirMax gear:

  1. For wireless uplinks I’d still need to mount an AP on the outside of the house. Brick exterior terribly degrades 5GHz signal.
  2. Mikrotik advertises gigabit, full-duplex. The headline numbers for AirMax AC gear are substantially slower and half-duplex.
  3. The 60GHz band frees me from concerns about interference from neighboring WiFi. Or my own.

I already had a Wireless Wire kit I’d intended to use for a PtP link at my old home, so I just needed to add a WAP 60G AP unit to enable PtMP. And figure out where to mount everything.

img_0268

The previous owners helpfully left me a couple holes where they’d mounted cameras. But climbing up the maximum reach of my ladder while drilling a mount above my head wasn’t something I really wanted to do.

Fortunately I found another set of holes coming off the living room. I wasn’t sure I could safely climb back down from that attic space, so first I embarked on a project to add another 2×4 step to the studs.

For mounts I used Ubiquiti’s UB-AM. On the house end, the Ethernet cable goes back to my main PoE switch in the bonus room closet. At the remote ends I’m using the Mikrotik PoE injectors with the data side connected directly to the data end of Ubiquiti injectors that power the WiFi APs. I figured it wasn’t worth installing switches in each location just to run a single AP, but if I install more devices later I may add them.

RouterOS is a bloody eyesore, but Mikrotik thankfully provides a quick-start interface for getting the units connected to each other and it was relatively painless.

The moment of truth was turning on the bandwidth test server on the AP side and getting both CPE units to bi-directional tests concurrently:

Screenshot 2018-11-06 at 9.16.06 AM

That is up to 1.9Gb/s of aggregate bi-directional throughput! Amazing. Individually I’m seeing about 1.3Gb/s, which is quite a bit less than the advertised 1Gb/s full-duplex rate, but 2-3X what I’d expect from AirMax AC gear in this scenario.

Our Homestead

Before I start posting about all of my home networking projects, I should probably describe the home and property. This is it:

Satellite view of my property showing the main house, pool, and detached garage / apartment.

The lot is 1.5 acres, roughly 180×400 if it were perfectly rectangular, with the front of the house about 120′ from the road. The house itself is the standard 40×30 box on a crawl space, with another 25×30 of garage / utility room and bonus room above. An addition off the garage provides a larger living room with a high vaulted ceiling. There’s attic access in the main part of the house, knee wall access on either side of the bonus room, and from the back side I can reach the living room’s attic space. There’s also some attic above the bonus room but the a/c ducts leave no room to get in there.

There’s a detached garage that was converted to a 2-bedroom apartment and came with tenants who pay half my mortgage. It also has attic access.

There’s a pool house that is basically a glorified shed. There’s an open area in the middle with small rooms to either side. One had been a proper bathroom but at some point in the past vandals ripped out the copper pipes.

So that’s what I’m working with. I have plans to bring Ethernet and in-wall access points to several rooms, blast WiFi across as much of the outdoors as I can reasonably manage, use 60GHz PtMP gear as wireless backhaul links for all three structures, give my tenants their own access point in the apartment, and much more.

My next post will be about deploying the PtMP gear.

UniFi Protect moves away from self-installs

Much online rage has been spilled this weekend over UniFi Protect not being made available for self-installation. If you’ve not been paying attention, UniFi Protect is a new NVR platform from Ubiquiti. Presently Ubiquiti says that UniFi Video will continue to be developed and supported… but nobody expects this to continue for very long. Protect is the new hotness, and since Ubiquiti doesn’t charge for their software, it is unimaginable that they will continue putting resources towards two separate products that do the same thing.

Today, the only way to get Protect is on the UCK-G2-PLUS — which has just launched for $199 with an 8-core ARM SoC w/ 3GB RAM and an easily upgradeable 1TB 2.5″ hard drive. Support for the UAS-XG — an attractive but otherwise bog standard 1U server with a $1,999 MSRP — is coming soon.

People are miffed for a variety of reasons. The G2+ offers very limited storage options. The UAS-XG is a fair value compared to buying an equivalent server from an Enterprise vendor, but it’s incredibly expensive relative to DIY or other thrifty options. There’s presently no middle ground.

And with the UAS-XG being a standard Intel server running Ubuntu 16.04 LTS, there’s no technical reason that Protect can’t be offered for self-installs. Using Docker — as they’ve done with UNMS, and had alluded to providing for Protect during the Beta cycle — would greatly reduce the support challenges of providing self-installable software for Linux.


I’m not sure how I feel about this. I have 5 UVC-G3-Flex cameras that I have been planning to deploy on the G2+ w/ Protect, with the expectation that if I was happy with the platform over time I would replace 6 Ring Spotlights, 4 Amazon CloudCams, and probably add a couple more.

That’s all within the advertised capabilities of the G2+, but with its limitation of a single 2.5″ drive it won’t be able to meet my continuous recording retention targets. I happen to have a 2nd G2+ already so maybe this isn’t a total deal-breaker for me, if it turns out that I really love Protect, but right now I’m questioning if Ubiquiti’s camera platforms are worth the lock-in and premium pricing.

I already have an NVR server that is substantially more powerful than the UAS-XG, an unlimited camera license for Sighthound, and a bunch of Reolink cameras from my old house which are comparable in quality to the G3 Flex.

I’m Baaaack!

Personal stuff has limited my participation in the Ubiquiti community for much of this year, but I’m finally getting my head back above water. The biggest news is that I bought a new-to-me house on 1.5 acres and have been consolidating households with my girlfriend and her children.

New home means lots of upcoming technology projects:

  • Pulling Ethernet to every room in the main house.
  • 60GHz Mikrotik PtMP link between the main house, pool house and detached garage apartment.
  • Building out a fully isolated network for the apartment dwellers.
  • Radius-assigned VLANs for wireless devices.
  • UniFi Protect and 5x UVC-G3-Flex deployment using a UCK-G2-PLUS.
  • Deciding on more permanent placement of APs, which are presently strewn haphazardly across the attic. Or possibly replacing most of them with UAP-IW-AC.
  • Getting complete coverage of the front and back yards using a combination of UAP-AC-M, UMA-D, and UAP-AC-M-Pro — there’s roughly 100′ of depth in the front yard and 200′ beyond the pool house.
  • Attempting to order 2Gb/s fiber service from Comcast.

I’m also back to dogfooding the USG with plans to bring a USG-XG-8 into service as soon as a couple of show-stopping bugs are resolved.