A friend recently sent me a Synology RackStation that was destined for e-waste. Full of drives no smaller than what I feed my existing storage server, no less. A good friend indeed, amiright?

He said it had been upgraded to the 6GB “maximum” — 2GB “onboard” plus a 4GB DDR3 SO-DIMM. I don’t know much about Synology hardware but in the past I’d randomly acquired the knowledge that sometimes the “onboard” RAM is actually a SO-DIMM on the underside of the board.

Underside of Synology motherboard showing additional SO-DIMM socket

And sure enough, there it is. Seems an odd design choice given that this RackStation’s motherboard is so much larger than it needs to be… but I guess odd choices are the norm for companies that tie their software and services to seemingly over-priced custom-engineered hardware instead of just selling software and services on their actual value.

Synology System Information screenshot showing 16GB RAM recognized.

So my RackStation now has 16GB RAM. In theory this system should support 32GB RAM but 16GB DDR3 SO-DIMMs carry around a 10X premium over 8GB so I’m not about to find out.

Dashboard of the Synology Active Backup for Business

To me the killer Synology feature is Active Backup for Business, which is only available on certain models (+ / x64?). As a total slut for centrally-managed backups and bare-metal restores, I moved to urbackup after Microsoft abandoned the fantastic client PC backup system included with Windows Home Server & Server Essentials. Urbackup is about the only Open Source backup system that does Windows decently, is properly multi-platform and multi-arch, and offers change block tracking and Hyper-V host-based VM backups as commercialized add-ons for reasonable fees (or free via the community edition of their commercialized virtual appliance).

ABB is much better by most measures. It’s prettier. It’ll do agentless VMware and Hyper-V VM backups. It can backup “unsupported” platforms via rsync and SMB. Backup times are fast — none of my daily tasks run over 15 minutes — and with a household full of laptop users that’s critical to keeping them current. I’ve yet to try a bare-metal restore but individual files and whole VMs run about as fast as the storage/network can muster.

I see a few areas where ABB could do better:

  1. Backup task settings are individual to the device. There are Templates whose settings are applied at the initial creation of a device’s backup task, but after that the task’s settings are independent from the original Template. There’s no mechanism to perform changes in bulk. It is possible to create a new task for multiple devices at once, but that will create individual tasks for each. Backed up data is tied to a particular task and the interface warns that removing a task will remove all the data, so that’s not a path to faking bulk updates. 

  2. From the Portal, the presentation of BitLocker-encrypted volumes within Hyper-V VMs is concerning. BitLocker-encrypted volumes from “PC/Mac” and “Physical Server” backup tasks are visible and browsable through the Portal like any other volume, but from a Hyper-V VM backup the volumes do not show up in the Portal at all. I tested an Instant Restore to Synology’s Virtual Machine Manager — the volume was properly restored and, unexpectedly, VMM provided vTPM functionality so the VM operated normally after initially entering the recovery key.

    So this is a case of the Portal interface being misleading and not an actual problem. 

  3. ReFS volumes are not supported. ReFS is over a decade old and still struggles with 3rd-party support. Heck, it’s not clear that Microsoft really wants to support it as a general-purpose filesystem. Which is sad because we’ve got nearly 20 years of ZFS advocates shouting at us that copy-on-write, checksumming filesystems are the greatest thing since the hierarchical filesystem and if you’re not using one you don’t care at all about your data and probably kick your dog.

    I mostly use ReFS for Hyper-V datastores so this is an effective way to filter them out from backups of a Hyper-V host as a “Physical Server” without having to manually customize their backup tasks. 

  4. BTRFS volumes are not supported. Which is odd because it requires BTRFS for backup storage. Despite its protests, in my testing it did backup an LVM-based BTRFS system but does not restore LVM to a usable state. BTRFS within a Hyper-V VM was fine. 

  5. Linux Agent is x64-only. If you want to backup ARM/MIPS/RISC-V/32-bit Linux devices, you’ll be doing it old school via rsync or SMB. But at least it’ll be a centrally-managed pull instead of an unmanaged client-initiated push where you’ll need to come up with some other method to notice when your backup jobs have failed (you always have monitoring of your important cronjobs, right?) 

  6. Desktop “PC/Mac” and Windows/Linux “Physical Server” are handled slightly different. A Windows “Physical Server” backup can be restored to VMware/Hyper-V/VMM while a “PC/Mac” backup cannot. A Windows “PC/Mac” device can be changed to a “Physical Server” but not the reverse. And, sorry Linux desktop users, you can only be a “Physical Server.” There’s also a minor scheduling difference, see below. 

  7. Backup scheduling is rigid. Backup tasks are scheduled for specific times and days-of-the-week and will not be made up if missed or interrupted. For PC/Mac backups it is possible to have a backup task triggered when a user logs off, the screen locks, and/or at startup, but for laptop users those may not be sufficient to stay within desired backup intervals.

    With all of the backup systems I’ve previously used, I would define backup windows and target intervals and the system would figure out when to actually initiate backups. Missed or interrupted backups would be made up automatically at the next window or availability of the client, depending on the configuration. 

A problem for Future Me is that DSM 7.2.x will go out-of-maintenance in mid-2025 and it’s probable that 7.3 will not support this hardware. The current nearest equivalent is the RS2423+ at $1,999.99. That’s a big chunk of change to spend up front for backups over the 7-9 years of expected support. A RS822+, DS1522+, DS923+, DS723+, or even DS423+ might be suitable for Future Me’s primary use case of backing up other systems, I’ll need to see how much storage backups consume after soaking for a year… but it’s hard to get over my preference for software that doesn’t lock me into hardware.

Taylor Swift: It's me. Hi. I'm the problem, it's me.

Zero downtime VM migrations are still magical to me

It has to be about 20 years since I first experienced vMotion and the technology still feels like magic. A few clicks of the mouse and the things I had running on this computer over here are running on that computer over there and nobody is the wiser.

A few years back I switched my virtualization hosts at home from VMware to Hyper-V and the tech geeks in my social circles always question why. It’s because Hyper-V frees me from having to run vCenter to get the magic — live migration is built-in and (mostly) Just Works™

Yesterday I needed to replace storage on the host that runs a couple bits of critical home infra that is still virtualized — Pi-hole, nginx, and Home Assistant — and was grateful that I could temporarily migrate it all to my other Hyper-V host instead of causing a prolonged outage.

Got a Home Assistant Yellow

Hi, my name is [redacted] and I’m a gadget-holic.

Home Assistant Yellow

I can’t make it make sense to me. I browsed mini PCs on Amazon for about a minute to find a J3455 8GB / 128GB micro PC for $120 before $30 off coupon. That’s cheaper than just the basic Yellow board and case, which still needs an impossible-to-obtain CM4 module and storage. This one is PoE and sports a $29 NVMe SSD and $80 CM4 w/ 8GB RAM and Wi-Fi, for a grand total of $243 before taxes and shipping. That’s J5005/J5105 money.

But, hey, it does have a Real Time Clock chip w/ a battery. That’s something that nearly all the little ARM SBCs and most of the CM4 carrier boards are lacking, a pain point that nobody notices until they do and then they can’t stop thinking about it.

I bought it ’cause I figure the HA people deserve my money, and I’d rather spend it on a gizmo than their cloud service. I’ll migrate one of my HA instances to it, to give it a proper chance, but I suspect it won’t be long before I want to migrate back to a proper PC that does everything faster and better.

Ultimate ESP32 BLE gateway?

I’ve been on the hunt for the “best” BLE-capable ESP32 devices to use with BLErry and ESPresense. Problem is that the ESP32 isn’t nearly as popular as ESP8266 in the IoT world.

CloudFree’s Light Switch is quite good but I discovered it way too late in the process of changing out my light switches. I have a few m5stack Atom Lite in use and ordered some LILYGO T-Dongle-S3 to mess with but the more I think about it the less I like the idea of infrastructure hanging off USB ports and wall plugs.

What I really want is a cheap ESP32 in the form of a wall wart or smart plug.

Enter the SwitchBot Smart Plug Mini*. It’s cheap — under $25 for a 4-pack on Black Friday sales. It has the ESP32-C3. It’s not at all friendly to opening, however, presently it is possible to OTA flash to Tasmota. And it has power monitoring, which is an uncommon feature in a cheap Tasmota plug. There’s not an official esp32c3-bluetooth build yet but they’re available from other sources and it’s not that difficult to roll your own.

SwitchBot’s Smart Bulbs are also ESP32-C3 and convertible to Tasmota, but I’m trying to keep away from bulbs.

I’ve also been eyeing the GL.iNet GL-S10, a $25 ESP32 device with Ethernet and PoE, but I’m resisting until it’s available from a US-based seller. The thing has been available for over a year and GL.iNet has plenty of US distribution for other products so I don’t get what the hold up has been.

* 11/21/2022 Update: After getting my hands on the Switchbot I would be cautious about using it for productive loads as it will not hold the relay state during a reboot or firmware update. This appears to be a hardware design decision. In theory that’s not a big deal — your Tasmota device shouldn’t be rebooting itself and if it ain’t broke don’t update — but it’s one of those footguns that’s likely to be forgotten about until it takes out a toe.

UniFi Mobile Router

No 5G. No LTE Cat 6. And locked into AT&T’s revenue sharing scheme so customers can’t onboard these to their existing plans and Ubiquiti gets a cut. Lame.

Mikrotik has 4G LTE Cat 6 devices with US carrier approval that are much cheaper than Ubiquiti’s devices, but no mobile variant. I will continue using the Netgear LB1121 for backup Internet until it dies.

Smart Switching

I’ve now installed 54 smart switches, dimmers, and relays of 8 varieties, plus mounted a dozen remotes. Six switch locations remain but they’re all unimportant.

If I had to do it all over again, I would make the CloudFree Light Switch and Amaker WKC-002 Zigbee my default switches. Those and the Lutrons have metal tabs and I’ve found they’re much easier to get properly aligned and flush with the wall plate in a multi-gang box. All the rest with plastic tabs have been challenging, the Martin Jerry Zigbee model most of all because the tabs are narrower. I also like the CloudFree Light Switch for having what I consider to be a premium feel. The Amaker was a nice late addition, it feels cheap but best conveys that you’ve successfully pressed it and it functions as a Zigbee router. The Lutrons feel like cheap junk but there are very few cloud-free no-neutral dimmer options.

An added perk with the CloudFree Light Switch is that it runs on an ESP32-C3 (RISC-V) chip which has Bluetooth / BLE instead of one of the more common ESP8266 variants that only do Wi-Fi. Took me maybe 30 minutes to roll my own Tasmota image with BLErry and get it feeding a BLE temperature / humidity sensor into Home Assistant. Shouldn’t be challenging to get it doing the same on ESPHome but I’m unlikely to try that myself.

Continue reading

De-Smart Bulbing

Between buying a second home and taking over the ADU at our primary residence to be my home office, the need to make 20-some new lights smart had me reconsidering our home automation strategy. Our main home has somewhere in the neighborhood of 80 Zigbee bulbs and I’m just sooooo over them. People can’t be trained not to flip switches, no amount of remotes on the walls and blocks on the switches will stop them. Plus my non-Hue bulbs never get firmware updates, so they’re older than manufacturers figuring out that maybe they shouldn’t all turn on after a power outage.

Also thanks to tariffs / COVID / inflation, the bulbs I used to routinely buy for $6-7 are now more like $10.

Continue reading

“Just Works”

Somethings I find I’ve unintentionally made something magic happen. This weekend I brought an unprovisioned UniFi Talk phone to our cabin in the mountains, expecting to need to perform some VPN trickery with a VPS to get a local Talk install properly receiving calls behind CGNAT. Imagine my surprise when I plugged it in and my existing Talk install back home discovered the new phone.

It adopted just fine, no problems making and receiving calls through the VPN tunnel.

The “magic” was that I run multicast-relay on all my personal networks and have it configured to also relay to my VPN network. All the wannabe Network Engineers’ heads are exploding at the thought, but I’m sure I had reasons when I decided to do that and, well, My Networks, My Choice.

Then I fired up my new Home Assistant install and quickly realized that auto-discovery across a VPN tunnel is not always a good thing 🤣 When I have more round tuits I will perhaps make things a bit more granular.