Writing
Incidents
My failures. What broke, why it broke, and how I recovered.
-
I Deleted /etc/passwd on My NAS
Two weeks into Linux, I pointed tee at the password file because I thought it worked like cat. The recovery from a shell with no name, and what it taught me.
-
Proving the Firewall Innocent
My firewall has a history of breaking legitimate traffic, making it the default suspect for every outage. Seven steps that clear or convict it in a minute.
-
Detection Is Easy, Recovery Is Hard
My firewall's watchdog detected every outage and kept failing to reboot the box. Every layer I added broke something new, including one kernel panic.
-
When Both Mirror Drives Fail the Same Way
A ZFS mirror lost both NVMe members to the same firmware bug, one night apart, and I fired the second shot myself. What redundancy actually assumes.
-
Three Bugs Wearing One Trench Coat
Four months of DNS outages that were never one problem. A failing port, a 534k entry multiplier, and a recovery command that planted an impostor every run.
-
My Backups Ran Green Every Night and Backed Up Nothing
A replication task can succeed every night for months while protecting zero bytes. The audit that found it, and the ZFS semantics that let it happen.