11/29/2021 12:51pm - Incident Post-Mortem Analysis

On November 29th, 2021 at 12:51pm EST, The lain.la datacenter (ha.) suffered a total power outage after a brief moment of voltage fluctuations on the main grid. This had been the first time since I started lain.la that a real, extended power outage had occurred, and I got to do a generator cutover and learn many lessons along the way. The incident ended at 1:53pm EST when all systems were restored.

I bought an UptimeRobot Pro Account

So if you've ever visited my actual homepage (https://lain.la) there was always a big fat UptimeRobot icon at the bottom where you could call me a shitty sysadmin by staring at my uptime stats. UptimeRobot is a pretty cool service and I had been freeloading for the past year on their free version which, admittedly, is VERY good.

So I got pro recently. They had a sale. It was like $5 a month. No brainer. Here's what I got for going pro:

Future Lain.la Plans - From Simple (not) to Stupid

It is no secret that I spend a lot of time (and money) on Lain.la to make sure it runs better than just about any other collection of freebie services. While the services themselves are generally par for the course, its the infrastructure where I like to shine. This means continually reviewing and improving things as load scales up to account for possible risks to infrastructure failure and balancing those risks against their cost. I try to account for just about everything - storms, power outages, stray backhoes. You name it - I've probably thought of it.

Proprietary Software: Why?

I've grappled with the thought recently that my peers may look down on me due to some proprietary software running in Lain.la's stack. The purists among you may completely discredit my infrastructure because of non-free software. Usually I pay these people no mind, because their fanaticism isolates precisely those that they wish to convert, but I needed to justify to myself and others exactly why I didn't use entirely FOSS for this project and why I disappointed ol' RMS.

October Updates and Metrics

Hello again! It has been a while since I wrote a new article, only because most things have been stable. We've been through another maintenance cycle (which is really just patching and certificate rotation these days) and I've documented my procedures and setup processes more so than ever. Here's some highlights of the updates:

The Lain.la Service Catalog

This article is partially to remind myself what the heck I have made myself responsible for but also list out all the nice things I do. Please see the homepage for links to these services where applicable.

Class A Services:

(Note: Class A Services are ones I take extra good care of. These are expected to stay afloat with minimal or no downtime, and high standards of performance.)