March 2023 Updates and Metrics

It's been a while since I've done one of these! Hello again, dear metrics nerds. There's been a lot since November's last metrics update that I'll run through, and then all the near-term future projects I'll go over:

Updates:

Historical updates are as follows:

Hosting Stuff From Home (Safely)

It occurs to me that I've never written an article on the most important foundational architecture decision I took, which is the method in which I use to publish things from my home safely and reliably.

Let's dive in. I'll try to make this one a simple article so it's more accessible to the masses. Self hosting is a good thing for anyone to do, after all, as it fights centralization!

Note: This article was last updated on August 4th, 2023.

The Redundant LAN Project

Another project complete! This one stretched one of my weakest skills - networking.

The Problem:

So, every server previously had two network connections - one dedicated to the management network, and one dedicated to the host network. Both of these connections were 10Gbit connections, via SFP+ fiber transceivers. However, if a transceiver, cable, NIC or even the entire switch crapped out, that host was toast, because both connections needed to stay up at all times. This scenario happened twice in recent memory:

Caching LVM for Pomf

After the slice range improvement, IOWait (The amount of time the CPU is deadlocked waiting for storage calls to finish) across the four edge nodes uses for Pomf traffic went up quite a bit due to the need to address a larger amount of fragmented files across the slow storage cache disk. Before it wasn't really a problem, but now when there's over 100,000 slices to manage, it changes the IOPS requirements quite a bit.

November Updates and Metrics

I'm still in one of the most aggressive infrastructure upgrade periods I've ever had for Lain.la. I spent 13 hours today just working on upgrades, servers, patching, etc, not to mention the largest maintenance window I've ever had to put up with this month. Let's go over some stuff.

October Updates and Metrics

Oh boy, how things have changed. This has been one of the most aggressive few months I've had for upgrades and changes.

Updates:
  • Stor1.lain.local is now fully operational, with a usable storage pool of 115TB.
  • Stor2.lain.local is now fully operational, with a usable storage pool of 90TB.