I'm still in one of the most aggressive infrastructure upgrade periods I've ever had for Lain.la. I spent 13 hours today just working on upgrades, servers, patching, etc, not to mention the largest maintenance window I've ever had to put up with this month. Let's go over some stuff.

Preface: This article is entirely my opinion, based on my direct experiences with EC-Council courseware, training, and examinations. I currently hold an EC-Council certification. This might change if they ever read this and manage to round up enough outsourced Indians to figure out who I am.

Oh boy, how things have changed. This has been one of the most aggressive few months I've had for upgrades and changes.

Updates:

• Stor1.lain.local is now fully operational, with a usable storage pool of 115TB.
• Stor2.lain.local is now fully operational, with a usable storage pool of 90TB.

Two incidents in a month. Ouch. And always when I'm sleeping, too.

So, at about 4:00am, the two main NYC lain.la nodes on BuyVM had their 1TB block storage mounts effectively jammed. No reads, no writes, no nothing. This wreaked all sorts of havoc - CPU deadlocked, Nginx deadlocked, nothing was going in or out of these two nodes. The other two nodes in Miami were fine. I woke up at about 10am, saw the problem, yanked the nodes, and popped open a ticket to BuyVM.

At approximately 7:30am on October 13th, 2022, Pomf suffered an upstream (uncached GETs and all POSTs) failure due to a loss of storage connectivity on the hypervisor (esxi3.lain.local) it was running on. The issue was traced back to a transient NIC failure, a Mellanox ConnectX-2 10GbE card, that had timed out in responding to OS commands. This led to the software iSCSI stack crashing irrecoverably. This condition was rectified at 11:30am after a hard reboot of the host, after approximately 45 minutes of troubleshooting and VM evacuations from it.

I jumped the gun on the storage server project. Decided to just go for it. And guess what? It's done! Pomf is on the new server right now!

Hello again! Time for updates and metrics.

Hello. I'm 7666. I single-handedly run this massive collection of services known as Lain.la. I started all this a little over three years ago (August 31st, 2020) as a productive outlet for myself, to make systems and services I'd be proud of that were also useful to me and my friends, while keeping privacy and security paramount.

Recently, I was asked by a friend to solve a long standing issue in my network for their freebie server. Under a network where NAT is being done at least two network hops upstream, the source IP address will be lost in translation, making it impossible to determine the real IP address of an incoming client to a server inside my network. That server will only see the upstream edge node's IP, not the user, such as 10.13.0.1.

This should be short and sweet. Simple ISP outage.

At almost exactly midnight on August 3rd, extreme latency was detected on all VPN links, and my game of Factorio that was in progress shit the bed. After doing quick debugging, I determined that all external connectivity was gone, and the VPN tunnels went dead completely. Checking the core router, I found that there was no upstream gateway responding, and udhcpc was sending discover requests with no reply.