Lain.la's next regular maintenance window will take place on November 13th, 2022 (*Occurring at whatever-time-I-wake-up-o-clock.). This article will detail what actually goes on in a maintenance window and serve to remind me what I actually have to do.

Recently, I was asked by a friend to solve a long standing issue in my network for their freebie server. Under a network where NAT is being done at least two network hops upstream, the source IP address will be lost in translation, making it impossible to determine the real IP address of an incoming client to a server inside my network. That server will only see the upstream edge node's IP, not the user, such as 10.13.0.1.

This should be short and sweet. Simple ISP outage.

At almost exactly midnight on August 3rd, extreme latency was detected on all VPN links, and my game of Factorio that was in progress shit the bed. After doing quick debugging, I determined that all external connectivity was gone, and the VPN tunnels went dead completely. Checking the core router, I found that there was no upstream gateway responding, and udhcpc was sending discover requests with no reply.

Well, here we are again. Indonesian porn breaking my infrastructure. We had a half hour outage of all HTTP and VPN based services on OPT1-4 due to overload from Pomf traffic. This was a little different, however. My script DIDN'T cause a cascade failure. No, actually there was just so much load spread out across all four endpoints that the whole thing collapsed. Let's go into that load profile.

First, an image.

Hey, I have a good excuse though. Multiple, actually!

Vacation!

I took a vacation in the middle of June! That's why not much got done. Did you notice though? Probably not - except for a poorly timed ISP outage, Lain.la kept humming along wonderfully while I was frolicking around London. I had to step in once or twice to handle abuse requests but otherwise, pain-free. Very happy that my infrastructure basically runs and heals itself in my absence. Makes the time investment minimal for daily operations.

Here's another updates and metrics article for your enjoyment.

Updates:

We've had quite a few improvements lately:

Quite the title. Last night we had a bit of a problem with the ol' Pomf clone. You see, something happened that I never expected to happen. Pomf hit its maximum outbound bandwidth threshold globally. This has never happened in the ~2 years I've been running all this.

Dumb article time because I get asked it a lot. No, it has nothing to do with Satan.

A long time ago (2013, almost a decade now... sheesh) I joined my first IRC channel on Freenode (now Libera) - #teamdoge. This was a mining pool, a basic MPOS (Massive Piece of Shit? No! Mining Portal, Open Source) installation with a dedicated little community where I mined my first dogecoins ever. Their web IRC gateway assigned a name of Teamdoge, an underscore, and four random numbers. The day I showed up, that RNG provided me my name. Teamdoge_7666.

Hello! It has been a while since I talked about networking, which is probably the most critical and advanced part of my infrastructure. Nobody's doing what I'm doing in this space, that's for sure!

To recap, these are all the iterations of Lain.la's networking/core infra that I've gone through: