Well, here we are again. Indonesian porn breaking my infrastructure. We had a half hour outage of all HTTP and VPN based services on OPT1-4 due to overload from Pomf traffic. This was a little different, however. My script DIDN'T cause a cascade failure. No, actually there was just so much load spread out across all four endpoints that the whole thing collapsed. Let's go into that load profile.

First, an image.

Hey, I have a good excuse though. Multiple, actually!

Vacation!

I took a vacation in the middle of June! That's why not much got done. Did you notice though? Probably not - except for a poorly timed ISP outage, Lain.la kept humming along wonderfully while I was frolicking around London. I had to step in once or twice to handle abuse requests but otherwise, pain-free. Very happy that my infrastructure basically runs and heals itself in my absence. Makes the time investment minimal for daily operations.

Here's another updates and metrics article for your enjoyment.

Updates:

We've had quite a few improvements lately:

Quite the title. Last night we had a bit of a problem with the ol' Pomf clone. You see, something happened that I never expected to happen. Pomf hit its maximum outbound bandwidth threshold globally. This has never happened in the ~2 years I've been running all this.

Dumb article time because I get asked it a lot. No, it has nothing to do with Satan.

A long time ago (2013, almost a decade now... sheesh) I joined my first IRC channel on Freenode (now Libera) - #teamdoge. This was a mining pool, a basic MPOS (Massive Piece of Shit? No! Mining Portal, Open Source) installation with a dedicated little community where I mined my first dogecoins ever. Their web IRC gateway assigned a name of Teamdoge, an underscore, and four random numbers. The day I showed up, that RNG provided me my name. Teamdoge_7666.

Hello! It has been a while since I talked about networking, which is probably the most critical and advanced part of my infrastructure. Nobody's doing what I'm doing in this space, that's for sure!

To recap, these are all the iterations of Lain.la's networking/core infra that I've gone through:

At approximately 2pm on April 10th, 2022, high latency was detected across all endpoint nodes:

There is evidence that this event may have began occurring as early as 12:06am on April 10th, but the full effects weren't felt until 2pm.

This will be a distinctly different kind of article from my usual writings, but it's a very important topic. I apologize if this one's a bit too stuck up.

Hello! I should really do this more often. We'll break this down into two sections: