~
A malicious actor, a nuisance masquerading as a public service, and a braindead web hosting company walk into a strange bar.
Stop me if you've heard this one before.
The malicious actor walks up to the bar and decks the Bartender. The Bartender, shocked, doesn't know how to respond. The nuisance decides that, in the best interest of everyone, it would be a good idea to pull out his cellphone and dial 911 a total of 27 times and provide information that there's a fight with the Bartender each time.
The cops show up faster than lightning, but it's not one or two cars, it's half the police department. They think a massive brawl has taken place because of the excessive 911 calls. They determine there was indeed a fight. The Bartender was indeed involved. There were 27 reports of fights. So they close the bar permanently; It must be a fight club! The Bartender protests, but that's the final decision on the matter. A fight is a fight they say. And 27 of them? Unthinkable. But, the Bartender protests more. He insists that the police don't even know who started the fight! This particular quandary startles the boys in blue. Not used to having to think this hard, they look at each other and mutter to themselves "You know, every call mentioned a Bartender..."
The cops turn their heads in complete unison to face the former proprietor of a lowly bar and book him for running a fight club.
~
Did you like the analogy? If not, too bad. Read more anyway.
A Malicious Actor
Some jackass thought it would be a good idea to upload a crap ton of binary files to my pomf clone which were apparently malicious. I haven't confirmed this myself, but it's probably safe to say they shouldn't be there. These files were crafted and uploaded in a way that evaded all of my security measures, which is really unfortunate. But not to worry! That hole is now fixed. Thank you, dear hacker, for helping me improve my services. I had to delete a couple extra files (oops.) but in the grand scheme of things, we came out unscathed thanks to my planning and infrastructure layout. It was bound to happen at some point anyway. This was the punch that started the entire thing in motion.
I won't go into more detail about the files themselves as to not tip off our illustrious malware slinging friend if he is indeed reading my articles. But, I will say, I don't blame him very much. It's not that he was serving malware that was the problem. It's the response of the following parties that REALLY sealed the deal.
A Nuisance Masquerading as a Public Service
Netcraft! There's the name. No hiding it here. I love calling out companies. I remember, a long long time ago, that I used to see these reports come in when I used to work at Linode. They were annoying, sometimes correct, sometimes not, but always a bit pompous (not unlike myself). It seems that Netcraft offers services, either paid or for free, that scans the internet or takes tips from customers about malware hosted on sites, in a "cut off the blood flow" kind of malware prevention strategy. It's valid. I do this myself in my line of work - directly contact who I can to get a message through about malware on a site that might be attacking my users. It sometimes works. But you know what these guys do? They email blast EVERYTHING. Your host? Yep. Every possible email address tied to your domain? Yep. Hell, it's possible they may have even gone after my registrar. Look at my inbox. The horror!
We'll get to that ticket email in a minute. Observant readers will know what's coming in the next section. Netcraft goes absolutely bananas trying to reach anyone and everyone they can to let them know there's a malicious file residing on my servers. And again - this on its own isn't too bad. It's annoying, but if I can stop the spread of malware, I'm happy to help, but there's a reasonable way to do it. This is not a reasonable notification method. I post my abuse inbox email everywhere and all you gotta do is take 1-2 clicks to find it in my FAQ, Homepage, even DNS TXT records. Attacking my host(s) with spurious abuse emails inconveniences them and leaves them liable to do something stupid. (I'm sure they're using a script that just shotguns this email out as many times as possible but it'd be nice if they were responsible about using it.)
For your entertainment - My response to Netcraft's ticket system:
I'll add an extra, lighter note here: I'm sure Netcraft's services are for the greater good. It seems like they will be more successful than not. But innocent website operators get dragged into this pissing match when they are not diligent. And even so, I will say, it's still not entirely their fault this is happening. The true, full blame lies on OVH's abuse department, where not even the customer service department knows what the hell is going on. Don't worry! I have more juicy logs.
A Braindead Hosting Company
Thus brings me to our most interesting conclusion. This is the part where we get takeaways (not the food kind)! I'm going to post them here and then explain them one by one:
- Do not use OVH for your hosting services.
- Have multiple cloud services at all times for all critical services.
- Factor into your reliability model the possibility of goody-two-shoes companies attempting to knock you entirely off the internet and have a planned response. This planned response can involve informing hosting companies of the risk so they're ready to NOT treat you poorly, or just shacking up with hosting companies that don't have terrible customer services (like Ramnode).
- (The above is very cynical, so just rewrite it as "Assume any single common thread in your infrastructure could be pulled at any time" if I'm being too salty for your taste).
To my first point, here you go. Here's the email I got at 4:30 in the morning.
Ignoring the busted English there, this would certainly seem odd. One unsubstantiated malware report can take down an entire account? This gives me ideas! Well ok, let's not think too hard on those ideas.
The file did exist, and it was an octet stream. No idea what it was, but usually files on my pomf clone don't get reported unless they were used SOMEWHERE, so there was likely merit to the malware report. I highly doubt OVH has the technical skill or even care enough to try to verify the claim however - so we're going with unsubstantiated as the final determination.
Here is my reply to their bewildering email:
This email has not been answered, and I wait on the edge of my seat for their reply, worried about the fate of my services!
Oh wait. No I'm not. That's not how I built lain.la. This brings me to my second point, have multiple cloud providers!
When you have a template-able endpoint system like I do, spinning up servers to route traffic out is a snap. All I need is a root shell to a VPS, anywhere, and a bandwidth cap that doesn't suck. I maintained my presence across RamNode and OVH, in case either one did something stupid like this, and that foresight paid off in spades. Because I own the hardware, the data, and the core infrastructure at MY home, on MY land, the only way someone can "terminate" my services is by getting past the barrel of my Beretta or showing up with a court order. Both are dubious prospects.
There is also the issue of domain loss but DNS is an unmitigated corporate disaster and ICANN can suc-
Anyway, don't put all your eggs in one basket.
My third point probably won't be an issue for anyone else. But if you intend to host high-risk services, maybe sending a note to your provider isn't a bad idea, to grease them up before the inevitable. Or just don't host high risk services... but where's the fun in that?
As my final flourish, I provide to you the ticket logs showing that OVH customer service cowers in fear at the sight of an OVH abuse agent. Must be fun to wield that power, eh? Also, they're generally useless. (The ticket has been reformatted for readability and some important data has been redacted or changed to protect privacy.)
https://ghostbin.lain.la/paste/apnj3
As always, my users come first and foremost, and not because of any financial gain but because even when shit happens, I'm having fun. Always have fun, even in situations like this.