Hello dear reader! I want to apologize for it taking me until March to get my first article of the year out, but man has it been a wild ride these past few months, and I have so much news to share. Here's an overview of the big things going on this month:
- The Forest Datacenter will be no more. I am moving!
- Lain.la will have a full 24 hour outage on March 23rd (Raincheck day: March 24th) so I can pack everything onto a big fat truck and drive it 13-16 hours.
- The Townhouse Garage Datacenter will be operational at the end of the outage once everything is offloaded, racked, and powered back up.
Goodbye, Forest Datacenter:
So, the biggest announcement of all is that I am moving halfway across the country! It is a huge decision to take, but after looking at the current economics of the housing market, my new opportunities out West, and of course, moving in with my lovely partner-in-crime Jade, it all added up to a new lease on life that was too good to refuse. And so, it is a done deal. I have myself a gorgeous two story townhouse that will become my new home. I cannot express it in words how excited I am. Here's a little taste of the place: (I'll get more pictures once the servers are online and there's furniture.)
Oh, and good news too: The DIY UPS is coming along as well, and there are no overhead power lines or fiber lines at the new location, reducing the risk of weather related outages. Hooray! I will also be working fully remote from the move date forward, meaning I will almost always be available to assist in case of a hardware failure or power outage (or if you screwed up your free VM). And the power costs are up to 50% cheaper out there, meaning Lain.la's running costs may drop by $100/mo or more! This means I can buy more servers, and I already have!
Please welcome ESXi4.lain.local, a Dell PowerEdge R630 with 2x E5-2690v4 CPUs and a whopping 384GB of RAM. This big boy will be the failover target for ESXi1, ESXi2, and ESXi3 should any of them break. It has been packed and shipped to the new place where Jade will take delivery sometime this week.
Yes, There Will Be An Outage:
I haven't forgotten about all of the services I run for everyone here. Now usually in my commitment to uptime I'd find a way to do this without an outage, but considering I have been the faithful steward of all of these services for three and a half years, I deserve a single day off, so I'm just packing all of it onto a truck and driving it myself to its new home. This is the simplest, lowest complexity, lowest risk way to get everything moved in one fell swoop. The date is set to be Saturday, March 23rd. If for some reason March 23rd doesn't work out, the backup outage day is Sunday, March 24th.
Fun Fact: The approximate bandwidth of a U-Haul loaded with all of Lain.la's servers is in excess of 10 gigabit!
I will do my best to reach out to everyone who has a free VM. I insisted on email submissions for exactly this reason - I should have everyone's email address to notify them of the upcoming outage. In addition, I will be putting a custom 502 page up so that anyone who hits my services while they are down will at least know why. Lastly, Pomf and other cached services may continue to operate in a partial capacity. If a file is cached at the edge, it will serve that file, and Nginx has been instructed to use stale cache files rather than returning the 502. For files NOT cached at the edge, well, you'll have to wait until the move is complete.
The Townhouse Garage Datacenter:
So yes, all of my servers will be installed into the private 2-car garage of my new home. Here's all the prep work I've done:
- Jade has generously provided a 42U APC rack that will be rolled into the townhouse's garage, where a single 20 amp circuit and AT&T gigabit fiber is waiting, making it into the Townhouse Garage Datacenter.
- I will get my semi-custom camera security system operational as well, allowing for the same level of security as before.
- Heating and cooling may be a concern in the warmer months, but I will have my existing temperature and humidity monitoring system in place to keep an eye on the situation. I can always install humidifiers, dehumidifiers, air purifiers, and cooling if need be.
- I will also personally install another 20 amp circuit into the fuse box and run it to the servers for anything that has dual power supplies, so that if the battery system fails or a breaker fails, there will still be a secondary power source feeding the redundant PSU in each redundant PSU equipped device (that's almost all of them).
- There is the potential to get a secondary ISP as there is a coaxial line run into the place, so if I want to add a second router and a second ISP I can still do so (provided the upload speed is at least 150mbps).
- I did latency testing from the new place to my existing edge nodes and noted that there was an increase in latency of 15ms across the board - this is perfectly fine. NYC latency will now be 23ms, and Miami latency will now be 56ms, a minor increase.
Here's the rack plan as of right now:
All in all, this plan should get Lain.la operational after a maximum downtime period of 24 hours. It will take a few hours to pack everything up safely, anywhere from 13-16 hours to drive to the new place, and then a few more hours to offload everything, rack it, network it, power it, and test it. It will be an extremely tough day, but one I'm willing to do for all of you. I hope to have some very nice pictures of the event after all is said and done, and I hope that everything goes smoothly. I do have two sets of backups in case something bad happens, and all of my storage is in varying arrays of RAID should a hard drive have a head crash or something along the way.
That's all for now. The next update on this blog will be from the new location!
-7666