Submitted by 7666 on

It's February, and I have a lot on my plate for the entire year. I wanted to give everyone an update on where I'm at with my projects. Yes, plural. We have two in the pipe:

  • The Proxmox Migration Project
  • The BuyVM Migration Project

As you may be aware, the BuyVM Migration Project kicked off at the start of the year when Francisco sold the company (which concerns me that new ownership might start making service changes) - this was not a project of choice but of necessity, although now I'm making the choice to commit to it fully because it also solves long-standing performance issues with Lain.la.

          

The Proxmox Migration Project

This project has not been on schedule due to a few reasons:

  • The BuyVM Migration Project taking 80% of my available time right now.
  • Proxmox and iSCSI don't get along very well. I've had to make some architectural changes that I don't know the implications of yet, such as ZFS over iSCSI.
  • There's some bugs with networking I haven't quite squashed yet preventing one of the KVM hosts from communicating properly. It's a tricky one.

For status, here is the previous project plan, expanded to show some new tasks I took care of recently. You can see we're about 1/3rd of the way through, but there's a lot more to go:

  • (Done!) Acquire hardware to build a Proxmox cluster ALONGSIDE the vSphere cluster.
  • (Done!) Configure and install all hardware.
  • (Done!) Load all operating systems - Proxmox Backup Server on the backup server, TrueNAS on primary storage, and PVE on the compute nodes.
  • (Done!) Decrypt VMs in VMWare for migration preparation.
  • (Done!) Rewire and power balance the entire rack.
  • (Done!) Build finalized network diagram (Lain.la 4.0)
  • (Done!) Deprovision old or unused services to minimize migration overhead.
  • (In Progress)  Configure all storage. PBS integrated into PVE, TrueNAS volumes integrated into PVE.
  • (In Progress) Configure all networking, redundancy, failover, VLANs, bridges, etc.
  • (Coming eventually) Fire up a bespoke VM to test everything above.
  • (Coming eventually) Once the cluster is confirmed working, test the migration plan by decrypting ESXi5's VMs and reinstalling ESXi5's OS to be PVE and importing the VMFS6 volume on disk. Configure the above networking and such too.
  • (Coming eventually) Convert the seedboxes on ESXi5 to qcow2 and boot VMs. If it works, we have a way forward.
  • (Coming eventually) Test a migration from existing storage (stor1) to new storage (stor4) of a VM from vSphere and refine that process.
  • (Coming eventually) Move all freebie VMs (yep, you're the guinea pigs!).
  • (Coming eventually) Get backups going to my specs.
  • (Coming eventually) Get offsites tested and going to my specs.
  • (Coming eventually) Move Pfsense.
  • (Coming eventually) Move all lain.la VMs under 1TB.
  • (Coming eventually) Figure out how the hell to move the big VMs (Pomf, Minio. Probably gonna rsync, honestly). Then move them.
  • (Coming eventually) Once the old cluster has been completely evacuated and everything has been working for AT LEAST two weeks, dump stor1, stor2, and esxi3's OS and reinstall with PBS, TrueNAS, and PVE and integrate them into the current cluster, creating an absolutely massive Proxmox cluster.

Here's a shot of KVM2. We're getting there!

The BuyVM Migration Project

So this is the new project on the books, and I'm just now doing a project plan for it. This one is much simpler than the Proxmox project, but also at the same time much more difficult as it requires resources I've never needed to acquire before such as transit. Here's what the general overview is like:

  • (Done!) Determine new architecture.
  • (Done!) Locate and contract new hosting provider.
  • (Done!) Acquire hardware for new edge nodes.
  • (Done!) Determine feasible migration path without outages.
  • (In Progress) Configure hardware, software, storage, OS, etc. for new edge nodes.
  • (In Progress) Determine upstream transit provider availability/blend/cost/etc.
  • (Coming eventually) Determine DDoS protection systems for upstream transit.
  • (Coming eventually) Rack new servers in 1 DC once the above pre-requisites are complete.
  • (Coming eventually) Configure new networking (IPv4, IPv6, OpenVPN, iptables, iDRAC)
  • (Coming eventually) Migrate Nginx config to new server along with certificates, web directories, etc.
  • (Coming eventually) Repoint monitoring script to new server once published. If it passes all tests, put it into DNS as server #5 and observe traffic and cache fill rate.
  • (Coming eventually) Load test: remove ALL BuyVM servers from DNS, leaving only the new one, to test system throughput. Revert when satisfied.
  • (Coming eventually) Bring the second server into production, test, and then eject Miami from the cluster.
  • (Coming eventually) Begin user migration procedure. We need to move DNS records and NAT mappings off of BuyVM's infra. Email everyone on Slice1-4. We're keeping 5-6.
  • (Coming eventually) Rack servers in the other DC as available. Repeat the above procedures. Eject NYC from the cluster. Project complete!

This project is about a quarter of the way done, but realistically the heavy lifting has yet to begin. Much to do there! Here's a picture of the new servers I bought. I call them "Francisco Killers".

Specs:

  • Dell R340s
  • 4.8GHz Cascade Lake CPUs 
  • 32GB of RAM 
  • RAID1 M.2 boot disks 
  • 10Gb networking
  • Single 7.68TB SAS SSD for cache
  • Dual PSUs
  • iDRAC Enterprise

Funding

For the moment, funding is not realistically holding anything up, but I have distributed quite a lot of money for the BuyVM Migration Project this year.

  • $6,000 to secure a hosting provider for a year with my specifications.
  • $2,500 for servers.
  • $3,000 for parts (drives, rails, RAM, CPUs, whatever).

This is in addition to last year's expenses for the Proxmox Migration Project:

  • $3,000 for servers.
  • $4,000 for parts (drives, rails, RAM, CPUs, whatever).

This means I'm out $18,500 thus far. My reserves are depleted enough to where I will not be doing any more projects for the full year, and support costs for each must not exceed another thousand each or we're gonna have a problem. If I have to renew BuyVM's contract in November I'm probably toast, too, so there is a hard deadline here. Thus far I've received donations totaling $77  $377, which may sound like a pathetic amount compared to the dollar figures above which is actually more than I expected to get despite the total cost still being extreme, but I really appreciate every vote of confidence and couldn't be happier that people care enough to even donate that much. I am very much used to being the thankless man behind the curtain, so anything is a delightful surprise. If you are interested in helping out I'll drop the usual link here, but I don't want to be too much of a beggar about it.

As always I will continue to forge ahead no matter what, even if my available time only permits me to do a little bit here and there. I appreciate your support. Even reading this article means the world to me.

-7666