(WARNING! Lots of big images!!! Total: ~25MB)
Hello again!
There is less than three months to go before my target date (September 15th, 2026) to have Pomf back up arrives. While I made it clear in my original shutdown post that I would not be making this a hard deadline, I have been working very hard to meet my goal regardless. I've had a few project trackers and other update posts in the past, but I wanted to aggregate all of them here since there is only one mission and one mission only: Revive Pomf. I'll keep this updated as I go.
First and foremost, in case anyone asks: Yes, your files are still safe. I have continued to maintain the archives. I even had a drive fail and replaced it immediately. Also, as of now, I'd consider the project on track to hit the return date, although the biggest constraint right now is my time.
Anyway, here's the big list.
- (Done!) Move off of BuyVM Entirely. I don't know if I ever actually announced this on this blog or not, or maybe people just noticed, but Pomf was running on my fully owned and operated hardware and networking at the edge, removing BuyVM (now acquired by "Cloudzy AI") from my stack completely, even before the shutdown. And I'm sure I announced this, but this expanded to me acquiring the company that now transits all Pomf traffic as well, which took a lot of time in parallel with the shutdown. All of these operations have stabilized and things are going very well, and it takes multiple third party points of failure out of the equation. Now it's my own routers and switches, my own IP space, my own transit, my own BGP. If I screw it up, well, that's fine, but when other people screw it up, that's when I have a problem. The 2 1U R340's are the edge nodes.
- (Done!) Acquire Hardware. Yep, this took a while. We're in the middle of a remarkably terrible AI hardware shortage because giant corporations can't help themselves torch the balance of supply and demand. This affects me too. I wanted to have all this hardware squared away by the end of May, but that just didn't happen. And you can't really make a whole lot of things happen if you don't have the hardware to do it. Well, I am happy to announce that all the hardware I needed made it here, and it only cost me an arm instead of the full market price arm and leg. The picture below is actually only about half of it, but I had to acquire it in chunks so I didn't really want to wait for the glamor shot, I just got to work racking it all. Also an additional complicating factor: I decided to redo the whole compute side of things, opting for 3x R640s instead of 2x R630s. The old R730 currently running ESXi will be stripped of parts and retired fully instead of added to the cluster. One less step to worry about.
- (Done!) Install all the hardware. Yes, this is a not insignificant part of the operation. I had a lot of stuff to build and configure and install. Here's two images of the scale I am working with here, should give you an idea.
- (In Progress) Get the Power Situation Handled. This is a weird one, but you've probably seen the various solar panels and batteries kicking around some images if you've followed me on Fedi or if I've slipped them into any of these articles. I am waiting for my solar system to be fully operational. The power company has yet to deliver an upgraded transformer, although I am told it will happen this month. While it doesn't explicitly prevent me from launching Pomf, it will take the full power cost out of the equation (which is *substantial*), which would be nice right now. Also, it adds additional redundancy layers to power delivery, because it would also be nice if Pomf didn't go down due to an extended power outage and I maintained my uptime as I always have: ferociously.
- (In Progress) Configure All Hardware. This step involves installing Proxmox on three hosts and Proxmox Backup Server on one host and TrueNAS on the other host, and then configuring the cluster, configuring remote storage, configuring networking, figuring out what remote storage method I want to use (ZFS over iSCSI? Raw remote ZFS? NFSv4?), and then just getting everything aligned and ready to start deploying VMs. This will take a minute. It's a large amount of resources and complexity to deal with. I also have to document all the port mappings and IP mappings for the hosts and each of their iDRACs and get those fired up too, now that I think about it. It's a huge timesink. It will probably take me a solid week or two. I'm getting there though, and my recent experience running Proxmox in production in two full clusters has been incredibly useful. This step will be done when I fire up a test VM and it all just works.
- (Coming eventually) Bridge the gap between Proxmox and VMWare. I have not investigated how to do this yet. I am sure it's possible, and all the hosts have line of sight to each other and each others' storages. However, there is a fundamental difference in operation. The current cluster has storage delivered via iSCSI block devices and VMWare manages a cluster-aware file system. Proxmox does not. Proxmox will consume storage via one of the protocols I mentioned above, but all of them are NOT cluster-aware file systems. The challenge will be - how do I get raw disk images over to Proxmox? Conversion should be simple on the disk image format itself, and the virtual hardware will likely all change and be a mess, but that's fine. I will need to document and handle that process too. But the big question is just, how do you get those block devices advertised to Proxmox? We'll see how this goes. Once that bridge is built, I can start vacuuming VMs over to the other cluster one by one. I already have a plan for Pomf, and that's to just make a whole new VM and rsync the data over, volume by volume, because migrating 90TB of virtual disks does not sound fun to me without checkpoints. I'll know this step is done when one VM migrates, and then it's just a sequential operation.
- (Coming eventually) Bootstrap the Backups. So, the current backup scheme involves a Veeam VM hooked to STOR2 on two very large LUNs. I will be naturally moving to Proxmox Backup Server on STOR3 and then rolling STOR2 into it post-migration. However, I have significant concerns about how PBS works vs Veeam. One such concern - How will PBS handle reading the VM disks for changes? Will it have to rescan the entire disk, or can it used changed block tracking? If it's the former, then this will make daily backups extremely difficult as the limit will be the disk arrays and then the 10Gb network. Even if we got 1GB/s of reads, that would take over a day to process 90TB, and this system is supposed to scale to 300TB+. I may be able to mitigate this by using 10TB volumes instead of larger volumes for Pomf as I already do now, but this will still be a scaling nightmare. The ideal is that it's an incremental backup each time and knows what changed and only vacuums out that for the next incremental. If that's the case, then it'll work like Veeam did and will be fully sustainable going forward. My next concern will be restores though. I know that you can restore a disk image and either boot or mount that, and that's fine, but now imagine doing that to 10TB or 50TB of data. File level restore is incredibly important, because I shouldn't have to mount the entire Pomf drive just to get at a single file. If PBS also offers mounting the restore disk over NFS vs. copying it to the host storage, that would also be a plus, because Veeam could do that too. I would then be able to mount the entire Pomf directory and yoink a file out over NFS and then shut it off without explicitly needing file level restore. Just, lots of research to do here (Image is not of the Lain.la backup servers, but the ones sitting in the cave in Missouri for Nulled).
- (Coming eventually) Migrate Literally Everything. Yep, by this stage we'll be ready to start the process of moving everything off, starting with VM disk images for specific services, repairing any VMs that break on the way over, and ending with a new Pomf VM and a migration of all the Pomf specific items (DBs, cronjobs, nginx config, etc.) and that migration itself ending with the rsync of all the directories over. To streamline the migration process, I am not taking any of the free VMs over. Anyone who has a free VM through me will be kicked off when it is time to decommission the old hardware, although I have reached out to everyone I possibly can to make sure they are accommodated with a Nulled VPS for free instead. If you have one of these machines and haven't submitted a request for a free Nulled VPS, I suggest you get on that by emailing me.
- (Coming eventually) Refurbish Literally Everything. Yeah once the migration is done I'm not done. I have a ton of catch-up work to do on the VMs itself, the network layout, etc. All the VMs need to be updated, all the software/applications checked over and updated where necessary. The firewall rules need a big cleanup, the edge nodes themselves could use some polishing. The Pomf website could use a coat of paint, and I also need to implement new security measures around CSAM, AI abuse, etc to ensure that Pomf doesn't get destroyed by new and emerging threats to simple file hosting. Maybe we should update the firewalls while we're at it too. I could also use some new tooling around abuse report processing. And this blog's even a few versions behind. I want to be sure that Lain.la is in the best possible state to handle the restart of Pomf, so all that technical debt that's been sitting will be rooted out and purged like the heresy it is, and only when I am satisfied, can we continue.
- (Coming eventually) Fold in the old hardware to the new cluster. So this step is a little less complex in that I will be scrapping ESXi3. But, the storage servers (STOR1 and STOR2) are perfectly good and hold a ton of storage (200TB usable). So what we will need to do is pull ESXi3 out of the rack, strip and re-allocate (or sell) parts, get whatever's left ready to give away, and then update TrueNAS on STOR1 and remove TrueNAS and add PBS to STOR2. TrueNAS can be a standalone system as an additional mount for the hosts on STOR1 (STOR4 being the new storage server), but STOR2 ideally will work in tandem with STOR3, so I'll have to figure out how multiple PBS servers work together. Will it be seamless? Will I need to spread the jobs across the two hosts? Can it multiplex (e.g. multiple jobs from multiple hosts going to multiple servers)? We'll find out! I'll feel like Anthony Bourdain mixing the fish from yesterday into the stew, mixing the old cluster and new. But hey, it's practical.
- (Coming eventually) Create an LLC. I almost forgot to add this part to the to-do list, but I find it to be tiresome paperwork, so can you blame me? I do need to incorporate Lain.la into an LLC, not because I want to sell products or anything, but because I need the legal protections of a properly formed company to ensure the continuity of Pomf. Many, many things that go on with Pomf are matters of a legal nature (remember the transparency log?), and it is discomforting to me that I must bear these entirely myself. I could be held personally liable for a spurious DMCA, for example. So this is a natural safety net. This will take some time and money to get right, so bear with me if this ends up holding up the timeline a bit.
When this ENTIRE LIST is completed to my satisfaction, the floodgates will open, everyone's files will be back exactly as they were, we'll have another 200TB of usable storage or so to work with which should last us at least for the next two years, every file will have daily backups again, all the hardware and software will be freshly updated, and Catbox won't have to front the entire load of Pomf traffic anymore and I won't have to feel bad about it.
Big dreams, but all within reach. Here's hoping I hit my goal! The road ahead is long and winding, as it always has been.
-7666