Submitted by 7666 on

Oh boy, how things have changed. This has been one of the most aggressive few months I've had for upgrades and changes.

Updates:
  • Stor1.lain.local is now fully operational, with a usable storage pool of 115TB.
  • Stor2.lain.local is now fully operational, with a usable storage pool of 90TB.

  • The Pomf edge caches are now 8x the size, allowing for much better cache hit ratios. This means less loading times for you!
  • Francisco enabled 10Gbit networking for all Pomf nodes, meaning a theoretical 40Gbit capacity for outbound traffic. I don't expect to actually get 10Gbit, but still. Huge, huge upgrade. (quick update: CPUs start to scream at 2gbit, so 8gbit is our max.)

  • The witness script has been upgraded yet again to fix some bugs. It has now gotten heavy use and is fully functional.
  • I have standardized on Broadcom 57810 10Gb SFP+ NICs for all servers. These are actually supported by ESXi 7.0.
  • Invidious is the newest service to join the fleet, at Invidious.lain.la
  • The Generac automatic backup generator project is being scoped out as I type this! The estimate is... pricey.
  • New environmental monitoring in the form of an APC Netbotz system will be installed as soon as I find a serial cable...
  • Pomf will now allow files up to 1GB.
Metrics:
  • Cluster stats:

  • VM Count: 34 On-prem, 11 unique Hentai nodes, 7 endpoints, total of 52 VMs.
  • Edge bandwidth - We're still running pretty hard here. 400TB extrapolated across all 4 edge nodes, plus Hentai and WAN usage. A single node is below:

  • Internal bandwidth between edge and datacenter (Cache ratio - approx. 100x, a huge improvement over the last metrics article. Only 5TB!):

  • Costs: Edge node costs have gone up a bit with the new caching, and power costs have significantly increased with the new storage arrays. We are now at $386.50 a month
  • Pomf:  
    Filesystem          1G-blocks  Used Available Use% Mounted on
/dev/mapper/volume1     9987G 4425G     5558G  45% /mnt/storage
Total File Count is
229468
  • Uptime: I expect this won't improve past 3 9's for right now due to the scale of changes I am making. The next maintenance window is very soon as well.

That's all for now!