Hello once again! It's been a long time since I've done a metrics post, I know. I'm sorry. Lots of stuff happening.
Updates:
- The Pomf LVM cache system is back in action, with six times the storage and also three times the cache. Performance seems to be a lot better this time, and WAN bandwidth utilization is falling like a rock (which is good)!
- I discovered that Verizon was the root cause of the 1-3% packet loss on my OpenVPN tunnels. Curtailing WAN bandwidth to below 300 mbps ensured sub 1% loss. This sucks, but it is what it is. The issue is now solved at least with the LVM system back in place.
- I now have a CyTube instance at https://cytube.lain.la/ with support for manifests via Pomf (and you can use Pomf for all your videos)!
- I am finally satisfied with my Seedbox deployment. Three VMs are fully operational now, and the fourth is just awaiting something to seed.
- The core router has been replaced with a pfSense box on a shiny new R330, cutting DD-WRT out of the core router role and allowing more granular reporting and much nicer networking features.
- We survived a 14 hour power outage. See more here: https://infrablog.lain.la/incident-12182023-1
- The Lain.la network map is now on version 2.4. See the changes here, and ignore the 2.3 in the URL. I swear it's 2.4: https://infrablog.lain.la/lainlav23-diagram
Metrics:
- Cluster stats:
- VM Count: 28 on-prem, 19 user VMs, 5 hentai nodes, 6 endpoints, and 1 management node for a total of 59 VMs. Hentai was scrapped largely so that's why there's a drop. A user VM is one of those free ones I give out. 19 user VMs is a lot! I'm happy to help!
- Edge Bandwidth: Currently looking at about 470TB outbound bandwidth a month. A little less than the last article, but still a huge amount. Here's a graph of one of the four main edge nodes. Multiply by four to reach the real total, then add a little bit extra for the rest of the infrastructure.
- Internal bandwidth has been high. But with the new caching architecture, I'm expecting this to be cut in half. Also, the graph has changed thanks to the core router replacement, this is a pfSense plugin called Traffic Totals. The chart below is pretty close to DD-WRT's. The traffic for December was 81TB out and 13TB in. A new record.
- Costs: These have gone up a little bit since the last article thanks to the extra edge cache. Total cost is now $595.50 a month.
- Uptime: Still pretty good! Most services are almost at 4 9's of uptime. See here for details. We took a hit due to some WAN interruptions caused by that 14 hour power outage.
So, that's all for now! As always, feel free to email me at 7666@lain.la to say hello or if you have questions or suggestions.