Pomf is getting stupidly popular these days. Storage usage is growing exponentially (don't worry I'm not deleting any files), bandwidth is through the roof (250TB a month and counting), and more and more sites are using it as a resilient hosting service for whatever it is they need hosting for. At this rate, we might eclipse Catbox.
So, I've seen more and more when vainly googling my site that in some isolated cases, sometimes Pomf is "slow". I want to list all the possible reasons why that might occur, but also let you know that if Pomf is slow for you, you're probably in the tiny minority (except during certain outages) considering how much data I push. But - I'll still fix as many slow speed complaints as I can because I don't want to run a shitty service. So let's go over some common slowness reasons!
You have requested a file from the archives
So, Pomf has 400GB of edge cache per cache node. This 400GB contains every file requested in the past 48 hours (unless it runs out of space, then it starts trimming the least popular files from the disk). Requests from cache are near instant - no waiting required. However, Pomf itself has way more than 400GB of total disk space - so not all files are in this cache. If you request a file and it is not cached (known as a cache miss), the file will be retrieved from the archives automatically. This usually occurs at approximately 25 MB/s, and with the new slice range improvements, most files should reach the edge node within a second or less (provided there is no refill backlog), but there can still be a delay from archival storage.
Please be patient if you are requesting an old file, or if you just uploaded a file and then immediately clicked it to download it - you will have to wait if it is a big file. The max download speed is 4 MB/s currently. There are also 4 of these cache nodes - so you may have up to 4 archive retrievals to sit around for, per file. After that - go nuts. You have the full bandwidth of the cache at your disposal.
***For videos: If you don't want to deal with the cache sync, just save the file instead of opening it directly in the browser. This disables range requests and begins a full file download like normal instead of activating the browser's built in video player.
BuyVM's networking has gone to shit again
Yes this happens more than I'd care for. You can see live network status on the infra uptime page. If any of those "Slice" boxes are down (1-4 carry Pomf traffic), Pomf might run a tad slower.
In the event that a BuyVM node fails, my monitoring system will take a few minutes before deciding it's time to yank the box. Then, there's a few more minutes before that DNS record expires (60 seconds, usually). So, let's say there's a 5-10 minute period before a node is fully terminated from the cluster that 25% of my traffic might hit before moving to a working node. It's a small window, but with thousands of requests a minute, it can happen. If you suspect a node is down (not responding, 502's, extremely slow), come back in a few minutes and see if you don't get a new node (check the request in the developer tools for a new IP address), or check and see if "dig lain.la" via a Linux terminal only returns three or two IP addresses instead of the usual four.
You are on a Pomf binge and got ratelimited (Sites that embed lots of video content may hit this)
It is extremely hard to hit this rate limit. In the interest of transparency, here are the actual limit lines:
- limit_req_zone $binary_remote_addr zone=by_status:10m rate=1r/s;
- limit_req zone=by_status burst=800 nodelay;
Translated to English: You can make 1 request a second, but you get a free pool of 800 requests to play with as a buffer. So you'd have to make 800 requests plus 1 request a second to exhaust the buffer. This is exceptionally hard to do without, again, writing an evil script. You can tell that you're hitting this because the backend response will be "429 Too Many Requests" or a "503 Service Unavailable" in the dev console. Sometimes HTTP 206 range requests or someone on a Pomf binge will hit this, but let me tell you a trick: Just force refresh the page and DNS should shove you to another server, where you get another 800 requests at your disposal!
***Please do not use this rate limiting info for evil. If you do, I will find you and block you, and then you will be sad because no more Pomf. Don't ruin nice things for everyone or yourself. I do also use Pomf for security research so I get my jollies off finding security problems - especially if that's you causing them. By the way, did you know I have a PAID (sorta) bug bounty program? Sniff the HTML on lain.la for more info.***
Your network sucks, throttles/blocks Pomf, or routing issues are bottlenecking data *somewhere*
This is impossible for me to debug, but yes this happens. Verizon used NXDOMAIN injection (they stopped now I think) to block Pomf some time ago. Sometimes mobile networks deprioritize this kind of data. Your network could be having a bad day. A fiber seeking backhoe has rerouted traffic all over the planet before hitting Pomf. You and every Indonesian wants to jack off via the same underwater fiber line. Any of these things can happen. Welcome to the internet. When you run a site on a (comparatively) shoestring budget, you can't have 40 CDN servers in each country ready to route around any internet problems and serve content as fast as possible to a user. You just gotta make do with what you have. I did move all the main edge nodes into the USA now across two primary DCs (there used to be a London one), so at least the performance is geographically predictable now, but still. Shit happens.
Anyway, that should be most, if not all, of the common reasons why Pomf might run like crap for you on a given day. I want to emphasize that 99.99% of requests to Pomf will run super quickly and with the full 32mbps of bandwidth that I allot per user. However - users will always be more vocal when things are going WRONG than they will when they're going RIGHT. See: Any product review page ever. Also see: Any ticket system ever.
Even with that in mind however, I encourage you to please email me at 7666@lain.la if you're having persistently shit performance from Pomf and I'll be happy to debug it with you. This is one of my most interesting hobbies, and I don't want to fall into disrepair. Help me help you. My inbox is always open.
Love, 7666