Yes, I'm putting this article under September. It happened in September. Ignore the fact it's now October. Please.
So, during the testing of my network troubles that I later traced to Verizon FIOS, I rebooted my old core router, a stalwart Dell PowerEdge R320 with a six core CPU, 32GB of RAM, dual PSUs, 2x1GbE, and dual SSDs in RAID 1. It ran the x86 compiled version of DD-WRT, which was dead simple to operate. To my surprise, DD-WRT didn't come back up. No, it hung for a good ten minutes before I plugged a monitor in and realized the RAID card had no virtual drives. What happened, you may ask? I have no fucking clue. The RAID card threw some esoteric error about the backplane not responding after giving it some more time, and then promptly dropped to "No boot device found". My supposedly bulletproof, highly redundant core router, just ate a bullet and croaked. Great. (That's why there's a yellow square in August's uptime.)
I was able to revive the thing by putting the SSDs into slots 2 and 3 instead of 0 and 1 and re-importing the virtual disk. That woke the controller/backplane back up and got it to boot and start shuffling packets again. But that was that. This core router was dead to me. I cannot tolerate a failure of that magnitude from a device of mine. Trust is extremely important in my hardware and software, and if I don't trust the reliability of a system, out the door it goes.
Thus, the core router replacement project was born ahead of schedule. Thanks to my paranoia, I already had a spare core router ready to go. A ripe, Dell Poweredge R330 with a four core CPU, 8GB of RAM, dual PSUs, 4x10GbE and 2x1GbE connectivity, and dual SSDs also in RAID1 on a newer RAID card model. This time, I told myself, I was putting pfSense on it, for a few reasons:
- Much richer feature set and firewalling.
- Reliable DHCP client! DD-WRT's fought me tooth and nail.
- The aforementioned 4x10GbE connectivity from dual Intel X520 cards. This is for Phase 2 of this project: Deprecating the virtualized pfSense and going full bare metal at the network layer!
Look at those beautiful SFP+ ports. Gorgeous.
So, with the help of Jade, I loaded pfSense onto the drives for this new system, prepped all the settings ahead of time via the console, and then yanked the plugs on the old router and ripped and replaced it in about 10 minutes flat. Once it booted, it immediately started passing traffic. Success! That was the easiest drop in replacement I had ever done. So, now Lain.la runs on two pfSense systems - the one bridging the VMs together, and the one doing the core router functions, similar to what your home router does, just with way more firepower.
Check back eventually for Phase 2, when we migrate all the pfSense configs from the virtual instance and move it to pure physical hardware. The virtual pfSense has been burning through a few cores' worth of CPU lately (see image!), possibly due to bugged AES-NI instruction set exposure from the hypervisor, so moving it to hardware works around the problem entirely.
With love,
-7666