Recently, I was asked by a friend to solve a long standing issue in my network for their freebie server. Under a network where NAT is being done at least two network hops upstream, the source IP address will be lost in translation, making it impossible to determine the real IP address of an incoming client to a server inside my network. That server will only see the upstream edge node's IP, not the user, such as 10.13.0.1. Their use case for passing the real IP through the whole tunnel was simple: How can we ban or track malicious users on our server if we don't have the user's IP address to work with?
NAT is a core component of my multi-VPN tunnel network. I don't like it, but it's the world I live in without an ASN, a /24 for my own private use, and BGP announcements (which are all VERY much outside my comfort zone). Lain.la is very scalable and secure-by-default, in that nothing is allowed to route inbound without my say-so at two critical junctions: The iptables rules on the edge nodes, and the NAT mappings on pfsense. The downside: doing double NAT puts you into this issue with the loss of the source IP downstream.
I just now found a reasonable way to solve the problem when you do not control the target server and need to route passively - ensure that you have provisions in place to send traffic destined for outside your network to a designated non-masquerade gateway via manipulation of the firewall's default route and then shut off masquerading entirely on that gateway only. Only traffic that isn't masqueraded will match the default route, as we expect there to be internal routes for the OpenVPN tunnels to handle that traffic. Note: This DOESN'T work if you want to turn masquerading off for all gateways - instead you need to designate a non-masquerade gateway and put your default route on that because we can only have one default route at a time - then send non-masquerade applications inbound through the same spot.
(Note: This article may be considered a companion or response to https://flisk.xyz/traffic-forwarding-without-masquerade.html)
I'll start this article off with an overview of the inbound and outbound routes that a packet takes, then we need to talk about the role of the IP masquerade.
INBOUND:
OUTBOUND:
Bonus note regarding outbound policy routing on the NAT gateways - policy routing is basically just taking firewall rules and making them control the default gateway of any traffic passing through them. See below:
In this firewall table, the rules of note are the ones with gateway not set to *. This means that if an outbound packet MATCHES the rule set here, use the gateway group provided in the gateway section. Gateway groups are a bit of pfSense magic that allow you to use tiered gateways for redundancy, with conditions for when to fail over to the next gateway in the tier, Here's how mine are set up (Oh - and these outbound rules actually have nothing to do with masquerade removal as these are only regarding outbound packets initiated by the SERVER - not replies to incoming packets. Shitty description on my part.):
Now, in the case of the inbound packet, IP masquerade is what rewrites the source IP to 10.8.0.1. IP masquerading is just a fancy term for automatic SNAT, or Source NAT. Its job is to help servers behind it get packets out to the internet. Otherwise, what happens is, the server behind the firewall tries to reply directly to the user on a public IP address, and pfSense doesn't have a route to that particular IP, so you are at the mercy of the default route of the router. If that default route happens to NOT be the gateway in which the packet came through, you will end up with triangular routing! See below.
This was the problem with masquerade. Turn it off, and replies go out the default route which in this case was NOT the inbound source of traffic. Turn it back on, and while the more specific routes will get the traffic out the right way, you lose the source IP because that's needed to select the correct gateway.
My solution to use the default route to send non-masqueraded traffic through isn't perfect, but gets the job done without screwing up any other routing, as we expect almost all routing to be covered by a more specific route thanks to the masqueraded IPs. It is also a permanent fix as it's now a static route in the routing table. The only downside is that if you have a multi-VPN failover system (or you publish apps on more than one IP), that backup IP address needs to be on masquerade mode or when it fails over to another gateway or your users come in via the backup IP, you'll be back at triangular routing. So you can only have this real IP functionality on one point - the default route.