VSX: Mixed Routed and Bridged Mode

[Anthony_Kahwati]

Hi CheckMates

I currently have a VSX deployment on R80.20 that is running a number of routed VS's who have access to a common V-Switch for access to our ISP VLAN. We are also running VSLS.

A requirement has recently come up for us to put in place a VPN head end that must not have it's public interface NAT'd. Enter the bridged firewall.

When I try to do a moc-configuration of this I get an error immediately at the first step that shows me the below:

I've been told by our support partner that the resolution for this is to change the VSX Bridge configuration  Bridge Active / Standby  State Determined by: from Standard Layer 2 Loop Detection to Checkpoint Cluster XL. They also suggest that I should not move to the CXL mode without a lot of thought and planning into the spanning tree protocol on the surrounding devices. For information, there are Cisco Nexus 7 or 9K's north and south and running VPC to the Checkpoint's bonded interfaces.

I'm going to catch up further with them to hopefully get an understanding of this as it's something a bit out of my Checkpoint experience but I was also hoping that someone else has some insight of how I can achieve what I want to, and, what the implications of moving from Standard L2 to CP CXL are?

Many thanks

[Bob_Zimmerman]

Functionally, there are no concerns in changing that setting for this exact cluster. You don't have bridging working now. Changing it lets bridging work with VSLS. The big concern with ClusterXL selection for active/standby state is that STP on adjacent switches doesn't always update quickly in a failover. After all, the interfaces are still up, so the path should still be good, right? So the cluster has to do weird ARP shenanigans to try to convince the adjacent switches to use a different path. This wouldn't break anything you currently have working, though it will probably take some experimentation to find the right settings on the adjacent switches.

Alternatively, you could use a normal, routed VS, add a proxy ARP statement for the VPN box's IP on the outside interface, then add a 32-bit interface route out some other interface to get the traffic to the VPN box. The VPN box would then have its default route point to the private address of the "some other interface" on your firewall. This will absolutely work, and the VPN box can have the public address it wants with no NATing. I would even say it's simpler than getting bridge mode to work with VSLS.

You could also change the cluster from VSLS to normal HA.

[Anthony_Kahwati]

Hi 

Thanks for your reply, it's really useful. This makes perfect sense actually, especially the part about the Bridged mode not being in place. I kept tieing the change to something that can affect the routed VS's but of course it wont.

On your point about using a routed VS. I'm guessing that you are saying to just update the local.arp file with the new address? I get that I can send and interface route rather than a next hop IP route towards the VPN, what I don't get is the VPN being able to send it's default route traffic to an address on a different subnet, even if it is out of that interface?

I'm guessing I am just missing something fundamental but to me, the VPN should drop traffic to it's default route next hop if the default route next hop is not in the same subnet?

Many thanks

[Bob_Zimmerman]

Routing is a lot simpler than people think. When a box gets a packet, it checks the destination against all of its interfaces to see if the packet is for it (some systems are "strongly bound", which is a little like antispoofing; we can ignore that for this discussion). If the box does not have an interface with that IP (and if it has IP forwarding enabled), it sends the packet to the FIB for a decision on where to send it next.

A FIB is implemented as a data structure called a binary trie (pronounced "tree"; no, I don't know why the weird spelling). You start at the root node, which is the mask 0.0.0.0. Each node has a link to the next two nodes. As an example of the links, 0.0.0.0/0 has a link to 0.0.0.0/1 and 128.0.0.0/1. 128.0.0.0/1 then has a link to 128.0.0.0/2 and 192.0.0.0/2. And so on. Each node also has an optional interface to send the traffic out (always defined if there is a route for that address/mask combination), and an optional gateway IP address (defined if the route has a gateway). Once you have crawled the trie to find the longest mask match for your destination IP, you use that node's interface to send the traffic. That's really all there is to the routing decision.

If there no gateway IP, you ARP for the destination IP. If there is a gateway IP, you ARP for the gateway IP. That's all the gateway address actually affects.

This gets a little more interesting on systems which support multiple FIBs (think VRFs, Linux network namespaces, OpenBSD rdomains). Ultimately, you just have multiple tries, and a little metadata connected to the packet tells the OS which trie to crawl for that packet.

Some operating systems try to make things "easier" by hiding some of this from you. This almost invariably results in making things much more confusing (much like VSX itself is more confusing than it would have been if they had just exposed the VRF or now network namespace functionality directly). You may be able to specify your default route in terms of an interface and a gateway directly, or you may need to specify a smaller route for the gateway first so the "helpful" OS knows which interface to use. I have never seen an operating system where this would not work, though.

Let's say the VPN endpoint's public interface (eth1) is 2.3.4.5, your Internet firewall is VS 5, VS 5's public interface (bond2) is 2.3.4.2, and VS 5's interface towards the VPN (bond1) is 10.20.30.1. Add a local.arp entry on VS 5 telling it to claim to own 2.3.4.5 on bond2. Add a route on VS5 telling it 2.3.4.5 is out bond1 with no gateway. Add a route on the VPN endpoint telling it 0.0.0.0/0 is behind the gateway 10.20.30.1. You might also need to add a route telling the VPN endpoint that 10.20.30.1 is out eth1, but you might not.

When the client 6.7.8.9 tries to connect, the Internet router will ask where 2.3.4.5 is, the firewall will respond with its MAC, the packet will go to the firewall. The firewall doesn't have an interface with that IP, so it will consult its FIB and get the decision to send it out bond1 with no gateway (thereby telling it to ARP for the destination, which is still 2.3.4.5). The traffic will get to the VPN box, and the VPN box will send a response. Assuming it doesn't cache the request details to prepopulate the response, it will consult its routing table and won't find a longer match than the default route, which will tell it to go out eth1 with the gateway address 10.20.30.1. It ARPs for 10.20.30.1, the firewall responds, the packet goes to the firewall, and the firewall sends it along.

Years ago, due to some insane requirements, I had to build a firewall which could be inserted between servers and their default gateways (so bridge mode), but which could also NAT the traffic to maintain symmetry through load balancers (so not bridge mode). That firewall works a lot like what I described above.

[Anthony_Kahwati]

Hey Bob

I just wanted to my thanks for you putting such a detailed reply in. I have to say, even as a network engineer I am learning something new with this. I need to digest it and spin it up on a lab if I can so just wanted to make sure I reply and say thanks rather than wait until I have had a go at what you suggest. 

Anthony

[Bob_Zimmerman]

Sure thing! Back when I took CCNA courses, there was a fairly early lab which demonstrated this (I think in CCNA 2, but might have been in 1), but they didn't really explain why it worked the way it did. I just filed the knowledge away, then it clicked years later when I was digging into the FreeBSD network kernel for an unrelated reason.

It's actually how ClusterXL VIPs work. This is most obvious in VSX, but normal clusters handle the inbound leg in the same way and can handle the outbound leg in the same way. Check Point has wrappers for several common Linux commands (to see this, run 'which ifconfig' and compare to 'find / -name ifconfig') which return cluster information on VSX. If you take a look at the interfaces with the real ifconfig or real ip, you will see that they have automatically-allocated addresses which are not on the same network as the cluster VIPs you specify for the VS in Dashboard. Similarly, if you check out the routes, you will see a route for the member network and a route for the VIP network pointing out the same interface.