Eliminating Routing Asymmetry between Two Different Physical Sites Running Check Point Firewalls

[Di_Junior]

Dear Mates

I need your help with regard to a complex issue that I have inherited and now I need to provide solutions for it.

We have two geographically separated sites (represented as SITE A and SITE B), and on both Sites, we are running Check Point Firewalls. Each Firewall is connected to a Router (ROUTER 1 - SITE A, and ROUTER 1 - SITE B). The two routers are then connected to other Routers in the user space through our MPLS network.

Below the Firewalls, there are another routers (ROUTER 2 - SITE A, and ROUTER 2 - SITE B), these two routers also know each other through our internal MPLS network.

Below the two routers (ROUTER 2 - SITE A, and ROUTER 2 - SITE B), there is our Datacenter where all our servers reside.

ROUTER 2 - SITE A has the default route pointing to FIREWAAL SITE A

ROUTER 2 - SITE B has the default route pointing to FIREWAAL SITE B

The issue that we are currently facing is that when our users (offices, Stores, etc) access our servers in the Datacenter, the traffic to the server can get in through SITE A, and the return traffic from the server can go back through the Firewall on SITE B on its way back to the User who requested it. Unfortunately, the traffic gets dropped because it did not get in from the Firewall in SITE B.

This issue is happens because Servers in our Datacenter have different default gateways (some have default gateway to SITE A, and others to SITE B)

I would like get your contribution based on your experience on possible solution to solve this problem. Your help will be appreciated.

Thanks

[phlrnnr]

Does Checkpoint have any plans to implement state-sync via a Layer 3 interface, and allow some sort of Active-Active clustering, not reliant on Layer 2?  Then the firewalls in this example could be setup as a cluster and share state info (assuming latency requirements, etc are met). Then the traffic flows could be asymmetric and the firewalls could still process them.  This would be very useful for larger enterprises with disparate datacenters and multiple egress points.

It seems the only other options today are:

1. Span L2 between datacenters and create a split-DC cluster (with all the caveats around stretching L2)

2. Using routing protocols, force all users to favor Site A, and only use Site B for DR, knowing that in a failure traffic will get re-routed to Site B and stateful flows will get dropped and have to be re-established.

[Kaspars_Zibarts]

Can't agree more, either go with HA site approach so you have a primary and backup sites or stretch L2 and make firewalls part of cross-site load sharing cluster. 

Or "learn" each server's default gateway and have path statically configured on R1  

[Ricardo_Gros]

Hi,

This is a rather hard one and I don´t think a solution can be provided just over Checkmates, You have several routing possibilities and probably the solution will require architecture changes. 

One solution is Source NAT on the LAN SIDE of the firewall but this I would not recommend as it is just a workaround to an actual bigger issue and could also have other impacts that I cannot access just by this design.

What kind of Routing protocols and where are they applied? if OSPF is being used I believe  we need Areas to be reflected on the design so we can understand what happens in terms of routing databases. 

Are the Servers  organized in network ranges? can we combine all servers with the Same default gateways into a subnet? 

Can you share some more detailed information the routing configuration of all devices?

[Maarten_Sjouw]

The only way to solve this is by using Dynamic routing and have the 2 routers-1 divide the user sites to split the load and this way by giving a better path for remote site 1 via path A and a better path for remotes site 4 via path B.

I know it can be done with routers to get preferences set this way, don't know how to do it though.

In the router 3 layer you will need to allow a vlan for the routing to be able to make sure it can get back to router-2 in site B when router 3 sete A is the HSRP master.

Another method would be to flip it around and set priority for specific server VLANs via the site B and for other VLANs via site A. This will also simplify the issue a bit more as you also set HSRP to the site with the priority route.

[Di_Junior · Accepted solution]

Hi All

Thanks for your contribution. I wish to share some more information with regards to this issue.

One of the possible solutions to this issue is to use routing protocols as already mentioned above. The figure bellow highlights what we wish to achieve. However, apparently there is a limitation in Check Point when it comes to redistribution routesto specific interfaces, instead it redistributes the routes to all information. Is this information about redistribution true? if now, how can one ensure that routes are only advertised on specific interfaces instead of on all interfaces.

Thanks

[Maarten_Sjouw]

You don't want to not redistribute specific routes, you should only have a difference in weight/priority which should be defined in the Router 3 level.

This way if either of the 2 site paths is interrupted it will still know how to get to the servers.

[Di_Junior]

Hi Maarten Sjouw

Can I make the firewalls announce the network ranges with priority on each site using OSPF? or we can only achieve this at Router 3 level.

Thanks in advance

[Maarten_Sjouw]

You should only do this on the Router 3 level, run OSPF from top to bottom and start at the beginning to make sure to accept and forward the weight of the routes through the whole path.

Make sure to set the weight of the mpls link between routers 2 at a uninteresting level, so the traffic will always take the route straight up/down, preventing the return traffic from router 3 Site B to take the MPLS path back to Site A again.

An alternative way to accomplish this, when you do not want to mess with Dynamic routing, is making use of tracking in the routers and statics based on tracking also here you need to have a second static that will take over when the tracking fails. Be sure to not set the tracking times to short.

[G_W_Albrecht]

Why is this question marked as a solution ? Or is: Is this information about redistribution true? if now, how can one ensure that routes are only advertised on specific interfaces instead of on all interfaces? a solution ? To which issue ?

[Ricardo_Gros]

Hi, 

What Maartens is saying I would just add: 

If you run OSPF only UP - DOWN on both A and B sides and have the mpls Link with lower priority like already stated you should be able to make sure that preferred routes are always in the same side. 

It could help to have 2 areas, running zero between Router 1 and 2 for example and running different  area for the firewall to avoid having loops with the IBGP link if the area 0 gets broken. 

One thing I would definitely do is use TAGS to identify traffic from side A and Side B. This should help you troubleshoot since you can identify origin of routes by the tag. 

Be sure to test all redundant scenarios after to make sure there is no flaw on the design, this has allot of possibilities and possibly we are not seeing all variables.

[Di_Junior]

Hi Ricardo

Thanks for your help.

Could you please elaborate more on the text bellow, specially when you say "different area for the firewall".

"It could help to have 2 areas, running zero between Router 1 and 2 for example and running different  area for the firewall to avoid having loops with the IBGP link if the area 0 gets broken."

Thanks in advance.

[Di_Junior]

Hi @Timothy Hall 

Is there anything you would add in addition to the wonderful contributions given above.

Thanks in advance

[Timothy_Hall]

Well there are three ways to deal with this asymmetry:

1) While this asymmetric routing situation can be made to work by disabling the "Drop out of state TCP packets" checkbox under Stateful Inspection in Global Properties, this will defeat a major security feature of Check Point firewalls and the asymmetric flow of packets can cause some really odd-looking performance issues if one of the paths (A vs. B) gets heavily congested.

2) While not ideal, you can force symmetry in this situation regardless of the server's default gateway by performing a Hide NAT on the inbound traffic from the offices/stores behind the INTERNAL interface IP of the firewall.  So if the offices/stores traffic comes in through Firewall A, prior to leaving Firewall A the source IP is NATted to the INSIDE IP address of Firewall A.  The return traffic will come back to Firewall A regardless of what the server's default gateway is.  If the offices/stores traffic comes in through Firewall B, prior to leaving Firewall B the source IP is NATted to the inside IP address of Firewall B.  This will force the symmetry you desire with the following impacts:

• Use of Hide NAT assumes that connections are initiated from the offices/stores to the servers at the bottom of your diagram.
• If there are any kind of ACLs on Router2/Router3/Servers leveraging certain source IP addresses for access control, those will no longer work because all traffic from the offices/stores will look like it is coming from a single IP address, that of the firewall's inside interface.
• Extra overhead will be incurred on the firewall to perform these NAT operations, and if there are a very large number of connections from all the offices/stores to a particular server destination IP address, there is a 50k concurrent connection limit per destination IP address when Hide NAT is involved that might get exceeded. 
• You'll probably need to use Manual NAT rules to obtain the granularity needed to make this work.

3) Use of OSPF as discussed previously - this will take much more planning and setup time but in the long run will give you a lot of extra flexibility to automatically deal with full or partial site failures, and even allow some load-sharing & balancing of traffic between the two sites.

--

CheckMates Break Out Sessions Speaker

CPX 2019 Las Vegas & Vienna - Tuesday@13:30

[Di_Junior]

Thank you very much. The second option which was also suggested by Ricardo Gros seems to be a quick win, and to overcome the 50k limitation, I can follow the Manual NAT configuration described in your book and mentioned above.

Once again Thank you.

[Timothy_Hall]

Right, just wanted to mention the NAT approach again but also include the limitations you might run into.  Good luck!

--

CheckMates Break Out Sessions Speaker

CPX 2019 Las Vegas & Vienna - Tuesday@13:30

[Di_Junior]

Hi Tim

I got your point. Thank you very much, I will update this thread after the implementation.

Regards

Dialungana Malungo

[Di_Junior]

Hi again Tim,

Just a question, do I also need to create ARP entries when doing Hide NAT to the Internal LAN as done when doing it to the public network (Internet).

Kind regards

[Timothy_Hall]

As long as you are hiding it behind the actual internal firewall LAN IP (or CIP/VIP), no you don't need a proxy ARP.  If you are plucking an arbitrary address from that internal VLAN and hiding behind that one via a manual NAT rule then yes you will need a proxy ARP entry for it.

--

CheckMates Break Out Sessions Speaker

CPX 2019 Las Vegas & Vienna - Tuesday@13:30

[Di_Junior]

Hi Tim

Taking this opportunity that we talking about Manual NAT,  I wanted to share the issue that I am facing in my Remote Access VPN.

I am trying to NAT the IP addresses of my Office Pool for Remote Access VPN into a range, I have done all the configuration, and created the Manual NAT rules, when I connect to the VPN, I can see the traffic and the NAT happening in smartview tracker, but the connection is not successful. For example, ping times out. 

When I remove the NAT configuration, the office pool address is being NATed to the IP address allocated in the firewall interface, and it works fine; hence users can access everything.

Regards

[Di_Junior]

To add on this, when I check on the wireshark, I can see that the traffic from one of the IPs in the range that is being used for Hide NAT is sending the request, but no reply from the machine in our local network.

[Timothy_Hall]

Are the addresses you are manually NATting the Office Mode addresses into being plucked from the same subnet/VLAN as the firewall's internal interface?  If so you need proxy ARPs for all of those addresses, see sk30197.  Are you seeing constant ARP requests with Wireshark on the firewall's internal interface when replies are not coming back after sending the NATted traffic into the network?  Also check the output of fw ctl arp, this will tell you what IP addresses the firewall thinks it needs to proxy ARP for.

--

CheckMates Break Out Sessions Speaker

CPX 2019 Las Vegas & Vienna - Tuesday@13:30

[Di_Junior]

Hi Tim

The issue was found. There was a change the needed to be made on the Global Property "Merge manual proxy ARP configuration" that was not checked. After enabling it, it started working perfectly.

Kind regards

Dialungana Malungo

[Di_Junior]

Hi Timothy Hall 

I would like to take this thread to ask you another question.

I quote from your book on page 299  with regards to IPS Protection Scope "Also consider the positioning of the subject firewall in your network; the correct setting is likely to be different for a firewall deploayed on the perimeter of your organization, as opposed to another firewall buried deep inside your internal network with no direct Internet access."

We currently have this situation where we have an external Firewall with direct access to the Internet, and an Internal Firewall Firewall with no direct Intenet Access. Based on your experience which IPS Protection Scope would you recommend for each Firewall.

For example: External Firewall -> Perform IPS Inspection on all Traffic, and Internal Firewall -> Protect Internal Hosts Only

Thanks in advance

[Timothy_Hall]

This setting is only relevant for an R77.30 or earlier gateway; using it in conjunction with IPS exceptions is a bit clumsy.  For an R80.10+ gateway you can explicitly define what IPS profile you want to apply to certain traffic right in your main TP policy; you can even have different IPS profiles of varying strictness applied to different types of traffic.

--
"IPS Immersion Training" Self-paced Video Class
Now Available at http://www.maxpowerfirewalls.com

[Di_Junior]

Thanks Tim, 

We are still using R77.30, but migrating to R80.20 soon.

[Di_Junior]

Thank you. I will update this thread after the implementation, I will sit with the other teams and discuss the possibilities that we have.

[Di_Junior]

Hi Tim

On your personal view, do you think that Check Point Maestro could be of help in this situation. In such a way that the firewall on both sites would be streched together with Maestro appearing as a single Firewall?

Another question is about CoreXL. I have a client who has a 20 cores Check Point firewall (all licensed), but the system has only 2 CoreXL cores, the other ones are SND. Is this a good scenario? if yes, why, if not why?

Thanks in advance

[Timothy_Hall]

I'd suggest starting a new thread for any future questions such as these, instead of stretching out this old thread.

1) I haven't done much with Maestro yet other than read the briefs/presentations.  Looks very cool for sure though!

2) I think you mean there are 2 SND/IRQ cores and 18 Firewall Worker cores on a 20-core firewall by default, and is that default split acceptable?  It will depend on how much traffic can be fully accelerated by SecureXL and how many high speed (10 Gbps+) LAN interfaces are present and how busy they are.  Since there are 4 SND/IRQ cores allocated byd efault for firewalls with more than 20 cores, I'd say in general adjusting to a 4/16 split on your 20-core firewall would probably be a reasonable starting point. 

If there actually is a split of 18/2, I've never seen anyone get that crazy with the number of SND/IRQ cores.  On R80.10 and earlier the locking and coordination overhead begins to exact a toll when there are more than 6 SND/IRQ instances, and the performance gains as each SND/IRQ core is added above 6 are not quite as linear, but doing so still improves performance if there is a high amount of fully-accelerated traffic.

[Di_Junior]

Hi Tim

Thanks for the suggestion, and I will proceed accordingly next time.

Yes, I meant 18/2 (SND/IRQ), I also cannot believe it myself, I am trying to find any reason why this is configured this way. Perhaps they had a reson to do so, which I fail to find any.

Regards