NAT performance (Hide NAT vs Static NAT)

[Graham1]

I will try to keep this as brief as possible while also giving the pertinent information.

We have a HA cluster with ISP device IPs.  We proxy ARP additional customer IPs (which we have had for 20 years) to this cluster.
We static NAT specific networks to these IPs as per historical business rules.  IE keep the IPs you can get.

When I have network nodes that use the Hide NAT method it is considerably quicker than any static NAT method, which I can assume is related to the Proxy ARP.  I am using the same 3 devices and changing their NAT method each time to eliminate some variables.  The access rule order is not changing during my testing.

My cluster is a pair of 6700s using 10GbE interface for ingress and egress.  I found an article about peak 

[Expert@fw-ext-01:0]# fw tab -t fwx_cache -s
HOST NAME ID #VALS #PEAK #SLINKS
localhost fwx_cache 8116 64374 68428 0


Can anyone point my to documentation that can explain away this difference to Management?

[PhoneBoy]

Please elaborate how HIDE NAT is "considerably quicker" than STATIC NAT.
Some packet captures might also provide clues as to what's actually going on here.
Also what version/JHF is this relevant to?

[the_rock]

Personally, I never heard one type of nat be faster than the other, thats news to me.

Andy

[Timothy_Hall]

Define "quicker" for Hide NAT.  Overall throughput?  Connection establishment time?  Connection via static NAT is established but stops working for brief periods?

My guess is you are using the gateway's address for your Hide NATs which of course the gateway will answer ARP requests for, but for your static NATs you probably have manual proxy ARPs defined.  Sounds like something is amiss with your proxy ARPs and they are not being learned promptly or being kept cached by the surrounding devices. If you suspect intermittent ARP problems consider arpwatch, the clish command is: set ip-conflicts-monitor

How many static proxy ARPs are we talking here?  Dozens?  Hundreds?  Thousands?  Your firewall's default route is explicitly defined to a next-hop IP address and not the just the interface itself right?  If defined to the interface that will cause an ARP table overflow which will cause a "rolling outage" behavior that will severely impact performance.  

The number of cached NAT entries in fwx_cache is unlikely to be related to your problem, unless the slowness is only happening at initial connection establishment time and the gateway's CPU is heavily overloaded.  Maybe.

[Graham1]

@the_rock  & @PhoneBoy  I am gathering the stats and captures shortly.  Given this is for a Guest DMZ network it isn't the greatest of urgencies.  

R81.20 JHF Take 76

@Timothy_Hall Yes the hide NAT is using the gateway's interface IP and yes the Proxy ARP entries are manual (8 of them)
I have found the affected route to the Guest network do in fact point to an interface and not an IP.  Default route points to ISP IP on the gateway's network.  I will investigate ARP table overflow.  It does appear to be overall throughput, not just initial connections.

[PhoneBoy]

The only routes that should ever point to an interface are directly connected networks on the same subnet.
Otherwise, you might get the "rolling outage" situation that @Timothy_Hall described, which I accidentally created once. 🙂

[the_rock]

No rush mate...when you got it, just send them over and we will review.

Andy

[Graham1 · Accepted solution]

Hey All.  
I figured this out and will give a summary since it has a bit of history.  Replying for future generations  🙂

Background:

1. We purchased an additional 6700 to create a HA cluster and traded in the 3600
2. Each device had its own ISP, and both had to reside on the cluster.  ISP-6700 for production traffic and ISP-3600 for Guest traffic.  Done this way to guarantee performance for business operations.
3. When planning, I put in a support request asking if PBR requires ISP Redundancy and was told yes
4. Implemented ISP redundancy with ISP-3600 as the secondary ISP and manual static NAT for the Guest network object to ISP-3600 (secondary ISP).
5. Held off the PBR due to sporadic performance issue that started this thread.  FYI Support wasn't able to nail down the root cause either, but I think it was a combination of ISP redundancy and the manual static NAT

The FIX:
Disable ISP redundancy and configure PBR

I dived into documentation and found a feature conflict between ISP Redundancy and PBR: Policy-Based Routing and Application-Based Routing in Gaia (checkpoint.com)  

@Mike_T thanks for this post, it really helped!  Solved: Route specific subnet out second ISP interface - Check Point CheckMates


Thanks for reading..

[the_rock]

Excellent...thanks for sharing!

Andy

[the_rock]

For the context, below sk states pbr is NOT supported with ispr.

Andy

https://support.checkpoint.com/results/sk/sk167135