NAT Exhaustion - Hide NAT failures

[Muazzam]

Environment:

MDS R80.20, Gateway R77.30 T216, Hardware 13800

Cores are not overloaded, stays around 30-60%

We see a lot a "hide NAT failure" messages in firewall logs. User reports latency and page not found at that time. Adding additional NAT addresses on the top of existing hide NAT addresses resolves the issue but my concern is the output of these commands that I am using to check the number of times each of my hide NAT is used.

[Expert@R77.30GTW]# fw tab -u -t connections | grep -ci bbxxxx0a
165032
[Expert@R77.30GTW]# fw tab -u -t connections | grep -ci bbxxxx0b
184938
[Expert@R77.30GTW]# fw tab -u -t connections | grep -ci bbxxxx0c
105793

Note: No errors messages or user complains at this point. Also note that these numbers have not changed uch in last few days, since the time we had the issues.

Are these numbers look real? If we divide the output by 2, still we are talking about 50K to 90K range that is theoretically not possible.

Is it possible that some connections got stuck, not getting released or something?

[Timothy_Hall]

More than 50k connections can utilize the same single Hide NAT address as long as the destination IP addresses for the connections are unique; this was recently clarified by Check Point and has actually been the case for some time.  So the 50k Hide NAT limit only applies if all those connections are to the same destination IP address.

On a 20-core box like a 13800 assuming the default 2/18 split, it is much more likely that the cause of the NAT hide failures is the static allocation of Hide NAT ports among the 18 worker cores you have under version R77.30.  The available 50k port range for each Hide NAT address is getting split 18 ways down to only 2,777 available ports per worker core.  This situation was covered on pages 154-156 of my "Max Power" book, please see the following SK to enable dynamic allocation of NAT ports among your Firewall Worker cores: sk103656: Dynamic NAT port allocation feature

I haven't seen any issues enabling this as long as your gateway is using a reasonably recent R77.30 Jumbo HFA.  On R80.10+ gateways, dynamic Hide NAT port allocation is automatically enabled by default for systems with 6 or more Firewall Worker cores defined.

[Muazzam]

Just checked this:

[Expert@R7730GTW]# fw ctl get int fwx_nat_dynamic_port_allocation
fwx_nat_dynamic_port_allocation = 1

#fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 19 | 46202 | 59712
1 | Yes | 18 | 51209 | 78751
2 | Yes | 17 | 42828 | 59348
3 | Yes | 16 | 59289 | 73150
4 | Yes | 15 | 60773 | 76596
5 | Yes | 14 | 49899 | 66817
6 | Yes | 13 | 49788 | 65572
7 | Yes | 12 | 49678 | 66009
8 | Yes | 11 | 63712 | 81640
9 | Yes | 10 | 51146 | 74882
10 | Yes | 9 | 41134 | 59263
11 | Yes | 8 | 47247 | 61053

There are no issues after we added the new NAT addresses. I was concerned about the output / number of connections per hide NAT IP address, if this make sense?

[Timothy_Hall · Accepted solution]

OK glad to see dynamic allocation is enabled.

I assume that your hide failures are not involving ports 60,001 through 65,536, these are "Extra/Global" NAT operations that are not supported across multiple CoreXL instances and have much smaller NAT port ranges for various operations.  You can read about Extra/Global NATs here: sk69480: 'NAT Hide failure - there are currently no available ports for hide operation' log appears ...

I assume you are aware of how to set up "Many to Fewer" Hide NATs as described below which is bit more convenient that defining multiple "Many to One" Hide NATs:

https://community.checkpoint.com/t5/General-Topics/R80-10-Hide-behind-many-question/m-p/3828?search-...

sk142833: How to create manual NAT rules in Many-To-Few mode

Beyond that you'll have to look into the uniqueness of destination IPs that your users are hitting, you do seem to have ~820,000 connections being handled by your firewall so I guess it's possible that there could be 50k concurrent connections from your user population all trying to hit the same destination IP via the same Hide NAT, in which case the only solution is doing more "Many to One" Hide NATs or implementing "Many to Fewer" Hide NATs as described above.

Edit: You can investigate the top 10 destination IP addresses per Hide NAT address (bbxxxx0a in this example) by doing something like this:

fw tab -u -t connections | grep -ci bbxxxx0a | awk '{ print $4 }' | sort -n | uniq -c | sort -nr | head -10