Get number of current NAT sessions in R80.30

[johnnyringo]

Seems like a basic question, but how can I get the number of current NAT sessions on an R80.30 gateway?

In R80.40, I've found two ways to do it: 

1) From CLI, run 'cpview'.  Navigate to Advanced -> NAT

2) SNMP walk OID tree 1.3.6.1.4.1.2620.1.56.1301

But neither work in R80.30

[Timothy_Hall]

The closest you can get in R80.30 and earlier is poking the fwx_nat_dynamic_port_allocation_print_stats kernel variable to 1 with fw ctl set int and it will dump NAT statistics like the ones below.  Note that this is not nearly as good as the stats provided by the cpview NAT screen in R80.40, that is because GNAT replaced the original Dynamic NAT port allocation feature in R80.40, and the original Dynamic NAT port allocation feature in R80.30 and earlier could not provide very good statistics.

--------------------------------
IPs in global port quota: X
IPs in local port quota: X
connections table entries count: X
fwx_alloc table entries count: X
Local allocation requests: X
Local free requests: X
Local failed allocations: X
Local successful allocations: X
Local retries for already used ports: X
Quota requests (count, amount): X, Y
Quota release for 4X (count, amount): X, Y
Quota release for time (count, amount): Y
--------------------------------

See these SKs for further reading, especially the second one which is relatively new.

sk103656: Dynamic NAT port allocation feature

sk172933: NAT FAQ

[johnnyringo]

Ahh thanks, I was afraid of that. 

As you've probably guessed, what brings on this question is we're running low on NAT hide ports.  These clusters are in Google Cloud, so upgrading to R80.40 requires a complete re-launch, so I'm likely going to just jump to R81. 

[Timothy_Hall]

I assume you are referring to the 50k concurrent connection limit per destination IP when hide NAT is in use?  If so you can increase the 50k limit by what I call a "many to fewer" Hide NAT to a pool of routable addresses (assuming you have some extras available).  Please see the following thread; this topic is also covered in the third edition of my book, pages 303-306.  

R80.10 - Hide behind many question

Also you may want to check out the NAT analysis tools written by @Kaspars_Zibarts which may be helpful in determining how close you are to running out of source ports and identifying the primary offenders:

NAT table (fwx_alloc) top users

NAT table (fwx_alloc) specific NAT IP address analysis

[johnnyringo]

Hmmm I see a limit of 16k.  The traffic flow is Internet -> GCP External Load Balanacer -> CheckPoint eth0 (external) interface -> port 25080 -> Internal IP address.  The CheckPoint performs a NAT hide due to GCP routing load balancer IPs internally. 

cpview on and R80.40 lab cluster shows this:

| CPVIEW.Advanced.NAT                                                                                                                                          21May2021 14:39:22 |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Overview SysInfo Network CPU I/O Software-blades Hardware-Health Advanced                                                                                                       |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| CPU-Profiler Memory Network SecureXL ClusterXL CoreXL PrioQ Streaming NAT MUX Routed RAD Conn-Tracker UP HTTP-Parser SSH-Parser CPAQ                                            |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Pool-IPv4 Pool-IPv6                                                                                                                                                             |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| General Statistics:                                                                                                                                                             |
|                                                                                                                                                                                 |
| Concurrent connections                2                                                                                                                                         |
| Connections session rate              0                                                                                                                                         |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| High port:                                                                                                                                                                      |
|                                                                                                                                                                                 |
| Instance  Hide IP             Dst IP                  Dport     Proto          Port Usage  Capacity      Used                                                                   |
| 0         10.111.222.12       10.111.222.39               0         6                   1    16,533        0%                                                                   |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Low port:                                                                                                                                                                       |
|                                                                                                                                                                                 |
| Instance  Hide IP   Dst IP        Dport     Proto          Port Usage  Capacity      Used                                                                                       |
| -         -         -                 -         -                   -         -         -                                                                                       |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Extra port:                                                                                                                                                                     |
|                                                                                                                                                                                 |
| Instance  Hide IP   Dst IP        Dport     Proto          Port Usage  Capacity      Used                                                                                       |
| -         -         -                 -         -                   -         -         -                                                                                       |
|   

[Timothy_Hall]

Let me guess, you have a 4-core firewall with a 1/3 split.  When there are 5 or less worker cores the available hide NAT ports are statically assigned to each worker instance which is why instance 0 in your case only has a capacity of 16,533 (16,533 * 3 = ~50k).  This limitation was called out in the first edition of my book which covered version R77.20, but then Check Point introduced the sk103656: Dynamic NAT port allocation feature.  You have basically two options to increase capacity:

1) Set up a hide behind many with a pool of three NAT addresses, that will give each of your three worker instances ~50k NAT capacity.

2) Forcibly enable the Dynamic NAT port allocation feature by setting fwx_nat_dynamic_port_allocation to 2.  I don't recall ever doing this myself on a firewall with 5 or less firewall workers, so I'm not 100% sure if it is supported but it should be.

[johnnyringo]

Yes, HA deployments in public cloud are 2 x 4 vCPU with the 3/1 data vs control plane split.  We've been downsizing to 2 vCPU instances though since the throughput requirements are low and connections is only  a couple thousand.   Knowing this, I will look to upsize them back to 4 vCPU.

Have to say, I am quite puzzled why CheckPoint would correlate the number of TCP/UDP ports to vCPU count.  That seems like a legacy architectural decision that should be revisited.  Modern CPUs are certainly cable of handling more than 16533 connections per core without issue.

---

1) Set up a hide behind many with a pool of three NAT addresses, that will give each of your three worker instances ~50k NAT capacity.

---

I assume this would also 3x the 16533 in the 2 vCPU instances as well and serve as an alternate fix to paying 2x in compute and licensing costs.    GCP does support additional IP ranges assigned to an instance, and the CheckPoint dynamic object used to do the NAT hide could be changed to match that range, so this is possible. 

The better option for us though is to use the CheckPoint as an L7 proxy, but this requires an upgrade to R81 so that TLS 1.3 and HTTP2 can be supported.  We're on that path anyway, I'm just trying to get a firmer understanding of the limits we're hitting under the existing setup.  

[johnnyringo]

To respond to my original question, I did find a simple command: fw tab -t fwx_alloc -s

> fw tab -t fwx_alloc -s
HOST                  NAME                                ID #VALS #PEAK #SLINKS
localhost             fwx_alloc                         8187   362   845       0

[the_rock]

You are correct, that is actually for nat entries.