FWX_CACHE, ICMP and fw_allow_simultaneous_ping

[Luis_Miguel_Mig]

Conclusions

**************

fw_allow_simultaneous_ping set to 1 creates an entry in the fwx_cache for every icmp request - in my environment accounted for the 30% of the fwx_cache

Background

****************

My fwx_cache was full. The fwx_cache limit is 10k (as by default) and I have 3 cores so   my fwx_cache limit is 3(nr of cores ) X 10k = 30k.
Looking more in detail  at fwx_cache I could see that aprox 13k/30k=43% of the fwx_cache entries were generated by ICMP traffic while looking at the connections database 9k/29k=31% of the connections were icmp.

Digging deeper in the  fwx_cache I could see the top sources, the top destinations and the top conversations and none of them were hitting any NAT configuration (I checked the connections database, smartview or smartlog)

So I have aprox 43% of my fwx_cache taken by ICMP traffic that didn't have NAT configuration whatsoever.

This is just a test that I run to show the problem more clearly:

I pinged IP-A from IP-B

# ./showtable.sh all global list:10:sdfcrnlm:IP-B

****** connections.log ******
TOP SOURCES
     40 IP-A;  
TOP DESTINATIONS
     40 IP-B;  
TOP CONVERSATIONS
     40 IP-A;     IP-B;
TOP FLOWS (ICMP=1;TCP=6;UDP=11;ICMPv6=58;)
      1 IP-A;  IP-B;  20468;  icmp;  1;
      1 IP-A;  IP-B;  20467;  icmp;  1;
      1 IP-A;  IP-B;  20466;  icmp;  1;
      1 IP-A;  IP-B;  20465;  icmp;  1;
      1 IP-A;  IP-B;  20464;  icmp;  1;
      1 IP-A;  IP-B;  20463;  icmp;  1;
      1 IP-A;  IP-B;  20462;  icmp;  1;
      1 IP-A;  IP-B;  20461;  icmp;  1;
      1 IP-A;  IP-B;  20460;  icmp;  1;
      1 IP-A;  IP-B;  20459;  icmp;  1;
TOP RULES
     40 1;  
TOP NATS

****** fwx_cache.log ******
TOP SOURCES
    202 IP-A
TOP DESTINATIONS
    202 IP-B
TOP CONVERSATIONS
    202 IP-A     IP-B
TOP FLOWS (ICMP=1;TCP=6;UDP=11;ICMPv6=58;)
      1 IP-A      IP-B    20475   00000001
      1 IP-A      IP-B    20474   00000001
      1 IP-A      IP-B    20473   00000001
      1 IP-A      IP-B    20472   00000001
      1 IP-A      IP-B    20471   00000001
      1 IP-A      IP-B    20470   00000001
      1 IP-A      IP-B    20469   00000001
      1 IP-A      IP-B    20468   00000001
      1 IP-A      IP-B    20467   00000001
      1 IP-A      IP-B    20466   00000001

****** fwx_alloc.log ******
TOP SOURCES
TOP DESTINATIONS
TOP CONVERSATIONS
TOP FLOWS (ICMP=1;TCP=6;UDP=11;ICMPv6=58;)

So just a simple ping that didn't require any NAT (see database and fwx_alloc output)   generated 202 entries in my fwx_cache in a couple of minutes.

Why? It doesn't make any sense at all.

I noticed that one my checkpoint cluster didn't have this problem, so I compared the configuration and I found out that this checkpoint cluster had  fw_allow_simultaneous_ping set to 0.

So I set fw_allow_simultaneous_ping  to 0 in a couple of checkpoint gateways and I could see straight away how the fwx_cache freed  up of the ICMP entries.

I am wondering if this is a bug or a known behavior but anyway as a result of fw_allow_simultaneous_ping set to 1 more than 30% of my fwx_cache was taken by ICMP traffic that didn't require any NAT.

By the way I could share the script if there is interest

echo "****** Syntax ******************************************************************************************************"
echo "showtable.sh (TABLE:connections|fwx_cache|fwx_alloc|sam_blocked_ips|all) (CPU:global|core) (OPERATION:summary|list,<nr>,sdcfplm,<pattern>)"
echo "example1: showtable.sh all global summary"
echo "example2: showtable.sh connections core list"
echo "TABLE: connections, fwx_cache, fwx_alloc, sam_blocked_ips or all tables"
echo "CPU: global or core, data globally or per core"
echo "OPERATION: summary or list/detailed information; <nr> number of iles; FLAGS: s(source), d(destination), c(conversations), f(flow), p(protocol), l(local with the latest fetched data),m(filter pattern); <pattern to filter>"
echo "************************************************************************************************************************"

[Timothy_Hall]

fwx_cache is used to cache all NAT table policy lookups, not just ones that result in an actual NAT operation but also anti-NATs or the complete lack of a NAT rule for that traffic as well.  So traffic not subject to NAT at all will still take an entry in the cache, this is expected behavior because is saves a NAT policy lookup which is the whole point of the cache.  Every new connection when it starts must be checked for NAT, and fwx_cache is checked first.  ICMP is never templated nor accelerated by SecureXL, so ICMP will always go F2F with no templating so it doesn't surprise me that it is taking up lots of entries in fwx_cache which is part of F2F.

The fwx_cache can get "full" as you observed, but all that means is that there will be more lookups against the NAT policy than there otherwise would be.  Not the end of the world by any means.

Having fw_allow_simultaneous_ping set to 1 (default is 0) causes extra stateful tracking information to be created for "stateless" ICMP traffic, so the cluster can keep track of simultaneous pings against the CIP and the dedicated IP addresses.   sk26874: Cannot simultaneously ping Virtual IP address of the cluster and IP addresses of physical i... has a nice explanation of this mechanism.  Not a surprise to me that fwx_cache is included in this.

Hopefully it goes without saying that fw_allow_simultaneous_ping should NOT be set to 1 on a non-clustered gateway, as it causes extra overhead for no good benefit.

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com.

[Luis_Miguel_Mig]

Thanks Tim. Very interesting to see that there could be entries for the anti-NATS. I didn't know that.

But this is what I see in my environment:

- if fw_allow_simultaneous_ping = 0, all the entries in the fwx_cache are expected, I mean there is explicit NAT configuration for them.

-if  fw_allow_simultaneous_ping =1, I can see a fwx_cache entry for any ICMP traffic and not only for the CLUSTER CIP or the member dedicated IP address.

So when fw_allow_simultaneous_ping = 1, I would expect to see fwx_cache entries only for the CIP and the dedicated ips and not for all ICMP traffic.

And if those fwx_cache entries are expected when fw_allow_simultaneous_ping is set 1, they should also be there when it is set to 0, right?

I mean  if there was something else  causing the fwx_cache entries for all ICMP traffic I would expect to see those entries for fw_allow_simultaneous_ping set to 0 and 1.

[Timothy_Hall]

My understanding is that setting fw_allow_simultaneous_ping to 1 adds the extra "stateful" info to *all* ICMP "connections" being tracked in the firewall's state tables, not just those going to the CIP and dedicated IP addresses.  Because there is now more uniquely identifying information being kept about ICMP, it creates many more separate entries in the various state tables because they can now be more easily distinguished from each other. 

When fw_allow_simultaneous_ping is set to zero, many ICMP "connections" get collapsed/matched against fewer state table entries, but this can cause the effect that fw_allow_simultaneous_ping was trying to counter in the first place.  I'm sure someone at Check Point with access to the code for this will be happy to correct me if I'm wrong.  🙂

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com.

[Luis_Miguel_Mig]

It would be great if someone from Checkpoint could confirm.

But I thing that I got t your point,  it may be difficult to add the extra info needed only for a few ips. The implementation may be easier for all ICMP traffic.

Anyway, I will see if I get any benefit in terms of performance now that my fwx_cache is not full any more.

As you mentioned, the benefit may not be relevant.