R80.x Performance Tuning Tip - Connection Table

[HeikoAnkenbrand]

What is a connection kernel table

One of the most important tables in the Check Point firewall is the "connection kernel table". Connections kernel table contains the specific information about the current connections (source, destination, protocol, timeout, etc etc etc). The direction of the connection is set by the first packet of the connection even though the connection may be bi-directional in reality.

In principle, the following keys are used:

•       Direction of the connection (0 = inbound, 1 = outbound)
•       Connection key
•       Connection type
•       Connection flags
•       Rule number in SmartDashboard, to which the connection was matched
•       An index of an INSPECT handler function that is executed on every packet that belong to the connection
•       Unique 128-bit connection identifier.
•       Anti-Spoofing cache
•       Set of per-connection bits
•       Connection modules' kernel buffers
•       Time to Live / Timeout

The information in the connections table is stored in the following form:

<direction,5-tuple-key;r_ctype,r_cflags,rule,service_id,handler,uuid1,uuid2,uuid3,uuid4,ifncin,ifncout,ifnsin,ifnsout,bits1,bits2,connection_module_kbufs@ttl/timeout>

Detailed explanation about each parameter can you find in the following SK Connections Table Format.

# fw tab -t connections -f -u

It is possible to manually delete an entry from the connection table with the following command.

You can find more about the topic in the following SK "How to manually delete an entry from the Connections Table".  Delete the desired connection entry:

# fw tab -t connections -x -e DIRECTION,SOURCE_IP,SOURCE_PORT,DEST_IP,DEST_PORT,PROTOCOL

Attention!

•       All subsequent packets of the deleted connection will be dropped as Out-of-State. The connection should be re-established.
•       This procedure will not remove entries from NAT translation tables.

Intressant are also the installed connection modules. Attention these are often confused with the "fw monitor chain modules". The modules have nothing to do with each other.  These connection modules can be displayed with the following command:

# fw ctl conn -a

Chapter

More interesting articles:

- R80.x Architecture and Performance Tuning - Link Collection
- Article list (Heiko Ankenbrand)

Tuning connection table

The goal is always to reduce the number of connections in the connection table. The following tips will give you some examples. First of all you have to see what your connetion table looks like at the moment. Everything else can be found in tip one.

What I don't want to enter here are different paths through the firewall in conjunction with SecureXL. More informations see here "R80.x Security Gateway Architecture (Logical Packet Flow)"

•       Fast path (Accelerated Path)
•       Medium path (PXL)
•       Slow path (F2F)

Tip 1

From my point of view all systems should be set maximum concurrent connections  to "automatically" (see picture).

Alternatively, you can toggle it to manually here. I wouldn't advise it, though. In most cases this is also set as default by older versions. If you use this, then please use it described below.

This command shows the current and maximum number of connections.

# fw tab -t connections -s

This command shows the maximum number of possible connections:

# fw tab -t connections | grep limit | grep -v Kernel | grep -v connections | grep -oP '(?<=limit ).*'

In this example there are 23876 peek concurrent connections and 14056 concurrent connections while the default limit is 25000. The connection table limit should be increased to ensure uninterrupted operation. However, it should be noted here that the increase also uses resources (RAM). I would carefully double the connection limit in this case to 50000. Locate the maximum concurrent connections setting for the firewall (normally found in the object’s properties) and increase the value. The increase should be done gradually and with care as it will also increase the memory usage of the firewall.

Attention! This changes the behavior of the firewall. Therefore please handle with caution.

TIP 2

Packets are dropped on security gateway because maximal number of symbolic links for connections was reached in connection table. What is the consequence! Very low traffic on the security gateway. How do I analyze the problem? The output of 'fw tab -t connections -s' command shows that the ratio of #SLINKS / #VALS is greater than/equal to 5. What can I do? Follow sk65463 

TIP 3

Turn on Aggressive Aging to have connections removed as quick as possible.

Aggressive aging is activated in IPS profile, or new connections may be dropped for the reason that the connections table is full when a given CoreXL firewall instance has far fewer connection entries than the connections table limit, or the 80% threshold to activate aggressive aging as seen in the output of 'fw ctl multik stat' command.

The aggressive aging timeout values must be lower than the stateful inspection default for  TCP session timeout (3600). The stateful inspection default session timeout values can befound in: Global Properties > Stateful Inspection > TCP session timeout 

Check Aggressive Aging status:

:# fw ctl pstat | grep Agg

TIP 4

When you decrease the start timeout and end timeout, the connections will removed as quick as possible if the connection is no longer used.

In the SmartDashboard go to „Policy->Global Properties“ and in the Stateful Inspection tab reduce the the following session timer (see picture):

•       TCP start timeout
•       TCP end timeout
•       UDP virtual session timeout
•       ICMP virtual session timeout
•       SCTP start timeout
•       SCTP end timeout
•       other IP protocols virtual session timeout

Please refer to the firewall’s user manual for more information on what the session timeout is.

Attention! This changes the behavior of the firewall. Therefore please handle with caution.

With the following command you can find the number of active sessions for „TCP start timeout“ in the connections table. If you change the value in "grep" you can also show other session timeouts!

# fw tab -t connections -u -f | grep "Expires:" |grep "/25;" |wc -l

Note: A high number of „TCP start timeout“ could also indicate a DDoS SYN flood attack. In the area of Distributed Denial of Service (DDoS), TCP SYN flooding was one of the first attack vectors that was encountered and it has remained significant even now. The TCP handshake process, which starts with a TCP [SYN] packet, requires that a server allocate a transmission control block (TCB) for incomplete and half-open connections. More see here "Best Practices - DDoS attacks on Check Point Security Gateway".

TIP 5

Under R80.20 there is a new command that in conjunction with the "Dynamic Dispatcher" displays connections with a high load. That's where I mention that CLI command here although it doesn't quite match the theme. This command shows the table with heavy connections (that consume the most CPU resources) in the CoreXL Dynamic Dispatcher. For more information about the CoreXL Dynamic Dispatcher, see sk105261.

CoreXL suspects that a connection is "heavy" if it meets these conditions:

•       Security Gateway detected the suspected connection during the last 24 hours
•       The suspected connection lasts more than 10 seconds
•       CoreXL FW instance that processes this connection causes a CPU load of over 60%
•       The suspected connection utilizes more than 50% of the total work the applicable CoreXL FW instance does

Here is an example:

# fw ctl multik print_heavy_conn

TIP 6

An important point in tuning is the order of the rules. Normally the "rule counter" helps here. Rules that are often used should be at the beginning of the ruleset.

It can also be helpful to view certain connections in real time. For this I have published some useful commands here. The commands are for version R80.20.

1) Find all rules to an IP address in the connections table (Change the IP😞

# fw tab -t connections -u -f | grep 10.1.2.81 | grep Rule | awk '{split($0,a,";"); print a[14];}' |sort -ng |uniq

2) Find all rules to an IP address in the connections table with IP and Port informations (Change the IP😞

# fw tab -t connections -u -f | grep 10.1.2.81 | grep Rule | awk '{split($0,a,";"); print a[14]," ",a[10]," ",a[6]," ",a[8]," ",a[9];}' |sort -ng

TIP 7

Another important point is the session timeout for a service. For example, the default session timeout for UDP is set to 40 seconds (TCP to 3600 seconds). This means, for example, that each DNS request is kept open for a maximum of 40 seconds after the last package. That's a bit much from my point of view! Therefore it is possible to change the virtuall session timeouts in the service. I always test here with about 20 seconds (see picture) and adjust it a little bit upwards or downwards. So the DNS requests do not remain so long in the connetion table. This can also be applied to other services.

References

Connections Table Format 
How to manually delete an entry from the Connections Table 
Packets are dropped on Security Gateway because maximal number of Symbolic Links for connections was... 
Best Practices - DDoS attacks on Check Point Security Gateway 
How Connections Table limit capacity behaves in CoreXL 
Performance Tuning R80.10 Administratio Guide
Performance Tuning R80.20 Administration Guide
Best Practices - Security Gateway Performance 

[AlekseiShelepov]

Tip 1

I find this advice quite strange.

As far as I know, all fresh installations of Gaia by default have "Automatically" parameter set for maximum number of connections. If you create a new cluster/gateway object with Gaia, it uses automatic allocation by default. But this limit of 25000 connections was set on IPSO by default and was carried out to Gaia after upgrades. Recently I replaced several clusters of appliances with IPSO and had to manually change the limit in gateway objects.

If you can see the reached maximum (peak) with # fw tab -t connections -s, then this gateway is in production already. What if we have peaks only on Sundays, with some automatic services running? We put the gateway in the network, increase number of connections x2-x3, but on Sunday it will be x5 connections. I can imagine what would happen for example with gateways in some online shops during sales days.

I believe that this limit of 25000 was not related to hardware itslef. Do you recommend to use 25000 connections as a start point for 5000 appliances as for 23000 appliances?

Timothy Hall‌, could you please comment on that? You wrote in the book that it is recommended to set this setting to automatic allocation on new gateways and OS and not try to manually set it.

Also, "Intressant" and "Reverenz" are German words.

[kaj0103]

Hi

Referring to Tip 1, the automatic connection table. 

We currently have a connection limit of 110000 set. It came out of a Spirent test and we found it a good cap.

I've recently had a look at the values on the connection tables and the fw ctl pstat output does not marry up with the fw tab -t connections -s output... see attached image.

The reason that I am looking at this is that we have been told to expect a maximum of another 50k new connections for O365. I'm sure it wont go anywhere near this but none-the-less, preparing for the worst case. 

I just wanted to ask if the "automatic" is a dynamic value that adjusts as needed or if it sets once and also are the two outputs above supposed to be of the same table or answering the same question?

Can anyone tell me what "anticipated means?"

Thanks

[Timothy_Hall]

Check Point's code on the 32-bit SecurePlatform and IPSO operating systems did not support a setting of "Automatically" for connection table size, 25k was an arbitrary limit from a very long time ago and not related to the underlying hardware type or amount of RAM.  For any new firewall object created that specifies Gaia as the OS, the setting will be "Automatically" by default.  However the point I made in my book is that if a firewall object was initially created with an OS of SecurePlatform/IPSO then changed/upgraded to Gaia later, the setting will remain at Manually and not be changed to "Automatically" just because the OS was set to Gaia on the firewall object.  So definitely a good idea to check this setting and make sure it is set to Automatically on all your firewalls using Gaia (which should hopefully be just about all of them these days).

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[HeikoAnkenbrand]

HiAleksei Shelepov,

thx, I totally agree with you.and Timothy.

From my point of view you should only set "automatically" alwys.

As far as I remember this is the case with every R7X version. With old R65 versions here 25000 was set.

To your second point!

With 23K Appliancen you should set more than 25K. Here it depends also on the environment. Also here I would set always to "automatically".

If a customer makes the setting fixed, he should adjust the values carefully. I have already seen environments in which it set to 2500000. And everyone was surprised that nothing worked anymore. Here I could write an article "The Best Check Point Configuration Errors"

I'll add the following:
From my point of view all systems should be set to "automatically" always.

Regards

Heiko

[AlekseiShelepov]

I can see that you updated your advice in Tip 1, that's good.

But then my next question is why you need to leave this part about manual configuration of maximum connections at all? Maybe just leave the recommendation to set it on automatic and how to check number of connections? I believe that you're talking about modern appliances, OS, and versions here. Or could you explain in what cases someone should use manual configuration then?

[HeikoAnkenbrand]

Hi Aleksei,

I don't see any need for it anymore. But some customers just don't want to change it. That's why I mentioned it as a precaution.

Please start a vote if we need the manual setting.

Regards

Heiko

[Maria_Pologova]

TIP 1 

I must disagree with your point of view here, as you don't take into consideration VSX environment. Quote from sk39555

On Virtual Systems the value should be increased only if you know that a specific VS should handle a large load of concurrent connections.

It is not recommended to change this setting to a high value for VSs that are not planned to process a large amount of connections in the near future, because a high value of "Maximum concurrent connections" consumes memory, even if no entries are recorded in the connections table.

Second point is that if you have R77.30 Management, before setting up Automatic calculation of connections, it is advisable to ensure that you have IPS profile assigned to the device, so Aggressive aging would work in case of low free memory - link. I do understand that this is not relevant to your topic, since you are talking about R80.x, however I'll just leave it here.   

TIP 6 

We need to mention that fw tab -t connections -u -f takes time and a bit CPU intensive. Use with caution.

I'd mention cpstat_blades command, which provides you top 5 rules based on hits. 

P.S. Reverenz? Are you sure? 

[HeikoAnkenbrand]

Yes, I did not describe VSX here.

Therefore good note!

THX

Heiko

[Martin_Arms]

Nice info.

[Jason_Carrillo]

Would you recommend making any changes to the default timeout, particularly for the TCP start timeout?

Since we've moved more of our clients to wireless we seem to be seeing instances where this may be getting triggered. The jury's still out as to whether this is causing an issue as the client should just try and reconnect.