S7PAC - Super Seven Performance Assessment Commands

[RickHoppe]

Did you watch the excellent TechTalk 'Security Gateway Performance Optimization with Timothy Hall'?

Link to the video: https://community.checkpoint.com/videos/7987

Link to the slides: https://community.checkpoint.com//docs/DOC-3169

The presentation ended with: "As Check Point administrators, you should always quickly run the “Super Seven” commands whenever you access a firewall to check performance"

So I've created a simple script that executes all super seven performance assessment commands that were mentioned by Timothy Hall . I called the script s7pac.

I'm thankful that this script is being mentioned in the third edition of the book Max Power 2020: Check Point Firewall Performance Optimization.

@Danny created a SmartConsole extension of s7pac. See this thread.

To install the script on a Security Gateway, simply run the following command in Expert mode:

curl_cli -o /usr/bin/s7pac tools.checkpoint.engineer/s7pac && chmod +x /usr/bin/s7pac

Or download s7pac here, copy it to your Security Gateway and make it executable.

GitHub repository: https://github.com/Rick-Hoppe/s7pac

Changelog

0.1 First version

0.2 - Added check to make sure script is started on a Security Gateway

      - Output to file in home-directory (format is s7pac_$(hostname)_$(date +%Y-%m-%d-%H%M).log)

      - Include page numbers of the book Max Power: Check Point Firewall Performance Optimization - Second Edition

0.3 Added RX-DRP check (thanks to @Ofir_Shikolski  for the code)

0.4 Updated with R80.30 references to the book Max Power 2020

0.5 Added ethtool -S output (thanks to @Andreas_Zentsch for the tip)

[JozkoMrkvicka]

Regarding RX/TX errors, output of the command "netstat -ni" can be modified to show only interfaces if RX/TX errors > 0 .

The same logic for SecureXL.

[RickHoppe]

That should be possible indeed. Unfortunately I'm not a scripting guru so I did not add intelligence to the script. If you have tips how to achieve this with the output of netstat -ni that will be really appreciated.

[Ofir_Shikolski]

Amazing one.

one question - why not using cpview? it contains all steps and even more

[RickHoppe]

Thanks.

While the CPView Utility can show you a lot of information while you browse through the various menus, this script merely focusses on the interesting stuff for SecureXL and CoreXL in just one “click” and especially when you need to check the performance of the firewall.

I’d recommend to use the various scripts out there like ccc, healthcheck  and hopefully max  soon too.

[Ofir_Shikolski]

try to add it to command5:

detect_rx_drops() {
(renice -20 $BASHPID > /dev/null 2>&1
for DEV in `ifconfig | grep -ie "^eth" | tr ":" " " | awk '{ print $1 }' | sed -e 's/^[ \t]*//' | sed '/^$/d'`
do
R1=`netstat -ni |grep -w "$DEV"| grep -v "lo" |awk '{ print $6 }' | tail -1 | grep -v "RX-DRP" | sed -e 's/^[ \t]*//' | sed '/^$/d'`
sleep 0.5
R2=`netstat -ni |grep -w "$DEV"| grep -v "lo" |awk '{ print $6 }' | tail -1 | grep -v "RX-DRP" | sed -e 's/^[ \t]*//' | sed '/^$/d'`
RXPPS=`expr $R2 - $R1`
if [ "$R2" -gt "$R1" ]
then
echo "interface $DEV: There are rx drops"
else
echo "interface $DEV: no RX drops"
fi
done
renice 0 $BASHPID > /dev/null 2>&1)
}

[RickHoppe]

Thanks! I will integrate it in the next version.

[RickHoppe]

Version 0.3 released and updated with RX-DRP check. Thanks again!

[Ofir_Shikolski]

Thanks  

Do you think that it also be good to add these 2 commands only for R80.20?

R80.20 - new interesting commands 

# fw ctl multik utilize   > shows the CoreXL queue utilization for each CoreXL FW instance

#  fw ctl multik print_heavy_conn   > shows the table with heavy connections

[RickHoppe]

Currently thinking about it how to integrate those new commands in a script that was originally based on a presentation with 7 commands :-).

[Susana_Perez-He]

Thanks Rick, Tim, all for this thread, it's being very helpful for me right now!😊

[RickHoppe]

s7pac has been updated to version 0.4 with references to the third editon of Max Power 2020: Check Point Firewall Performance Optimization when running this script on R80.30. 

[Andreas_Zentsch]

Hi Rick, there is an error in your script. The dropped packets under ifconfig or netstat -ni are increasing because the linux received pakets which can not be used liek spanning tree or other broadcasts. Thats why they are bein dropped. If you want to see buffer issues you can only use the command ethtool -S. If that states 0 everything is ok. Please check also this article: https://access.redhat.com/solutions/504293 for further details. 

So this result in your script is wrong: interface eth6.2079: RX-DRP percentage is 0.34755% (must be <0.1%)

[RickHoppe]

Hi Andreas, what exactly do you mean? Do you believe the calculation is wrong in my script or the use of the command netstat -ni? One of the performance goals is to have a RX-RDP rate <0.1% on all interfaces. The goal is not 0.

The netstat -ni command is one of the Super Seven Performance Assessment Commands as described by @Timothy_Hall in various presentations and his books. My script refer to his books where you can find more information about the commands and results that are given back to you. One of the subsequent commands Tim explains in his books when an interface is showing an RX-RDP rate in excess of 0.1% is actually ethtool -S.

[Timothy_Hall]

Hi Rick,

What I believe Andreas means is that not all RX-DRPs are frames discarded due to a lack of ring buffer space.  This changed substantially in Gaia 3.10 when the NIC drivers were updated.  In the past an RX-DRP was almost always a "rx_missed_errors" (or possibly rx_fifo_errors rx_no_buffer_count) as shown by ethtool -S, which indicates a ring buffer slot was not available and the frame was lost.  However in Gaia 3.10 if a frame arrives with an EtherType that has no "registered receiver" (in other words there is nothing present to process that specific EtherType), RX-DRP gets incremented even though it was not a buffer miss.  Unfortunately there is no counter incremented under ethtool -S when this happens.  You'll see this happen with BPDUs/STP, IPv6 and other protocols.

I did see this to some degree in Gaia 2.6.18 (and mentioned it in the third edition of my book, pages 208-210) but it seems to happen much more often under Gaia 3.10.  The classic way to verify this as mentioned in my book is to note the RX-DRPs accumulating on that interface then crank up tcpdump on that interface which places it in promiscuous mode.  If the constant RX-DRPs cease for as long as tcpdump is running (because it is a registered receiver for everything), that is the issue.

So Rick what you might want to do is in your script look at the rx_missed_errors/rx_fifo_errors/rx_no_buffer_count counters under ethtool -S instead of just RX-DRP when making a determination about whether interface buffer miss (RX-DRP) percentage is >0.1%.

[RickHoppe]

Hi Tim,

Thanks for the explanation. I will add the rx_missed_errors/rx_fifo_errors/rx_no_buffer_count counters to the script and they will be shown for each interface regardless of whether RX-DRP percentage is >0.1% or not. The user will also be informed now that this is only calculated for each interface over the past 0.5 seconds. So it will still calculate using netstat -ni but additionaly shows these counters too for further investigation.

Thanks @Andreas_Zentsch for the tip.

[RickHoppe]

Version 0.5 has now been released containing the rx_missed_errors/rx_fifo_errors/rx_no_buffer_count counters from ethtool -S.

[Andreas_Zentsch]

Hi Rick,

sorry but there are still problems with your script.

Your script requests the variables: RXDRP1, RXDRP2 and RXOK. Then you still do the calculation: if [ "$RXDRP2" -gt "$RXDRP1" ]. Which is not correct as mentioned above. This needs to be removed completly for current versions. 

Instead you only need to caclulate with "ethtool -S". But this works only for the interface not for VLANs. So you need to fix the loop as follow first:

detect_rx_drops() {
(renice -20 $BASHPID > /dev/null 2>&1
for DEV in `fw ctl iflist | grep -v "\." | awk '{ print $3 }'`
do

Next thing is, there are different queues where drops can occur (0-6):

rx_queue_0_drops: 0
rx_queue_1_drops: 0
rx_queue_2_drops: 0
rx_queue_3_drops: 0
rx_queue_4_drops: 0
rx_queue_5_drops: 0
rx_queue_6_drops: 0

All of them should be checked gainst an intervall to see if the firewall really drops packets.  

Not sure if you need to do some math at this point. Usually the firewall should not drop any packet. So for me would that be an alert. 

 If this script is going to support people to maximize there performance you also need to add fw ctl pstat to request the fragmented packets versus real transmitted packets on that firewall. If the ratio to high the performance will be degraded a lot. 

Please review also these commands:

fw ctl multik print_heavy_conn
fw ctl multik heavy_conn_analyzer

It would make sense to add them to your script as well as it clearly states which connections do use most of the performance of the firewall. But yes, it has nothing to do with the packet flow. 

[RickHoppe]

Hi Andreas,

This script was originally based on just seven commands. Of course, there are many more useful commands and I do appreciate your suggestions to add extra commands. But that is really beyond the scope of s7pac. My intention with s7pac was to run those seven commands and refer to specific parts of the books Tim wrote, so you can continue investigation from there.

This could be a starting point for a new script but there is also Check Point's own healthcheck script. That one has a RX Drop Check too and it's calculation is based on values in /sys/class/net/<INTERFACE>/statistics/rx_packets and /sys/class/net/<INTERFACE>/statistics/rx_dropped. However it does not include the fw ctl multik print_heavy_conn and fw ctl multik heavy_conn_analyzer commands at this moment.

[Andreas_Zentsch]

Hi Rick,

alright, thank you for your reply. I dont know the health script you are refering too. Do you know the name? 

Thank you,

Andreas

[Timothy_Hall]

sk121447: How to perform an automated health check of a Gaia based system

[Andreas_Zentsch]

Thank you, much better explanation as i did. 🙂 Thanks for that. I tried your approach and started a tcpdump and you are right the drops dissapeared. Too bad we can not find out which packets have been dropped since this would make it easier for people to see that they might have been a layer 2 problem on the network when their counter is simply too high. 

[the_rock]

Truly fantastic presentation and post, wow! Thanks very much to everyone.