Part 7 - Performance Analysis and Optimization Flow

_Val_ · ‎2020-06-22

In the prior article numerous commands to help assess possible performance bottlenecks were demonstrated. However keeping track of all the different commands and how to interpret them can be truly daunting, and frankly are beyond the scope of a beginner-level series such as CP4B.

But there is good news: the CheckMates community has already done most of the work for you!

When responding to CheckMates postings about unacceptable firewall performance or mysteriously high CPU utilization, over time the author of these series developed a set of commands to run called the “Super Seven”. Once the output from these commands run on the problematic firewall were posted, Mr. Hall would examine the outputs and provide customized analysis along with recommendations for improving firewall performance. The original “Super Seven” commands were:

fwaccel stat - covered in previous article
fwaccel stats -s - covered in previous article
grep -c ^processor /proc/cpuinfo - Displays total number of CPUs/Cores
cat /proc/smt_status (2.6.18 kernel), lscpu | grep Thread (3.10 kernel) - covered in previous article
fw ctl affinity -l -r - covered in previous article
netstat -ni - covered in previous article
fw ctl multik stat - display balance of current/peak connections across Firewall workers
cpstat os -f multi_cpu -o 1 - Show live CPU/core utilization

Follow-up command outputs that might be requested in some cases:

enabled_blades – show which blades/features are enabled on the firewall
free -m - display firewall memory and swap utilization
fw ctl multik get_mode – display status of Dynamic Dispatcher (enabled/disabled)
fw ver – display firewall code version
cpinfo -y all - display current Jumbo HFA patch level
cpprod_util FwIsUsermode - Display User Space FW (USFW) status

Over time, the same types of postings concerning firewall performance issues were posted over and over again. CheckMates user Rick Hoppe developed a free script called s7pac which automates the collection of information from these commands, flags any results that may indicate a performance problem, and even provides page number references from the Max Power 2020 book for further reading!

More information about this tool (which is run from the command line of a firewall) can be provided here:

https://community.checkpoint.com/t5/General-Topics/Super-Seven-Performance-Assessment-Commands-s7pac...

CheckMates user Danny Jung has also created a free “SmartConsole Extension” that allows the s7pac script to be run directly from the SmartConsole GUI, for more information see this CheckMates thread:

https://community.checkpoint.com/t5/SmartConsole-Extensions/SmartConsole-Extension-Performance-Asses...

Screenshot 2020-06-22 at 13.32.06.png

Here is sample output from the s7pac script run from the command line:

[Expert@CSBFW-PROD-A:0]# /usr/bin/s7pac
+-----------------------------------------------------------------------------+
| Super Seven Performance Assessment Commands v0.3 (Thanks to Timothy Hall) |
+-----------------------------------------------------------------------------+
| Inspecting your environment: OK |
| This is a firewall....(continuing) |
| |
| Referred pagenumbers are to be found in the following book: |
| Max Power: Check Point Firewall Performance Optimization - Second Edition |
| Available at http://www.maxpowerfirewalls.com/ |
| |
+-----------------------------------------------------------------------------+
| Command #1: fwaccel stat |
| |
| Check for : Accelerator Status must be enabled (R77.xx/R80.10 versions) |
| Status must be enabled (R80.20) |
| Accept Templates must be enabled |
| Message "disabled" from (low rule number) = bad |
| |
| Chapter 9: SecureXL throughput acceleration |
| Page 278 |
+-----------------------------------------------------------------------------+
| Output: |
Accelerator Status : on
Accept Templates : enabled
Drop Templates : disabled
NAT Templates : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
NAT64, GTPAcceleration, SCTPAcceleration,
McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256

+-----------------------------------------------------------------------------+
| Command #2: fwaccel stats -s |
| |
| Check for : Accelerated conns/Totals conns: >25% good, >50% great |
| Accelerated pkts/Total pkts : >50% great |
| PXL pkts/Total pkts : >50% OK |
| F2Fed pkts/Total pkts : <30% good, <10% great |
| |
| Chapter 9: SecureXL throughput acceleration |
| Page 287, Packet/Throughput Acceleration: The Three Kernel Paths |
+-----------------------------------------------------------------------------+
| Output: |
Accelerated conns/Total conns : 0/35 (0%)
Accelerated pkts/Total pkts : 0/3983956 (0%)
F2Fed pkts/Total pkts : 3983956/3983956 (100%)
PXL pkts/Total pkts : 0/3983956 (0%)
QXL pkts/Total pkts : 0/3983956 (0%)

+-----------------------------------------------------------------------------+
| Command #3: grep -c ^processor /proc/cpuinfo && /sbin/cpuinfo |
| |
| Check for : If number of cores is roughly double what you are excpecting, |
| hyperthreading may be enabled |
| |
| Chapter 7: CorexL Tuning |
| Page 239 |
+-----------------------------------------------------------------------------+
| Output: |
6
HyperThreading=disabled

+-----------------------------------------------------------------------------+
| Command #4: fw ctl affinity -l -r |
| |
| Check for : SND/IRQ/Dispatcher Cores, # of CPU's allocated to interface(s) |
| Firewall Workers/INSPECT Cores, # of CPU's allocated to fw_x |
| R77.30: Support processes executed on ALL CPU's |
| R80.xx: Support processes only executed on Firewall Worker Cores |
| |
| Chapter 7: CoreXL Tuning |
| Page 221 |
+-----------------------------------------------------------------------------+
| Output: |
CPU 0: eth1-04
CPU 1: Sync Mgmt
CPU 2:
CPU 3:
CPU 4:
CPU 5:
All:

+-----------------------------------------------------------------------------+
| Command #5: netstat -ni |
| |
| Check for : RX/TX errors |
| RX-DRP % should be <0.1% calculated by (RX-DRP/RX-OK)*100 |
| TX-ERR might indicate Fast Ethernet/100Mbps Duplex Mismatch |
| |
| Chapter 2: Layers 1&2 Performance Optimization |
| Page 28-35 |
| |
| Chapter 7: CoreXL Tuning |
| Page 204 |
+-----------------------------------------------------------------------------+
| Output: |
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Mgmt 1500 0 43600162 2 0 0 36013528 0 0 0 BMRU
Sync 1500 0 61606193 0 0 0 283637876 0 0 0 BMRU
eth1-04 1500 0 5426765033 0 495 495 4930747640 0 0 0 BMRU
lo 16436 0 8398237 0 0 0 8398237 0 0 0 LRU

interface eth1-04: There are no RX drops

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1 | 8 | 8 | 84 | 16 | ? | 4 |
| 2 | 5 | 6 | 89 | 11 | ? | 4 |
| 3 | 4 | 5 | 91 | 9 | ? | 4 |
| 4 | 4 | 4 | 92 | 8 | ? | 4 |
| 5 | 4 | 4 | 92 | 8 | ? | 4 |
| 6 | 4 | 4 | 93 | 7 | ? | 4 |
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1 | 14 | 16 | 70 | 30 | ? | 2726 |
| 2 | 4 | 11 | 86 | 14 | ? | 2726 |
| 3 | 5 | 8 | 87 | 13 | ? | 5452 |
| 4 | 5 | 6 | 89 | 11 | ? | 2726 |
| 5 | 8 | 9 | 84 | 16 | ? | 2726 |
| 6 | 10 | 9 | 81 | 19 | ? | 5452 |
---------------------------------------------------------------------------------

Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1 | 10 | 26 | 64 | 36 | ? | 694 |
| 2 | 3 | 9 | 89 | 11 | ? | 694 |
| 3 | 5 | 8 | 87 | 13 | ? | 694 |
| 4 | 5 | 10 | 86 | 14 | ? | 694 |
| 5 | 6 | 14 | 80 | 20 | ? | 695 |
| 6 | 7 | 13 | 80 | 20 | ? | 697 |
---------------------------------------------------------------------------------

+-----------------------------------------------------------------------------+
| Thanks for using s7pac |

Bottom line: CheckMates and its users stand ready to assist you with firewall performance issues! The s7pac results output can be posted to the CheckMates forum, where you will receive expert feedback from the CheckMates community on possible configuration adjustments to improve performance.

Editor's Notes

In addition, we recommend our community members to study and follow Best Practices - Security Gateway Performance SecureKnowledge article.

This concludes the Check Point for Beginners (CP$B) series on performance optimization, and we hope you have enjoyed it!

About the author

Performance Optimization Series are written for you by Timothy Hall.

Timothy has continuously worked with Check Point products starting in 1997, been an instructor for official Check Point training classes since 2004, and is the author of the book "Max Power 2020: Check Point Firewall Performance Optimization".