This is update #2, update #1 is accessible here.
This article provides updates for the Shadow Peak "Max Capture: Know Your Packets" self-guided video series that covers taking packet captures on Check Point Security Gateways using tools such as fw monitor, tcpdump, and cppcap. New techniques and updates for this course will be provided free of charge to all CheckMates users in posts such as this one.
This update addresses a rather concerning interaction between fw monitor -F and the fw ctl zdebug + drop commands when they are run simultaneously; both commands were extensively covered in the Max Capture course. One major caution with using fw monitor -F mentioned in the Max Capture course was avoiding a situation where you would start receiving a completely unfiltered capture, as it could significantly impact the performance and even the stability of the firewall in extreme cases. This is due to the fact that fw monitor -F captures packets directly in the sim/SecureXL driver, whose proper functioning is critical for timely handling of traffic and avoiding packet loss at the network level. One way to have this happen was making a typo in the <srcIP,srcPort,dstIP,dstPort,protocol> traffic filter or trying to use a wildcard in the matching filter. In this case no filtering debug flags at all would be set in the sim/SecureXL driver, and an unfiltered capture would be the result.
As it turns out the fw ctl zdebug + drop command also interacts with the sim/SecureXL debug filtering settings. As a result if both commands are run simultaneously, they can interfere with each other and suddenly cause an unfiltered capture to occur.
Note that fw monitor -e captures traffic in the F2F path chain modules on the Firewall Workers/Instances and is not subject to this effect. tcpdump and cppcap also do not interact with the sim/SecureXL debug filter flags.
Consider this first sequence of events:
Prompt1# fw ctl zdebug + drop
Prompt2# fw monitor -F 8.8.8.8,0,0,0,0 -F 0,0,8.8.8.8,0,0
Both running commands work properly. However if CNTRL-C is hit to stop the fw ctl zdebug + drop, it essentially performs a fw ctl debug 0 which resets all debugging flags and debug filters in sim/SecureXL and the Firewall Workers/Instances. Result: The fw monitor -F capture suddenly becomes unfiltered!
Now consider this second sequence:
Prompt1# fw monitor -F 8.8.8.8,0,0,0,0 -F 0,0,8.8.8.8,0,0
Prompt2# fw ctl zdebug + drop
The fw monitor -F capture is properly filtered initially, until fw ctl zdebug + drop is run, at which time the fw monitor -F capture suddenly becomes unfiltered!
And now the final sequence:
Prompt1# fw ctl zdebug + drop
Prompt2# fw monitor -F 8.8.8.8,0,0,0,0 -F 0,0,8.8.8.8,0,0
Both running commands work properly. Now we CNTRL-C the fw monitor -F command first, and the properly filtered capture stops. The fw ctl zdebug + drop command continues to work properly until we terminate it. However note that when both commands are running simultaneously in this fashion, any traffic dropped by sim/SecureXL (this is not common - most drops occur on a Firewall Instance/Worker) will only be shown by fw ctl zdebug + drop if it matches the active fw monitor -F capture filter. This is because there is one (and only one) set of sim/SecureXL debug filtering settings that these two commands share, which is the root cause of this entire issue.
TAKEAWAY: If you need to run these two commands simultaneously, the proper order to keep them from interfering with each other is:
fw ctl zdebug + drop
fw monitor -F (filter)
When the capturing/debugging is complete:
Stop the fw monitor -F command
Stop the fw ctl zdebug + drop command
Another option to ensure any rare drops in sim/SecureXL can always be seen would be to use tcpdump/cppcap for the packet capturing portion (these tools can successfully capture even fully-accelerated traffic in almost all cases) while fw ctl zdebug + drop is running instead of utilizing fw monitor -F.
One other update: Max Capture page 81 states the following:
Note that in some Check Point documentation, use of the command fw ctl zdebug drop is shown instead without the
extra "+" character. When including this "+" character, drops performed by SecureXL (although much more rare than
drops by Firewall/INSPECT) will be included in the output as well. To ensure that all drops are seen (not just those by
INSPECT) use fw ctl zdebug + drop with the "+" character.
This statement is incorrect, drops that occur inside sim/SecureXL will still be shown even if the "+" is not included. As far as I can tell adding the "+" option causes additional "housekeeping" debug output to appear that is indirectly related to the dropping of packets, in addition to the standard output showing the direct dropping of packets by Check Point code. While these extra "housekeeping" messages probably won't make much sense to most Check Point administrators, they may well be useful to TAC and R&D if a case is opened. Therefore the recommendation is to always include the "+" option with fw ctl zdebug + drop just in case you capture a rare, intermittent drop event while this command is running and need to open a case with TAC for further clarification.
A big thanks to Max Capture customer @Robert_Sisk for bringing this issue to my attention. Thanks for reading!
Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
