- Products
- Learn
- Local User Groups
- Partners
- More
Welcome to Maestro Masters!
Talk to Masters, Engage with Masters, Be a Maestro Master!
Join our TechTalk: Malware 2021 to Present Day
Building a Preventative Cyber Program
ZTNA Buyer’s Guide
Zero Trust essentials for your most valuable assets
Be a CloudMate!
Check out our cloud security exclusive space!
Check Point's Cyber Park is Now Open
Let the Games Begin!
As YOU DESERVE THE BEST SECURITY
Upgrade to our latest GA Jumbo
CheckFlix!
All Videos In One Space
I have a firewall that has two bridges, BR1 and BR2. These bridges handle traffic for two equal cost links. Packets can route asynchronously without issue all day long (with SXL disabled), but I'm curious about fragments. What happens when packet #1 is fragmented into 2 parts before it reaches the firewall where packet #1 frag #1 arrives at BR1 and packet #1 frag #2 arrives at BR2?
This works:
What happens here? Does the firewall reassemble them or discard them?
Message "fwchain_frag Reason: wait for more fragments" does not necessarily indicate any sort of a problem. By itself, a drop for this reason simply means the Security Gateway is handling a fragmented packet.
Whenever a fragment arrives to the Security Gateway, it buffers (holds) the packet in its memory. After the fragment has been held in memory, Security Gateway drops the fragment being processed by the kernel with reason: "fwchain_frag Reason: wait for more fragments".
It does this for each fragment until the last piece (fragment) of the packet is received.
Once Security Gateway has all fragments of the packet, it re-assembles the full packet, then sends it back through the kernel inspection engine as a whole, reassembled packet to be inspected, then routed, etc.
If, for some reason, all the fragments of the packet were not received in time (1 second by default @Timothy_Hall has already mentioned that.), then you will see another drop log: "held chain expired". At this point, the fragments that were previously held in memory are discarded.
If you do not ever see drops for "held chain expired", but only see "fwchain_frag Reason: wait for more fragments", then nothing is being dropped/blocked by the Security Gateway. It simply performs virtual fragmentation reassembly.
If you see "fwchain_frag Reason: wait for more fragments" followed by "held chain expired", then most likely, you are not receiving all the required fragments. This traffic is failing.
The Security Gateway is functioning as designed. "dropped by fwfrag_expires Reason: timeout has expired for fragment" is also an indicator that the timeout has expired and the fragmented packet has been dropped.
Doesn't matter which bridge group the packet traverses, it still has to go through the INSPECT driver either way. If the second part of a fragmented packet arrives first out of order, the firewall will hold it for up to 1000ms waiting for all other fragments to arrive (based on default Inspection Settings), virtually reassemble them for inspection, then forward the original frags assuming they pass inspection. So the asymmetry of firewall interfaces isn't really a problem here, but can run afoul of antispoofing enforcement.
Asymmetry becomes a problem when one of the two packet streams (forward or return) associated with a single connection is not traversing the firewall at all.
Thanks, Tim. That's what I'd assume to be the behavior, as well. However, when looking at fw ctl psatat, I see a large number of expired packets (over 50%):
Fragments:
65604306 fragments, 12766422 packets, 39920270 expired, 0 short,
When I capture packets on all 4 physical interfaces, I see the frags come in, but never leave. Frag #1 arrives at (subseconds) HH:MM:SS.624739 and frag #2 arrives at HH:MM:SS.624763, which seems within the firewall's hold time.
A debug shows:
fw ctl zdebug -m fw + drop
;[cpu_29];[fw4_4];fwfrag_expires: IP fragment expiration reached, freeing cookies;
;[cpu_29];[fw4_4];fw_log_drop_ex: Packet proto=-1 ?:0 -> ?:0 dropped by fwfrag_expires Reason: timeout has expired for fragment;
;[cpu_29];[fw4_4];fwfrag_expires: IP fragment expiration reached, freeing cookies;
;[cpu_29];[fw4_4];fw_log_drop_ex: Packet proto=-1 ?:0 -> ?:0 dropped by fwfrag_expires Reason: timeout has expired for fragment;
If frags #1 and #2 arrive on the same bridge, life is good and the frags leave the firewall. So it would seem to be that the frags MUST come in the same interface, but I'm looking for confirmation that this is the case. Thoughts?
Previously this sort of processing (fragmented packets) was done F2F, now (in R80.20+) it's done in the F2V path. Suspect this might be a bug and it's probably worth a TAC case if you haven't opened one up yet.
I know for a fact it doesn't matter when the firewall is in routed mode, but in bridged mode perhaps packets from each bridge group are "stuck" to a different Firewall Worker core instance. All packets (and fragments) of a connection must be handled on the same worker core. Sounds like you already have SecureXL disabled, but that doesn't mean quite the same thing it did in R80.10 and earlier.
In your fw ctl zdebug output do you see different references for the worker instance number (fw4_X) for fragments of the same connection?
The _X value changes, but since this log doesn't show IPs, I'm not sure if it's for the same connection or not...
Message "fwchain_frag Reason: wait for more fragments" does not necessarily indicate any sort of a problem. By itself, a drop for this reason simply means the Security Gateway is handling a fragmented packet.
Whenever a fragment arrives to the Security Gateway, it buffers (holds) the packet in its memory. After the fragment has been held in memory, Security Gateway drops the fragment being processed by the kernel with reason: "fwchain_frag Reason: wait for more fragments".
It does this for each fragment until the last piece (fragment) of the packet is received.
Once Security Gateway has all fragments of the packet, it re-assembles the full packet, then sends it back through the kernel inspection engine as a whole, reassembled packet to be inspected, then routed, etc.
If, for some reason, all the fragments of the packet were not received in time (1 second by default @Timothy_Hall has already mentioned that.), then you will see another drop log: "held chain expired". At this point, the fragments that were previously held in memory are discarded.
If you do not ever see drops for "held chain expired", but only see "fwchain_frag Reason: wait for more fragments", then nothing is being dropped/blocked by the Security Gateway. It simply performs virtual fragmentation reassembly.
If you see "fwchain_frag Reason: wait for more fragments" followed by "held chain expired", then most likely, you are not receiving all the required fragments. This traffic is failing.
The Security Gateway is functioning as designed. "dropped by fwfrag_expires Reason: timeout has expired for fragment" is also an indicator that the timeout has expired and the fragmented packet has been dropped.
Thanks, Heiko. Based on what you shared, I'm thinking the gateway must not know how to assemble fragments arriving on different bridges, event if they are for the same connection, resulting in the "dropped by fwfrag_expires Reason: timeout has expired for fragment;" and "fwfrag_expires: IP fragment expiration reached, freeing cookies;" messages. Agreed?
About CheckMates
Learn Check Point
Advanced Learning
YOU DESERVE THE BEST SECURITY