What happens to asynchronous fragments on a firewall in bridge mode?

[Brian_Deutmeyer]

I have a firewall that has two bridges, BR1 and BR2.  These bridges handle traffic for two equal cost links. Packets can route asynchronously without issue all day long (with SXL disabled), but I'm curious about fragments.  What happens when packet #1 is fragmented into 2 parts before it reaches the firewall where packet #1 frag #1 arrives at BR1 and packet #1 frag #2 arrives at BR2?

This works:

What happens here?  Does the firewall reassemble them or discard them?

[Timothy_Hall]

Doesn't matter which bridge group the packet traverses, it still has to go through the INSPECT driver either way.  If the second part of a fragmented packet arrives first out of order, the firewall will hold it for up to 1000ms waiting for all other fragments to arrive (based on default Inspection Settings), virtually reassemble them for inspection, then forward the original frags assuming they pass inspection.  So the asymmetry of firewall interfaces isn't really a problem here, but can run afoul of antispoofing enforcement.

Asymmetry becomes a problem when one of the two packet streams (forward or return) associated with a single connection is not traversing the firewall at all.

[Brian_Deutmeyer]

Thanks, Tim.  That's what I'd assume to be the behavior, as well.  However, when looking at fw ctl psatat, I see a large number of expired packets (over 50%):

Fragments:
65604306 fragments, 12766422 packets, 39920270 expired, 0 short,

When I capture packets on all 4 physical interfaces, I see the frags come in, but never leave.  Frag #1 arrives at (subseconds) HH:MM:SS.624739 and frag #2 arrives at HH:MM:SS.624763, which seems within the firewall's hold time.

A debug shows:

fw ctl zdebug -m fw + drop
;[cpu_29];[fw4_4];fwfrag_expires: IP fragment expiration reached, freeing cookies;
;[cpu_29];[fw4_4];fw_log_drop_ex: Packet proto=-1 ?:0 -> ?:0 dropped by fwfrag_expires Reason: timeout has expired for fragment;
;[cpu_29];[fw4_4];fwfrag_expires: IP fragment expiration reached, freeing cookies;
;[cpu_29];[fw4_4];fw_log_drop_ex: Packet proto=-1 ?:0 -> ?:0 dropped by fwfrag_expires Reason: timeout has expired for fragment;

If frags #1 and #2 arrive on the same bridge, life is good and the frags leave the firewall.  So it would seem to be that the frags MUST come in the same interface, but I'm looking for confirmation that this is the case.  Thoughts?

[PhoneBoy]

Previously this sort of processing (fragmented packets) was done F2F, now (in R80.20+) it's done in the F2V path. Suspect this might be a bug and it's probably worth a TAC case if you haven't opened one up yet.

[Timothy_Hall]

I know for a fact it doesn't matter when the firewall is in routed mode, but in bridged mode perhaps packets from each bridge group are "stuck" to a different Firewall Worker core instance.  All packets (and fragments) of a connection must be handled on the same worker core.  Sounds like you already have SecureXL disabled, but that doesn't mean quite the same thing it did in R80.10 and earlier. 

In your fw ctl zdebug output do you see different references for the worker instance number (fw4_X) for fragments of the same connection?

[Brian_Deutmeyer]

The _X value changes, but since this log doesn't show IPs, I'm not sure if it's for the same connection or not...

[HeikoAnkenbrand · Accepted solution]

Message "fwchain_frag Reason: wait for more fragments" does not necessarily indicate any sort of a problem. By itself, a drop for this reason simply means the Security Gateway is handling a fragmented packet.

Whenever a fragment arrives to the Security Gateway, it buffers (holds) the packet in its memory. After the fragment has been held in memory, Security Gateway drops the fragment being processed by the kernel with reason: "fwchain_frag Reason: wait for more fragments".
It does this for each fragment until the last piece (fragment) of the packet is received.

Once Security Gateway has all fragments of the packet, it re-assembles the full packet, then sends it back through the kernel inspection engine as a whole, reassembled packet to be inspected, then routed, etc.

If, for some reason, all the fragments of the packet were not received in time (1 second by default @Timothy_Hall  has already mentioned that.), then you will see another drop log: "held chain expired". At this point, the fragments that were previously held in memory are discarded.

If you do not ever see drops for "held chain expired", but only see "fwchain_frag Reason: wait for more fragments", then nothing is being dropped/blocked by the Security Gateway. It simply performs virtual fragmentation reassembly.

If you see "fwchain_frag Reason: wait for more fragments" followed by "held chain expired", then most likely, you are not receiving all the required fragments. This traffic is failing.

The Security Gateway is functioning as designed. "dropped by fwfrag_expires Reason: timeout has expired for fragment" is also an indicator that the timeout has expired and the fragmented packet has been dropped.

[Brian_Deutmeyer]

Thanks, Heiko.  Based on what you shared, I'm thinking the gateway must not know how to assemble fragments arriving on different bridges, event if they are for the same connection, resulting in the "dropped by fwfrag_expires Reason: timeout has expired for fragment;" and "fwfrag_expires: IP fragment expiration reached, freeing cookies;" messages.    Agreed?

[PhoneBoy]

That’s how I interpret it anyway.