Drop optimization

[Maria_Pologova]

We are considering to enable this feature on a highly utilized gateway, where from time to time we observe spikes, which are usually related to high amount of dropped traffic.

What is your experience about this feature? Do you encounter any problems while using Drop optimization?

[JozkoMrkvicka]

We have enabled 1 year ago drop templates on around 20 clusters with no negative effect.

[Maria_Pologova]

Thank you Jozko.

May I wonder what version you are running on?

[JozkoMrkvicka]

All clusters are on R77.30. It was activated on R77.30 also.

[Timothy_Hall]

Assuming you have the latest GA Jumbo HFA for your gateway, this feature seems to work well. However in the real world I don't generally advocate turning on features like this and thus further complicating the configuration if there is no need for it.  It sounds like you have already done some analysis indicating that excessive drops are impacting the firewall, so you have a good justification for enabling this feature.

However should you start seeing any odd behavior, you will most definitely want to be on the lookout for the log entries indicating that this feature went active and started actually blocking packets that exceeded the default thresholds.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[Maria_Pologova]

Unfortunately, my problematic gateway is on R77.10. It is on the way to converting to VS. But meanwhile I did stabilize performance more or less (can't not to mention your book at this point). 

However, seeing CPU spikes exactly at time when burst of traffic appeared to be blocked, make me think that Drop Templates is a right thing to try.

One thing I don't understand though, what can happen or what kind of odd behavior might be seen, if this feature is to accelerate "unwanted" traffic that is blocked anyway.

[Timothy_Hall]

Drop Templates and Drop Optimization have a bit of a checkered history (covered in my book), and while they are supported on R77.10 I can't really recommend enabling them on that version under any circumstances.  Check these out these issues fixed in R77.10 jumbo Take 122:

Security Gateway might crash during boot if drop optimization is enabled in 'Firewall Policy Optimization' per sk90861.
Refer to sk105182.

Output of 'fwaccel stat' command shows:

Accelerator Status : off by Firewall (too many general errors (Number_Larger_than_10) (caller: cphwd_offload_drop_templates))

Refer to sk100467 (Scenario 1 - Number of elements in kernel table 'src_ranges_list' exceeds the limit).

Even if you have that take loaded I wouldn't turn them on for R77.10.  R77.30 gateway with the latest GA Jumbo HFA or R80.10+?  Sure.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[_Val_]

R77.10 is out off support for a while, just saying...

[AlekseiShelepov]

As I understand, if Drop Templates are enabled this traffic will be more affecting SND cores and take most of dropped traffic from FWK. Does it add a lot of new work for SND? For example, if we have only one-two SNDs. They already process this traffic to some point, when Drop Templates are disabled.

[Timothy_Hall]

The first packet of a new connection that does not have a SecureXL Accept Template present will always be sent F2F for a Firewall/Network Policy Layer lookup, even packets that eventually get dropped in F2F.  Optimized drops try to prevent a large amount of dropped traffic from consuming CPU resources on a Firewall Worker core by offloading those drops into SecureXL, where they will consume much less CPU time prior to being discarded.  Here is an excerpt from my book:

When Drop Optimization is enabled, the default thresholds dictate that if more than
101 drops per second occur on an individual Firewall Worker core, the feature activates.
Drop Templates for the offending traffic are automatically formed and inserted into the
SecureXL Acceleration Layer; drops begin occurring immediately in the Accelerated
Path with no need to visit the Firewall Path, thus saving valuable CPU resources on the
Firewall Worker cores.

Once the overall number of drops declines to less than 20 per second (based on the
default thresholds), the Drop Optimization feature deactivates and all the dynamically
created Drop Templates are removed from the Accelerated Path. Drops now occur once
again in the Firewall Path, with full logging if specified in the Track column of the
security policy rule matching and dropping the traffic.

The additional load on the SND/IRQ cores to track the number of drops should be pretty negligible, as should the process to dynamically insert and remove the drop templates.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[HeikoAnkenbrand]

Hi Maria,

I totally agree with Timothy.

Here you can see this again graphically in red SecureXL Drop Template and blue F2F drops. I have described it all in detail in my article R80.x Security Gateway Architecture (Logical Packet Flow).

Regards,

Heiko

[phlrnnr]

I thought that Drop optimization did NOT log drops to SmartLog when it was enabled.  However, I'm seeing different results in R80.20.  Do you know if this has changed?

Here is a log entry from SmartLog that shows it was dropped due to a drop template. 

I also see corresponding entries in 'fw ctl zdebug -T drop':

@;2726633805;13Nov2019 16:39:17.455745;[cpu_10];[fw4_0];fw_log_drop_ex: Packet proto=17 156.154.196.26:27788 -> <removed>:33470 dropped by fwmultik_handle_no_match Reason: Drop template (inbound);

Overall, this actually makes me happy - easier for troubleshooting.  However, I wanted to make sure something isn't misconfigured....

thanks!

[Timothy_Hall]

Sounds like the drop template was not offloaded to SecureXL and the drop is actually occurring on a worker core ([fw4_0]) which is why it was logged:

sk131793: Drop templates are enabled but not offloaded to SecureXL

sk150812: High CPU when traffic is dropped by fw_workers

[phlrnnr]

@Timothy_Hall thanks for the reply!  Here is a little more data.  Based on the results of 'fw ctl zdebug -T drop' command, it shows that all the drops due to reason "dropped by fwmultik_handle_no_match Reason: Drop template" are dropped by either cpu_10 or cpu_12.  These are SND cores (hyperthreading is on):

# fw ctl affinity -l
eth1-01: CPU 2
eth1-02: CPU 10
eth2-01: CPU 12
eth2-02: CPU 12
eth1: CPU 10
eth2: CPU 12
Mgmt: CPU 0
Kernel fw_0: CPU 19
Kernel fw_1: CPU 9
Kernel fw_2: CPU 18
Kernel fw_3: CPU 8
Kernel fw_4: CPU 17
Kernel fw_5: CPU 7
Kernel fw_6: CPU 16
Kernel fw_7: CPU 6
Kernel fw_8: CPU 15
Kernel fw_9: CPU 5
Kernel fw_10: CPU 14
Kernel fw_11: CPU 4
Kernel fw_12: CPU 13
Kernel fw_13: CPU 3
Daemon lpd: CPU 3 4 5 6 7 8 9 13 14 15 16 17 18 19
Daemon in.asessiond: CPU 3 4 5 6 7 8 9 13 14 15 16 17 18 19
Daemon fwd: CPU 3 4 5 6 7 8 9 13 14 15 16 17 18 19
Daemon topod: CPU 3 4 5 6 7 8 9 13 14 15 16 17 18 19
Daemon mpdaemon: CPU 3 4 5 6 7 8 9 13 14 15 16 17 18 19
Daemon cpd: CPU 3 4 5 6 7 8 9 13 14 15 16 17 18 19
Daemon cprid: CPU 3 4 5 6 7 8 9 13 14 15 16 17 18 19

In addition, I ran the kernel debugs in sk131793, and the results are as follows:

@;2921916193;14Nov2019 8:53:50.962661;[cpu_19];[fw4_0];1:[SID: 16905746] {sec_rb} up_nrb_rb_sec_allow_drop_templates_per_sub_policy: Offloading drop template allowed for this sub policy;
@;2921922205;14Nov2019 8:53:51.911328;[cpu_19];[fw4_0];1:[SID: 16905854] {sec_rb} up_nrb_rb_sec_allow_drop_templates_per_sub_policy: Offloading drop template allowed for this sub policy;

I don't see any messages stating "cannot offload drop template" as the sk mentions.

Finally, 'fwaccel stats -d' shows the drop template value increasing:

# fwaccel stats -d
Reason Value Reason Value
-------------------- --------------- -------------------- ---------------
general reason 2553005 CPASXL decision 0
PSLXL decision 5241968 clr pkt on vpn 0
encrypt failed 0 drop template 123785401
decrypt failed 8 interface down 0
cluster error 0 XMT error 0
anti spoofing 1285984 local spoofing 0
sanity error 131 monitored spoofed 0
QOS decision 0 C2S violation 0
S2C violation 6748 Loop prevention 0
DOS Fragments 0 DOS IP Options 0
DOS Blacklists 0 DOS Penalty Box 0
DOS Rate Limiting 0 Syn Attack 0
Reorder 3075 Expired Fragments 363950

During the kernel debug I generated traffic that got dropped, and again, each time, it generated a log with Drop Reason "matched optimized drop" (like I showed in my previous post).

So, do you think these are being accelerated and logged?  Or still being sent to a worker core?

[Timothy_Hall]

Looks like the drop templates were offloaded just fine to SecureXL and the drops are accelerated.  The [fw4_0] in your debug output gave me the incorrect impression that the drop was happening on a Firewall Worker instance. 

The default logging of traffic discarded by drop templates is probably a behavioral change introduced with the big SecureXL changes in R80.20+, in prior releases traffic matching a drop template was not logged by default but could be enabled by setting the kernel variable sim_track_dropdb=1.  That kernel variable doesn't seem to exist anymore in R80.20+. 

[Sanjay_S]

Hi Timothy,

We are running on R81 Take 44 and still see this drop with the message "matched optimized drop". As a temporary fix we created a new rule on the top of existing rule with the specific port which fixed the issue.

May i know what could be the reason and how to avoid this.

Regards,

Sanjay S

[Timothy_Hall]

Please provide the exact redacted log card with the message.  I am not able to distinguish if you are referring to early drop optimization as part of column-based matching or SecureXL drop optimization via Drop Templates.