Firewall worker saturation 77.30

[Khalid_Aftas]

Hi,

We are having one GW on vsx 77.30 that is still not updated to r80.30 like our other firewalls (in this crisis..), we are experiencing the "known" issue of having one fwk overloaded by an elephant flow while others doing well, and causing an impact.

Our others GW on r80.30 does not have that issue thanks to Dynamic Dispatcher etc.

As migrated to r80.30 is not an option in the immediat feature, i was thinking how to further finetune this gw to cope with this workload.

We have 4 cores for SND 8 Cores for VSs, the main VS has 10 CoreXL instances.

Is there any otherway on r77.30 to finetune loadbalancing in order to not have one instance fully utilized ?

Kr,

Khalid

[Timothy_Hall · Accepted solution]

Please see my CPX 2020 presentation titled "Big Game Hunting: Elephant Flows" which is based off a chapter of my book:

https://community.checkpoint.com/fyrhh23835/attachments/fyrhh23835/member-exclusives/432/3/CPX_Big_G...

Your remediation options are going to be pretty limited by your code version and the use of VSX (which has more or less the same limitations of USFW in this context).  I wouldn't recommend trying to do static priority queuing as mentioned in the presentation on version R77.30, as that was the version the Priority Queues feature was first introduced in.  Your two best options in my opinion are sim fastaccel (sk139772: SecureXL Fast Accelerator (sim fastaccel) for Non Scalable Platforms R77.30/R80.10) and fw samp quotas (sk164472: How to configure Rate Limiting rules for DoS Mitigation (R80.10 and older)). 

However these two features were still in their relative infancy in the R77.30 release, so I'd STRONGLY recommend loading up the latest GA Jumbo HFA onto R77.30 before trying to enable them, to help ensure doing so won't cause other issues.

[HeikoAnkenbrand · Accepted solution]

Hi @Khalid_Aftas,

@Timothy_Hall describes this well.

You can also find information here:

- R80.x - Performance Tuning Tip - Elephant Flows (Heavy Connections)
- R80.x - Architecture and Performance Tuning - Link Collection 
- R80.x - Top 20 Gateway Tuning Tips 

[HeikoAnkenbrand]

R77.30 is out of support:

[Khalid_Aftas]

Yeah customer knows, new boxes are waiting to take over 🙂

I will give this a try @Timothy_Hall : is there any sideeffects/impact for those two bypass methode ?

[Khalid_Aftas]

The issue i have know is to pinpoint that elephant flow, without the Priority queues "debugs" as per documentation.

The Firewall Priority Queues cannot be enabled in the following scenarios: In R77.30 and R80.10 VSX mode 

[Kaspars_Zibarts]

Something does not sound right - you said you have 4 SXL and 8 FWK split, yet one VS has 10 workers enabled?

How many cores do you have in total and how they are split between different VSes and interfaces?

fw ctl affinity -l

would be a start

Which core is actually maxing out? 

[Khalid_Aftas]

sorry for the typo
4 cores for SND 1 Unused 11 for FWK
We have indeed 10 Instances for main vs (1)

CPU core them selfs are not overloaded, fwk process is (fwk1-3)






eth1-01: CPU 0
eth1-02: CPU 1
eth1-03: CPU 2
eth1-04: CPU 3
eth1-05: CPU 0
eth1-06: CPU 0
eth1-07: CPU 0
Mgmt: CPU 0
eth2-01: CPU 0
eth2-02: CPU 1
eth2-03: CPU 2
eth3-01: CPU 0
VS_0: CPU 5 6 7 8 9 10 11 12 13 14 15
VS_1: CPU 5 6 7 8 9 10 11 12 13 14 15
VS_2: CPU 5 6 7 8 9 10 11 12 13 14 15
VS_3: CPU 5 6 7 8 9 10 11 12 13 14 15
Interface eth3-02: has multi queue enabled
Interface eth3-03: has multi queue enabled
Interface eth3-04: has multi queue enabled

[Kaspars_Zibarts]

technically you could try isolating these three FWKs to dedicated CPU cores and not share those with other VSes / FWKs.

That is quite a bit of planning as you would need quite good knowledge of other VS FWK  CPU loading. Not a quick job but possible 🙂

I'm not saying that it will fully resolve your issues but at least you will protect remaining VSes / FWKs as now your cores are shared amongst all virtual systems.

[Khalid_Aftas]

VS1 is the main VS 2&3 are for some testing, have almost no traffic, so in this case i don't think it will help, it's just a limitation of the vsx on 77.30 without dynamic dispatcher.

[Timothy_Hall]

There aren't really side effects for setting fw samp quotas unless you make a mistake configuring them; SecureXL directly handles the quota enforcement very efficiently and the acceleration status of traffic is not impacted by application of these quotas.

The side effect of using sim fastaccel/fw ctl fast_accel was mentioned in my CPX presentation, mainly that by traffic being forced into the Accelerated path it is being "whitelisted" to some degree, and not all enforcement called for by your security policy will be applied to that traffic.  As such it is strongly recommended to only use this feature if both whitelisted systems are reasonably trusted and hopefully under your organization's direct administrative control.

[Khalid_Aftas]

Thx for the detailed explanation as always.

What are the other options to "find" the elephant flows on vsx 77.30(as priority queues debug don't work) a part from cpview ?

[Timothy_Hall]

I don't know if the cpview "Top Connections" screens work with VSX, as they don't currently work with USFW enabled either.

Please read my CPX presentation as it describes how to find elephant flows with VSX/USFW enabled; basically by using the connstat and CPMonitor utilities.

[Arik_Ovtracht]

Hi,

just wanted to point out - the "Top Connections" view in CPview really does not work in VSX, but it does work in USFW starting from R80.40.

[Khalid_Aftas]

What are those connections that are displayed then ? with bandwidth and number of packets