Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Khalid_Aftas
Contributor

Firewall worker saturation 77.30

Jump to solution

Hi,

We are having one GW on vsx 77.30 that is still not updated to r80.30 like our other firewalls (in this crisis..), we are experiencing the "known" issue of having one fwk overloaded by an elephant flow while others doing well, and causing an impact.

Our others GW on r80.30 does not have that issue thanks to Dynamic Dispatcher etc.

As migrated to r80.30 is not an option in the immediat feature, i was thinking how to further finetune this gw to cope with this workload.

 

We have 4 cores for SND 8 Cores for VSs, the main VS has 10 CoreXL instances.

 

Is there any otherway on r77.30 to finetune loadbalancing in order to not have one instance fully utilized ?

 

Kr,

 

Khalid

 

 

0 Kudos
Reply
2 Solutions

Accepted Solutions
Timothy_Hall
Champion
Champion

Please see my CPX 2020 presentation titled "Big Game Hunting: Elephant Flows" which is based off a chapter of my book:

https://community.checkpoint.com/fyrhh23835/attachments/fyrhh23835/member-exclusives/432/3/CPX_Big_G...

Your remediation options are going to be pretty limited by your code version and the use of VSX (which has more or less the same limitations of USFW in this context).  I wouldn't recommend trying to do static priority queuing as mentioned in the presentation on version R77.30, as that was the version the Priority Queues feature was first introduced in.  Your two best options in my opinion are sim fastaccel (sk139772: SecureXL Fast Accelerator (sim fastaccel) for Non Scalable Platforms R77.30/R80.10) and fw samp quotas (sk164472: How to configure Rate Limiting rules for DoS Mitigation (R80.10 and older)). 

However these two features were still in their relative infancy in the R77.30 release, so I'd STRONGLY recommend loading up the latest GA Jumbo HFA onto R77.30 before trying to enable them, to help ensure doing so won't cause other issues.

"Max Capture: Know Your Packets" Video Series
now available at http://www.maxpowerfirewalls.com

View solution in original post

0 Kudos
Reply
14 Replies
Timothy_Hall
Champion
Champion

Please see my CPX 2020 presentation titled "Big Game Hunting: Elephant Flows" which is based off a chapter of my book:

https://community.checkpoint.com/fyrhh23835/attachments/fyrhh23835/member-exclusives/432/3/CPX_Big_G...

Your remediation options are going to be pretty limited by your code version and the use of VSX (which has more or less the same limitations of USFW in this context).  I wouldn't recommend trying to do static priority queuing as mentioned in the presentation on version R77.30, as that was the version the Priority Queues feature was first introduced in.  Your two best options in my opinion are sim fastaccel (sk139772: SecureXL Fast Accelerator (sim fastaccel) for Non Scalable Platforms R77.30/R80.10) and fw samp quotas (sk164472: How to configure Rate Limiting rules for DoS Mitigation (R80.10 and older)). 

However these two features were still in their relative infancy in the R77.30 release, so I'd STRONGLY recommend loading up the latest GA Jumbo HFA onto R77.30 before trying to enable them, to help ensure doing so won't cause other issues.

"Max Capture: Know Your Packets" Video Series
now available at http://www.maxpowerfirewalls.com

View solution in original post

0 Kudos
Reply
HeikoAnkenbrand
Champion
Champion


R77.30 is out of support:
R77_30_end_of_support_sk152052.JPG

Khalid_Aftas
Contributor

Yeah customer knows, new boxes are waiting to take over 🙂

 

I will give this a try @Timothy_Hall : is there any sideeffects/impact for those two bypass methode ?

 

 

0 Kudos
Reply
Khalid_Aftas
Contributor

The issue i have know is to pinpoint that elephant flow, without the Priority queues "debugs" as per documentation.

 

The Firewall Priority Queues cannot be enabled in the following scenarios: In R77.30 and R80.10 VSX mode 

0 Kudos
Reply
Kaspars_Zibarts
Authority
Authority

Something does not sound right - you said you have 4 SXL and 8 FWK split, yet one VS has 10 workers enabled?

How many cores do you have in total and how they are split between different VSes and interfaces?

fw ctl affinity -l

would be a start

Which core is actually maxing out? 

0 Kudos
Reply
Khalid_Aftas
Contributor
sorry for the typo
4 cores for SND 1 Unused 11 for FWK
We have indeed 10 Instances for main vs (1)

CPU core them selfs are not overloaded, fwk process is (fwk1-3)






eth1-01: CPU 0
eth1-02: CPU 1
eth1-03: CPU 2
eth1-04: CPU 3
eth1-05: CPU 0
eth1-06: CPU 0
eth1-07: CPU 0
Mgmt: CPU 0
eth2-01: CPU 0
eth2-02: CPU 1
eth2-03: CPU 2
eth3-01: CPU 0
VS_0: CPU 5 6 7 8 9 10 11 12 13 14 15
VS_1: CPU 5 6 7 8 9 10 11 12 13 14 15
VS_2: CPU 5 6 7 8 9 10 11 12 13 14 15
VS_3: CPU 5 6 7 8 9 10 11 12 13 14 15
Interface eth3-02: has multi queue enabled
Interface eth3-03: has multi queue enabled
Interface eth3-04: has multi queue enabled
0 Kudos
Reply
Kaspars_Zibarts
Authority
Authority

technically you could try isolating these three FWKs to dedicated CPU cores and not share those with other VSes / FWKs.

That is quite a bit of planning as you would need quite good knowledge of other VS FWK  CPU loading. Not a quick job but possible 🙂

I'm not saying that it will fully resolve your issues but at least you will protect remaining VSes / FWKs as now your cores are shared amongst all virtual systems.

0 Kudos
Reply
Khalid_Aftas
Contributor
VS1 is the main VS 2&3 are for some testing, have almost no traffic, so in this case i don't think it will help, it's just a limitation of the vsx on 77.30 without dynamic dispatcher.
0 Kudos
Reply
Timothy_Hall
Champion
Champion

There aren't really side effects for setting fw samp quotas unless you make a mistake configuring them; SecureXL directly handles the quota enforcement very efficiently and the acceleration status of traffic is not impacted by application of these quotas.

The side effect of using sim fastaccel/fw ctl fast_accel was mentioned in my CPX presentation, mainly that by traffic being forced into the Accelerated path it is being "whitelisted" to some degree, and not all enforcement called for by your security policy will be applied to that traffic.  As such it is strongly recommended to only use this feature if both whitelisted systems are reasonably trusted and hopefully under your organization's direct administrative control.

"Max Capture: Know Your Packets" Video Series
now available at http://www.maxpowerfirewalls.com
0 Kudos
Reply
Khalid_Aftas
Contributor
Thx for the detailed explanation as always.

What are the other options to "find" the elephant flows on vsx 77.30(as priority queues debug don't work) a part from cpview ?
0 Kudos
Reply
Timothy_Hall
Champion
Champion

I don't know if the cpview "Top Connections" screens work with VSX, as they don't currently work with USFW enabled either.

Please read my CPX presentation as it describes how to find elephant flows with VSX/USFW enabled; basically by using the connstat and CPMonitor utilities.

"Max Capture: Know Your Packets" Video Series
now available at http://www.maxpowerfirewalls.com
Arik_Ovtracht
Employee
Employee

Hi,

just wanted to point out - the "Top Connections" view in CPview really does not work in VSX, but it does work in USFW starting from R80.40.

Khalid_Aftas
Contributor
What are those connections that are displayed then ? with bandwidth and number of packets
0 Kudos
Reply