- CheckMates
- :
- Products
- :
- Quantum
- :
- Security Gateways
- :
- Re: Firewall worker saturation 77.30
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Are you a member of CheckMates?
×- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Firewall worker saturation 77.30
Hi,
We are having one GW on vsx 77.30 that is still not updated to r80.30 like our other firewalls (in this crisis..), we are experiencing the "known" issue of having one fwk overloaded by an elephant flow while others doing well, and causing an impact.
Our others GW on r80.30 does not have that issue thanks to Dynamic Dispatcher etc.
As migrated to r80.30 is not an option in the immediat feature, i was thinking how to further finetune this gw to cope with this workload.
We have 4 cores for SND 8 Cores for VSs, the main VS has 10 CoreXL instances.
Is there any otherway on r77.30 to finetune loadbalancing in order to not have one instance fully utilized ?
Kr,
Khalid
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please see my CPX 2020 presentation titled "Big Game Hunting: Elephant Flows" which is based off a chapter of my book:
Your remediation options are going to be pretty limited by your code version and the use of VSX (which has more or less the same limitations of USFW in this context). I wouldn't recommend trying to do static priority queuing as mentioned in the presentation on version R77.30, as that was the version the Priority Queues feature was first introduced in. Your two best options in my opinion are sim fastaccel (sk139772: SecureXL Fast Accelerator (sim fastaccel) for Non Scalable Platforms R77.30/R80.10) and fw samp quotas (sk164472: How to configure Rate Limiting rules for DoS Mitigation (R80.10 and older)).
However these two features were still in their relative infancy in the R77.30 release, so I'd STRONGLY recommend loading up the latest GA Jumbo HFA onto R77.30 before trying to enable them, to help ensure doing so won't cause other issues.
Exclusively at CPX 2025 Las Vegas Tuesday Feb 25th @ 1:00pm
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Khalid_Aftas,
@Timothy_Hall describes this well.
You can also find information here:
- R80.x - Performance Tuning Tip - Elephant Flows (Heavy Connections)
- R80.x - Architecture and Performance Tuning - Link Collection
- R80.x - Top 20 Gateway Tuning Tips
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please see my CPX 2020 presentation titled "Big Game Hunting: Elephant Flows" which is based off a chapter of my book:
Your remediation options are going to be pretty limited by your code version and the use of VSX (which has more or less the same limitations of USFW in this context). I wouldn't recommend trying to do static priority queuing as mentioned in the presentation on version R77.30, as that was the version the Priority Queues feature was first introduced in. Your two best options in my opinion are sim fastaccel (sk139772: SecureXL Fast Accelerator (sim fastaccel) for Non Scalable Platforms R77.30/R80.10) and fw samp quotas (sk164472: How to configure Rate Limiting rules for DoS Mitigation (R80.10 and older)).
However these two features were still in their relative infancy in the R77.30 release, so I'd STRONGLY recommend loading up the latest GA Jumbo HFA onto R77.30 before trying to enable them, to help ensure doing so won't cause other issues.
Exclusively at CPX 2025 Las Vegas Tuesday Feb 25th @ 1:00pm
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Khalid_Aftas,
@Timothy_Hall describes this well.
You can also find information here:
- R80.x - Performance Tuning Tip - Elephant Flows (Heavy Connections)
- R80.x - Architecture and Performance Tuning - Link Collection
- R80.x - Top 20 Gateway Tuning Tips
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
R77.30 is out of support:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yeah customer knows, new boxes are waiting to take over 🙂
I will give this a try @Timothy_Hall : is there any sideeffects/impact for those two bypass methode ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The issue i have know is to pinpoint that elephant flow, without the Priority queues "debugs" as per documentation.
The Firewall Priority Queues cannot be enabled in the following scenarios: In R77.30 and R80.10 VSX mode
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Something does not sound right - you said you have 4 SXL and 8 FWK split, yet one VS has 10 workers enabled?
How many cores do you have in total and how they are split between different VSes and interfaces?
fw ctl affinity -l
would be a start
Which core is actually maxing out?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
4 cores for SND 1 Unused 11 for FWK
We have indeed 10 Instances for main vs (1)
CPU core them selfs are not overloaded, fwk process is (fwk1-3)
eth1-01: CPU 0
eth1-02: CPU 1
eth1-03: CPU 2
eth1-04: CPU 3
eth1-05: CPU 0
eth1-06: CPU 0
eth1-07: CPU 0
Mgmt: CPU 0
eth2-01: CPU 0
eth2-02: CPU 1
eth2-03: CPU 2
eth3-01: CPU 0
VS_0: CPU 5 6 7 8 9 10 11 12 13 14 15
VS_1: CPU 5 6 7 8 9 10 11 12 13 14 15
VS_2: CPU 5 6 7 8 9 10 11 12 13 14 15
VS_3: CPU 5 6 7 8 9 10 11 12 13 14 15
Interface eth3-02: has multi queue enabled
Interface eth3-03: has multi queue enabled
Interface eth3-04: has multi queue enabled
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
technically you could try isolating these three FWKs to dedicated CPU cores and not share those with other VSes / FWKs.
That is quite a bit of planning as you would need quite good knowledge of other VS FWK CPU loading. Not a quick job but possible 🙂
I'm not saying that it will fully resolve your issues but at least you will protect remaining VSes / FWKs as now your cores are shared amongst all virtual systems.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There aren't really side effects for setting fw samp quotas unless you make a mistake configuring them; SecureXL directly handles the quota enforcement very efficiently and the acceleration status of traffic is not impacted by application of these quotas.
The side effect of using sim fastaccel/fw ctl fast_accel was mentioned in my CPX presentation, mainly that by traffic being forced into the Accelerated path it is being "whitelisted" to some degree, and not all enforcement called for by your security policy will be applied to that traffic. As such it is strongly recommended to only use this feature if both whitelisted systems are reasonably trusted and hopefully under your organization's direct administrative control.
Exclusively at CPX 2025 Las Vegas Tuesday Feb 25th @ 1:00pm
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What are the other options to "find" the elephant flows on vsx 77.30(as priority queues debug don't work) a part from cpview ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I don't know if the cpview "Top Connections" screens work with VSX, as they don't currently work with USFW enabled either.
Please read my CPX presentation as it describes how to find elephant flows with VSX/USFW enabled; basically by using the connstat and CPMonitor utilities.
Exclusively at CPX 2025 Las Vegas Tuesday Feb 25th @ 1:00pm
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
just wanted to point out - the "Top Connections" view in CPview really does not work in VSX, but it does work in USFW starting from R80.40.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
