IPS bypass under load - any way to exclude certain cores ?

[TomShanti]

Hi,

we have a core assigned to our sync interface.

This interface now triggers the IPS bypass under load condition even though the "relevant" fw_worker cores have no high usage.

Already found this SK but it does not help: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

So is there a way to exclude a certain core from the calculation ?

Regards Thomas

[PhoneBoy]

It specifically says any one core (not average CPU usage).
Don't see how a specific core can be excluded.

[Timothy_Hall]

In my experience I wouldn't recommend enabling the IPS Bypass Under Load feature under any circumstances.  As you discovered all it takes is one core going above the thresholds (either SND or Worker) to kill all IPS enforcement, which is very likely to happen with a busy gateway and virtually guaranteed with the presence of elephant flows/heavy connections.  The real-world effect is that IPS enforcement is pretty much always disabled; this Bypass feature made sense in the old days when firewalls only had a few cores and any one of them becoming saturated by IPS enforcement duties caused a very noticeable effect.  However with so many firewall cores these days, time has passed this feature by as implemented and it is frankly no longer relevant or advisable.   Here are the notes from my IPS Immersion Video class about this topic:

Click to Expand

This controversial feature will disable all IPS inspection completely (essentially running the ips off command) when both High
Thresholds are exceeded, and re–enable IPS inspection when both Low thresholds are met. Note that all it takes is for ONE
core to reach these thresholds for IPS enforcement to be disabled on ALL Firewall Worker cores FOR THE ENTIRE GATEWAY.
See the following SK for more information about this potentially unexpected effect: sk107334: IPS Bypass is triggered even when CPU utilization is not over the defined threshold.  

[TomShanti]

Hi Timothy,

while disabling the feature solves the issue afaik it was also designed to cope with kind of DOS attacks caused by high IPS load (I know that its a bad work around for wrong sizing 😎).

It is not ideal to bypass IPS but the design with calculating bypass through all Cores is quite bad. It should be triggered by some other "intelligent" thresholds.

Regards Thomas

[Timothy_Hall]

Agreed the calculation mechanism for IPS Bypass needs to be updated to consider the presence of so many more cores on today's firewalls, and is why I can't recommend ever enabling IPS Bypass in its present form.  Tuning the IPS feature to reduce CPU load is far more likely to be fruitful, I think some guy wrote a book about that very topic...

[CheckPointerXL]

Hello Tim, no fix in 2023? Just crazy that we cannot stop ips process based on average CoreXL utilization....

[Chris_Atkinson]

Recent JHF takes do have some fixes (bypass under load) but no fundamental change to the mechanism itself to my knowledge.

There are however some tweaks possible per: sk62848

[PhoneBoy]

Rather than bypassing IPS when a specific core goes 100%, how about use more (less utilized) cores?
This is what happens with R81.20 and HyperFlow.

[Chris_Atkinson]

Indeed this is another option, provided your appliance has 8-cores or more as a prerequisite for HyperFlow (sk178070).

If not you'll have to employ other optimization/tuning strategies per above. 

[Timothy_Hall]

IPS Bypass Under Load was unusable on a modern Check Point gateway except maybe a Quantum Spark.  As originally designed if any one core went over the CPU threshold, IPS was turned off on all cores which basically means IPS is constantly off in the real world.  However in R81.10 Jumbo HFA 110+ and R81.20 Take 26+ (and I assume R82) the calculation is based on the average CPU load of *all* cores instead which must cross the CPU threshold to start a bypass:

PRJ-46941,
TPP-3290
UPDATE: IPS bypass triggers will now be activated based on the average CPU load exceeding the high threshold, as opposed to the previous implementation, where a single CPU load triggered the bypass. The change will result in more effective security measures without unnecessary bypasses.

So it is actually usable now but I still would not recommend it.