topic Re: cpu detective and SND congestion in General Topics

cpu detective and SND congestion

Luis_Miguel_Mig — Fri, 27 Dec 2024 15:49:25 GMT

I have just noticed that the CPU detective is only designed to capture elephant flows if the FW worker is the problem.
If the SND process is congested then the CPU detective doesn't capture heavy connections.

sk166454 article is consistent what it has been described above.

I was wondering if someone knows how to workaround this constrain

I think it make sense and it would be very useful if we could find heavy connection if the SND is congested.

Re: cpu detective and SND congestion

AkosBakos — Fri, 27 Dec 2024 18:16:10 GMT

Hi @Luis_Miguel_Mig

And what about spike_detective?

I usually use this to determinate the high CPU usage on the GW.

Search for this string in the /var/log/messages.

akos

Re: cpu detective and SND congestion

AkosBakos — Fri, 27 Dec 2024 18:26:59 GMT

Hi @Luis_Miguel_Mig

And one more really useful command: fw ctl multik print_heavy_conn

https://support.checkpoint.com/results/sk/sk178070

Maybe it can help.

Akos

Re: cpu detective and SND congestion

the_rock — Fri, 27 Dec 2024 19:46:35 GMT

I had seen Tim Hall give that command many times, its great!

Andy

Re: cpu detective and SND congestion

Timothy_Hall — Fri, 27 Dec 2024 20:09:42 GMT

On a Quantum Force (9000/19000/29000) or Lightspeed appliance, 100% CPU utilization on the SND cores (at least as reported by Linux-based tools such as vmstat or top) does not necessarily indicate congestion as UPPAK is enabled, which uses poll mode instead of interrupts to grab traffic for processing.

Please provide more information about your appliance model number and Jumbo HFA version.

Re: cpu detective and SND congestion

Luis_Miguel_Mig — Mon, 30 Dec 2024 10:12:44 GMT

My two cores dedicated to SND are usually around 15% utilization. During this period of heavy traffic no only the cpu was over 80% but it also had a big impact in traffic latency traversing the FW.

I could see the cpu cores over 80% (and caused by SNDs) during the traffic spike thanks to CPU spike detective and cpview -t.

As I said, sk166454 article describes how CPU spike detective only captures elephant flows when the FW worker is under pressure.

My question is if anybody knows any workaround, trick or configuration settings that could allow CPU spike detective to capture "fw ctl multik print_heavy_conn" when SND is over 80% usage.

Re: cpu detective and SND congestion

Timothy_Hall — Mon, 30 Dec 2024 13:35:18 GMT

Having the Spike Detective report spikes on the SNDs doesn't seem possible, as it is oriented to detecting user/process space thread/process spikes, whereas SecureXL/sim is mostly in the kernel on the SNDs. Could probably write some kind of monitoring script that would check the SND load every 60 seconds or something, then try to grab some stats when it goes over some threshold.

Re: cpu detective and SND congestion

Timothy_Hall — Mon, 30 Dec 2024 13:43:13 GMT

Also you bring up a good point; I'm not sure that fw ctl multik print_heavy_conn will show elephant/heavy flows detected on an SND core, as the detection mechanism was originally developed to trigger Priority Queueing which only happens on the worker instances. Tagging @AmitShmuel for a clarification on whether elephant flow detection happens on SND cores for fastpath traffic; I believe the answer is no.

Re: cpu detective and SND congestion

Luis_Miguel_Mig — Thu, 02 Jan 2025 10:12:50 GMT

Yeah, you are right. I guess that the only options we have in that case is either netflow or rule accounting.
I guess none of them may help, they both will require more resources just when the firewall is congested due to an elephant flow.
My guess that rule accounting may be less demanding in terms of cpu resources, the problem is that it may be a pain to set it up for all the rules in your firewall.
It would be nice if there was an option to option in the global settings activate rule accounting for all the rules.

I guess another option may be to use the REST api and active rule accounting one by one with a script.