Hello:
We are detecting high CPU usage in the virtual firewalls that we have. How can I know what is causing it?
The firewall when you execute the command "top", indicates that the PID 14995 - COMMAND - fwk5_dev, has a CPU% above 100.
In snmp v3 monitoring, I get the High CPU alert. Checking indicates that CPU # 768 is the one with this behavior. (this is indicated by the spectrum tool)
===================================================================
top - 17:15:10 up 17 days, 5:08, 1 user, load average: 4.17, 4.55, 3.99
Tasks: 502 total, 4 running, 498 sleeping, 0 stopped, 0 zombie
Cpu(s): 11.3%us, 2.9%sy, 0.0%ni, 79.2%id, 0.0%wa, 0.1%hi, 6.4%si, 0.0%st
Mem: 32779220k total, 32229544k used, 549676k free, 903992k buffers
Swap: 18892344k total, 260k used, 18892084k free, 20568592k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14995 admin 0 -20 860m 250m 64m S 112 0.8 267:59.91 fwk5_dev
15117 admin 0 -20 1241m 633m 85m S 37 2.0 4296:42 fwk3_dev
18274 admin 15 0 735m 204m 40m S 35 0.6 36:59.04 fw_full
14902 admin 0 -20 1105m 496m 84m S 15 1.6 3562:06 fwk13_dev
19721 admin 15 0 0 0 0 R 15 0.0 508:07.98 cphwd_q_init_ke
===================================================================
I have 16 CPUs, how to know which one is using the largest CPU?
How do I interpret the output of the CPVIEW that identifies two with high cpu.
Hi @raquinog,
Can you send some more screenshots:
# fw ctl affinity -l -a
# fwaccel stats -s
# top (press 1 to show cors)
More can be seen in the output of the CLI commands.
I think you do a lot of content inspection. So the PSLXL path will probably be very busy.
If this is the case you can adjust the affinity if necessary.
---
A small calculation sample for the utilization of process fwkX:
- fwk0_X -> fw instance thread that takes care for the packet processing
- fwk0_dev_X -> the thread that takes care for communication between fw instances and other CP daemons
- fwk0_kissd -> legacy Kernel Infrastructure (obsolete)
- fwk0_hp -> (high priority) cluster thread
Hi @raquinog,
Can you send some more screenshots:
# fw ctl affinity -l -a
# fwaccel stats -s
# top (press 1 to show cors)
More can be seen in the output of the CLI commands.
I think you do a lot of content inspection. So the PSLXL path will probably be very busy.
If this is the case you can adjust the affinity if necessary.
---
A small calculation sample for the utilization of process fwkX:
- fwk0_X -> fw instance thread that takes care for the packet processing
- fwk0_dev_X -> the thread that takes care for communication between fw instances and other CP daemons
- fwk0_kissd -> legacy Kernel Infrastructure (obsolete)
- fwk0_hp -> (high priority) cluster thread
Latest update on my issue:
TAC could not suggest how many fwk processes I should spawn per VS, so looking at the top output made an educated guess, my CPU levels are now running healthier (note we not using manual affinity).
We have 6 SND cores allocated and 26 worker cores now according to cpview.
VS4 =1 core
VS5 = 6 cores
VS6 = 2 cores
VS7 = 1 core
VS8 = 2 cores
VS9 = 8 cores
VS10 = 8 cores
so we have a slight oversubscription.
TAC are suggesting the below now which makes no sense.
"I went through some random perf reports and it looks like that most of the reports are filled with swapper or FWKs (the highly utilized one was fwk1 on the 31st March report).
Regarding swapper - The official description of the process is that swapper is executed when no other tasks are running on the CPU. In other words, swapper means that the CPU did nothing and ran to rest so we can
Now, as we see a Virtual Switch is highly utilized it looks like its happening due to Multiple Virtual System are connected to this Virtual Switch.
If traffic should pass from one Virtual System via Virtual Switch to another Virtual System, then 'wrp' jump takes place. Meaning - the traffic does not physically pass via the Virtual Switch. Instead, by design, traffic is forwarded (F2F) to the Check Point kernel.
As a result, in cases of a high amount of traffic, the 'fwk_dev' process that belongs to the relevant Virtual Switch might cause high CPU utilization.
As the fine-tuning on the systems is already done and secureXL is already optimized.
I would suggest segregating the Virtual Switch here and re-routing the traffic via some other V-SWs as well by adding some more V-SWs to the VSX system.
This might require a design change but at the moment, We do not see any other possible fine-tuning."
My observations:
Is not by design that a VSW should be used to connect multiple VS so they can use a common physical interface?
I have 5 VS's using this VSW, and in the current climate this is not heavily used.
The VSW is used to connect a interface to the VS's on a common internet facing vlan.
The bandwidth stats for the VSW over the last 30days is attached, and you can see its hardly used.
We are not using the VSW to pass traffic between VS's.
| User | Count |
|---|---|
| 23 | |
| 11 | |
| 10 | |
| 9 | |
| 8 | |
| 8 | |
| 7 | |
| 6 | |
| 6 | |
| 5 |
Thu 12 Dec 2024 @ 10:00 AM (COT)
LATAM: Únete a nosotros para "Explorando R82: Innovaciones y Funciones en CiberseguridadThu 12 Dec 2024 @ 05:00 PM (CET)
From Lab to Reality: Understanding MITRE's 2024 ATT&CK Evaluations (Americas)Thu 12 Dec 2024 @ 11:00 AM (CST)
From Lab to Reality: Understanding MITRE's 2024 ATT&CK Evaluations (EMEA)Thu 12 Dec 2024 @ 10:00 AM (COT)
LATAM: Únete a nosotros para "Explorando R82: Innovaciones y Funciones en CiberseguridadThu 12 Dec 2024 @ 05:00 PM (CET)
From Lab to Reality: Understanding MITRE's 2024 ATT&CK Evaluations (Americas)Thu 12 Dec 2024 @ 11:00 AM (CST)
From Lab to Reality: Understanding MITRE's 2024 ATT&CK Evaluations (EMEA)
