topic Re: High CPU utilization during process fwk0_dev_0 in General Topics

High CPU utilization during process fwk0_dev_0 (UMFW vs. KMFW)

HeikoAnkenbrand — Sun, 22 Dec 2019 17:43:56 GMT

What can be the problem on a 16 core open server R80.30 if the following process fwk0_dev_0 is at 300%-400% high CPU utilization. This prozess "fwk0_dev" indicates VSX in older versions. VSX is not enabled on this system.

What does this prozess do?

Who can help here?

# top

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19437 admin 0 -20 18.402g 0.015t 538544 S 309.9 1.0 8626:21 fwk0_dev_0 19831 admin 20 0 755376 186344 41628 S 5.6 0.0 459:42.51 fw_full

# pstree -l |grep -A 15 -B 15 fwk0_dev_0

-cprid_wd---cprid |-cpwd-+-AutoUpdaterServ-+-AutoUpdater---22*[{AutoUpdater}] | | `-sleep | |-DAService_scrip-+-DAService---5*[{DAService}] | | `-sleep | |-avi_del_tmp_fil---sleep | |-ci_http_server | |-cpd---4*[{cpd}] | |-cphamcset | |-cpview_services | |-cpviewd---{cpviewd} | |-4*[dlpu] | |-fw_full-+-in.acapd---{in.acapd} | | |-in.asessiond---{in.asessiond} | | |-in.msd---3*[{in.msd}] | | |-stormd | | |-ted---temain---46*[{temain}] | | |-usrchkd | | |-vpnd---3*[{vpnd}] | | `-10*[{fw_full}] | |-fwk_forker---fwk0_dev_0---33*[{fwk0_dev_0}] | |-fwk_wd | |-lpd

Re: High CPU utilization during process fwk0_dev_0

_Val_ — Tue, 17 Dec 2019 11:40:32 GMT

UM FW enabled?

Re: High CPU utilization during process fwk0_dev_0

Timothy_Hall — Tue, 17 Dec 2019 13:54:36 GMT

Looks like USFW is enabled due to the presence of the fwk_forker process.

Edit: My guess as to the purpose of the fwk0_dev_0 is that it acts as the liaison between the multiple fwk Firewall Worker processes and the single fwmod kernel driver instance; note that its process priority is jacked up to the maximum possible (-20).

I use the lsmod command to determine if USFW is enabled like this, which also provides insight as to the implementation of USFW:

Re: High CPU utilization during process fwk0_dev_0

HeikoAnkenbrand — Tue, 17 Dec 2019 15:49:14 GMT

Hi @Timothy_Hall,

thanks for your answer. That's what I think too.

I think someone from R&D should take a look at this.

The problem is that this process generates a very high CPU load. 300%-400% of 1600% (16 cores *100%) are used by this process. This is very high for me!

On the firewall, the CPU load of the firewall workers is otherwise very low 1-5% utilization per core.

Please can someone from R&D say something about this CPU utilization?

Is that normal for fwk0_dev_0 process?

Alternatively I can also open a TAC case.

Re: High CPU utilization during process fwk0_dev_0

HeikoAnkenbrand — Tue, 17 Dec 2019 15:45:44 GMT

Hi @Timothy_Hall, @_Val_

I was able to clear up some of the issue.

The open server was installed incorrectly. During the installation 64 cores were visible on the gateway because HT was activated. I think the person who installed the FW didn't know that HT is not supported on an open server:-)

Due to the many cores, the FW was activated in "User Mode". When I do this in the lab, I see a very high utilization of the process fwk0_dev_0 in "User Mode". If I switch the same firewall to "Kernel Mode" I can't find this process anymore.

I will set the productive firewall in "Kernel Mode" in the maintenance window.

But the question is , why does the process generate such high CPU utilization in "User Mode FW"?

Re: High CPU utilization during process fwk0_dev_0

_Val_ — Tue, 17 Dec 2019 15:49:13 GMT

Most probably it was configured incorrectly in the first place. I see no case for support here.

Re: High CPU utilization during process fwk0_dev_0

HeikoAnkenbrand — Tue, 17 Dec 2019 15:52:49 GMT

Correct 👍

Thanks to all

Re: High CPU utilization during process fwk0_dev_0

HeikoAnkenbrand — Tue, 17 Dec 2019 18:27:41 GMT

Hi @Timothy_Hall,

To make sure that USFW is activated, run the following command:

# cpprod_util FwIsUsermode

1 = User Mode Firewall
0 = Kernel Mode Firewall

If necessary, you can also add it to the third edition of your book.

Re: High CPU utilization during process fwk0_dev_0

Timothy_Hall — Wed, 18 Dec 2019 14:26:20 GMT

Yeah but I like the lsmod way better. 😁

Book? Not familiar with that obscure word...

Re: High CPU utilization during process fwk0_dev_0

HeikoAnkenbrand — Tue, 17 Dec 2019 18:28:21 GMT

👍

Re: High CPU utilization during process fwk0_dev_0

Wolfgang — Wed, 18 Dec 2019 12:33:47 GMT

Hello @all,

very very interesting thread......

I can see the same behaviour on a customer system ( open server ) with 8 cores.

We had a TAC case open but with no results at the moment.

"cpprod_util FwIsUsermode" shows "1"

I don't know how about of hyperthreading at the install (customer did this).

Can we change usermode to kernel mode on the fly with only changing "cpprod_util FwIsUsermode" ?

Does it work on ClusterXL without interruption ?

Wolfgang

Re: High CPU utilization during process fwk0_dev_0

Timothy_Hall — Wed, 18 Dec 2019 14:36:52 GMT

Generally I don't recommend changing the state of USFW from its default without consulting TAC. If it is enabled by default I'd leave it enabled as Check Point must feel that it would provide a tangible benefit. It seems that if the 3.10 kernel is present and you are using R80.30, USFW will generally be enabled, but the rules about when USFW is enabled vs. disabled by default seem to be pretty complex. I think the complexity concerning USFW being enabled/disabled by default stems from the dual kernels (2.6.18 vs. 3.10) available for R80.30, hopefully this will get clarified in R80.40 which supposedly will require use of the 3.10 kernel.

To sum up, I don't think there is any cause for alarm about fwk0_dev_0 eating lots of CPU, it is just a side effect of USFW being enabled, which causes most firewall CPU utilization to occur in process/user space instead of in the kernel.

Changing the state of USFW requires a reboot, the instructions are here: sk149973: How to enable USFW (User-Space Firewall) on a 23900 appliance

Re: High CPU utilization during process fwk0_dev_0

Wolfgang — Wed, 18 Dec 2019 14:55:13 GMT

I'll be waiting for response from TAC.....

The behaviour is really strange. We had 4 open servers, all installed with R80.30.

2 have USFW on, 2 not. Same hardware model, only different CPU cores.

We did the same install routine on all servers.

Why is it on on 2 of the servers and on the others not ? (I'll hope, TAC has an answer)

Wolfgang

Re: High CPU utilization during process fwk0_dev_0

Timothy_Hall — Wed, 18 Dec 2019 15:02:22 GMT

I've seen the same, to date I have not been able to get a clear answer other than "it depends" as far as whether USFW will be enabled by default or not. What I've been able to figure out so far is that if kernel 2.6.18 is in use, USFW will not be enabled by default upon install or upgrade, but it can be manually enabled. If the 3.10 kernel is in use it definitely seems to get enabled on firewalls with 40+ cores, and possibly on smaller ones. Beyond that I'm not sure...so if you could get a straight answer from TAC on this it would be appreciated...

How are the cores different between your firewalls that have USFW enabled vs. those that do not?

Re: High CPU utilization during process fwk0_dev_0

HeikoAnkenbrand — Wed, 18 Dec 2019 17:21:06 GMT

According to SK the UMFW is enabled from R80.30 by default. I'm afraid I noticed that a little differently. To confirm this I called a friend (He's a HP dealer.) and asked him if he had a HP DL380 with more then 40 cores in his company:-) Two hours later we were sitting in his LAB and installed R80.30 on this system.

R80.30 kernel 3.10 more then 40 cores -> UMFW is enabled (checked on HP DL 380 G10 2 * Platinum 8180MProcessor 28 cores = 56 cores)
R80.30 kernel 3.10 less then 40 cores -> UMFW is disabled (checked on HP DL 380 G10 1 * Platinum 8180MProcessor 28 cores)

Unfortunately I was not allowed to take the server with 56 cores home with me 🤣.

The installation under R80.30 (UMFW/KMFW) depends on how many cores the system has.
The licensed cores do not play a role for UMFW/KMFW.

R80.30 kernel 2.6 -> UMFW is disabled (checked on VMWare with 30 cores and with 46 cores)

Now we have generated traffic on a FW in the UMFW with a traffic generator. Approx. 500 MBit throughput (100 connections) and see there, the process fwk0_dev_0 generates 321% CPU utilization. I see this as problematic, because some cores with 80% CPU load were seen during the tests.

I think the R&D should say something about this topic if it is a problem or not ...so if you could get a straight answer from TAC on this it would be appreciated too...

Re: High CPU utilization during process fwk0_dev_0

HeikoAnkenbrand — Wed, 18 Dec 2019 17:33:03 GMT

Re: High CPU utilization during process fwk0_dev_0

Wolfgang — Wed, 18 Dec 2019 17:38:51 GMT

I agree with @HeikoAnkenbrand . These high CPU utilization could be a problem.

We had around 500 - 700Mbit Throughput on a 8 core machine, all workers are everytime utilized around 40-70%.

If we disable all blades ( ThreatPrevention, APCL, AV, AB etc.) except firewall, changes are only to 35-65%.
Same system with R77.30 and USFW off was on 10-20% CPU utilization.

I‘m really awaiting an answer from TAC.

Wolfgang

Re: High CPU utilization during process fwk0_dev_0

Jul_Kapendale — Thu, 19 Dec 2019 09:22:45 GMT

Same problem:

-HP DL 360 G10

- R80.30 kernel 3.10

- 8 core FW

- 600MBit throughput

- ThreatPrevention, IA, AV, AB, IPS

- USFW enabled

> process fwk0_dev_0 with high CPU utilization 300%-400%

I‘m really awaiting also an answer from TAC:-)

Re: High CPU utilization during process fwk0_dev_0

Timothy_Hall — Thu, 19 Dec 2019 16:10:56 GMT

Right, I don't get how USFW was enabled on your 8-core firewall unless you manually enabled it yourself. Edit: Are you only licensed for 8 cores but there are more than 40 cores on the system? The thing is that there WILL be a performance penalty when USFW is enabled due to processing having to come up through the kernel to a fwk process then back down into the kernel, whereas with USFW disabled almost everything except security server process space "trips" are handled down in the kernel which is more efficient. So unless there are more than 40 cores I don't understand why USFW would get enabled. If there are more than 40 cores, USFW makes sense as utilizing the extra cores beyond 40 would offset the performance penalty vs. being limited to 40 cores in the kernel with USFW disabled.

Obviously we are waiting for TAC here, but once again in my opinion I don't think the high CPU utilization by fwk0_dev_0 is necessarily a problem. That process is clearly multi-threaded which explains CPU utilization over 100%. I've speculated earlier that when USFW is enabled this process may be the liaison between all the fwk Firewall Worker processes, and the state tables which are still stored in the kernel. When USFW is disabled, everything is down in the kernel and it all just gets lumped in under si or sy CPU cycles and you can't see the individual kernel elements consuming CPU.

When USFW is enabled everything except state table storage and SecureXL is mostly outside the kernel and visible as individual processes, so seeing high CPU utilization by a process may look alarming but I don't think it is an actual problem. I don't believe the high CPU being consumed by fwk0_dev_0 is a bottleneck, it is just a side effect of fwk process to kernel interaction. If throughput does seem to be heavily impacted with USFW enabled, there may need to be some usual tuning performed based on the "Super Seven" and such.

Or I could be wrong...

Re: High CPU utilization during process fwk0_dev_0

HeikoAnkenbrand — Thu, 19 Dec 2019 18:25:47 GMT

HI @Jul_Kapendale

I agree with @Timothy_Hall here. UMFW should only be enabled automatically if more than 40 cores were active during installation on a R80.30 3.10 kernel.

But maybe your system has more then 40 cores on board.

For a firewall with only 8 active cores (according to license), I would recommend to change the firewall to "Kernel Mode Firewall" otherwise this will produce a performance issue.