cancel
Showing results for
Did you mean:
Highlighted
Ruby

High CPU utilization during process fwk0_dev_0 (UMFW vs. KMFW)

What can be the problem on a 16 core open server R80.30 if the following process fwk0_dev_0 is at 300%-400% high CPU utilization. This prozess "fwk0_dev" indicates VSX in older versions. VSX is not enabled on this system.

What does this prozess do?

Who can help here?

# top

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
19437 admin      0 -20 18.402g 0.015t 538544 S 309.9  1.0   8626:21 fwk0_dev_0
19831 admin     20   0  755376 186344  41628 S   5.6  0.0 459:42.51 fw_full

# pstree -l |grep -A 15 -B 15 fwk0_dev_0

-cprid_wd---cprid
|-cpwd-+-AutoUpdaterServ-+-AutoUpdater---22*[{AutoUpdater}]
|      |                 -sleep
|      |-DAService_scrip-+-DAService---5*[{DAService}]
|      |                 -sleep
|      |-avi_del_tmp_fil---sleep
|      |-ci_http_server
|      |-cpd---4*[{cpd}]
|      |-cphamcset
|      |-cpview_services
|      |-cpviewd---{cpviewd}
|      |-4*[dlpu]
|      |-fw_full-+-in.acapd---{in.acapd}
|      |         |-in.asessiond---{in.asessiond}
|      |         |-in.msd---3*[{in.msd}]
|      |         |-stormd
|      |         |-ted---temain---46*[{temain}]
|      |         |-usrchkd
|      |         |-vpnd---3*[{vpnd}]
|      |         -10*[{fw_full}]
|      |-fwk_forker---fwk0_dev_0---33*[{fwk0_dev_0}]
|      |-fwk_wd
|      |-lpd

Tags (2)
2 Solutions

Accepted Solutions
Highlighted
Ruby

Tags (1)
Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0 (UMFW vs. KMFW)

thanks for your answer. I have also a couple of follow-up questions:

For my tests I switched a LAB firewall R80.30 kernel 3.10 with 4 cores under VMWare to a UMFW.

1) Is this really the process fwd_dev you describe? I can't find anything here to process fwd_dev with "top" and "ps" on the firewall. I think this is the process "fwk0_dev_0". Here I can find more threads with top if USFM is enabled (see CLI commands).

2) What are the following processes responsible for and what is the function?

- fwk0_X
- fwk0_dev_X
- fwk0_kissd
- fwk0_hp

I think out loud:

If I use cpconfig to change the CoreXL cores, the number of processes fwk0_X and fwk0_dev_X (X stands for the CoreXL instance)  will also change. I think those are our firewall workers processes at a USFW. If this is the case, I can use the following command "top -Hbn1 -p <prozess ID fwk0_dev_0>to see how the firewall worker threads are being utilized. When I look at the process fwk0_dev_0 with top I see a very high load in sum. This could be the sum of all thread utilizations from process fwk0_dev_0.

A small calculation sample for the utilization of process fwk0_dev_0:

max_CoreXL_number            max_CoreXL_number
fwk0_dev_0      =      ∑       fwk0_x                    +                fwk0_dev_x          +        fwk0_kissd        +          fwk0_hp
x=0                                              x=0

This would explain the high process utilization of the process. Should that be the case, I answered one of my questions myself 🙂

I'd appreciate it if we could work this out. Thanks in advance.

CLI commands:

-------------------------------------------------------------------------

[Expert@GW2:0]# cpprod_util FwIsUsermode
1                                                                                                                             >>> USFW is enabled

[Expert@GW2:0]# top

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10219 admin 0 -20 1070m 449m 134m S 2 24.0 0:17.19 fwk0_dev_0          >>> fwk0_dev_0 process id 10219

[Expert@GW2:0]# top -Hbn1 -p 10219                                                              >>> threads for process 10219 (fwk0_dev_0)

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10219 admin 0 -20 1070m 449m 134m S 0 24.0 0:03.49 fwk0_dev_0
10220 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.00 fwk0_kissd
10436 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.57 fwk0_0
10437 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.64 fwk0_1
10438 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.67 fwk0_2
10439 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.80 fwk0_3
10440 admin RT -20 1070m 449m 134m S 0 24.0 0:00.76 fwk0_hp
10441 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.15 fwk0_dev_1
10442 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.09 fwk0_dev_2
10443 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.09 fwk0_dev_3

# pstree -p 10219
fwk0_dev_0(10219)-+-{fwk0_dev_0}(10220)
|-{fwk0_dev_0}(10436)
|-{fwk0_dev_0}(10437)
|-{fwk0_dev_0}(10438)
|-{fwk0_dev_0}(10439)
|-{fwk0_dev_0}(10440)
|-{fwk0_dev_0}(10441)
|-{fwk0_dev_0}(10442)
-{fwk0_dev_0}(10443)

Tags (1)
51 Replies
Highlighted

UM FW enabled?

Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

Looks like USFW is enabled due to the presence of the fwk_forker process.

Edit: My guess as to the purpose of the fwk0_dev_0 is that it acts as the liaison between the multiple fwk Firewall Worker processes and the single fwmod kernel driver instance; note that its process priority is jacked up to the maximum possible (-20).

I use the lsmod command to determine if USFW is enabled like this, which also provides insight as to the implementation of USFW:

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com
Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

I think someone from R&D should take a look at this.

The problem is that this process generates a very high CPU load. 300%-400% of 1600% (16 cores *100%) are used by this process. This is very high for me!

On the firewall, the CPU load of the firewall workers  is otherwise very low 1-5% utilization per core.

Is that normal for fwk0_dev_0 process?

Alternatively I can also open a TAC case.

Tags (1)
Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

I was able to clear up some of the issue.

The open server was installed incorrectly. During the installation 64 cores were visible on the gateway because HT was activated. I think the person who installed the FW didn't know that HT is not supported on an open server:-)

Due to the many cores, the FW was activated in "User Mode". When I do this in the lab, I see a very high utilization of the process fwk0_dev_0 in "User Mode". If I switch the same firewall to "Kernel Mode" I can't find this process anymore.

I will set the productive firewall in "Kernel Mode" in the maintenance window.

But the question is , why does the process generate such high CPU utilization in "User Mode FW"?

Tags (1)
Highlighted

Re: High CPU utilization during process fwk0_dev_0

Most probably it was configured incorrectly in the first place. I see no case for support here.

Highlighted
Ruby

Correct 👍

Thanks to all

Tags (1)
Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

To make sure that USFW is activated, run the following command:

# cpprod_util FwIsUsermode

1 = User Mode Firewall
0 = Kernel Mode Firewall

If necessary, you can also add it to the third edition of your book.

Tags (1)
Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

Yeah but I like the lsmod way better. 😁

Book? Not familiar with that obscure word...

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com
Highlighted
Ruby

👍

Highlighted
Gold

Re: High CPU utilization during process fwk0_dev_0

Hello @all,

I can see the same behaviour on a customer system ( open server )  with 8 cores.

"cpprod_util FwIsUsermode" shows "1"

I don't know how about of hyperthreading at the install (customer did this).

Can we change usermode to kernel mode on the fly with only changing "cpprod_util FwIsUsermode" ?

Does it work on ClusterXL without interruption ?

Wolfgang

Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

Generally I don't recommend changing the state of USFW from its default without consulting TAC.  If it is enabled by default I'd leave it enabled as Check Point must feel that it would provide a tangible benefit.  It seems that if the 3.10 kernel is present and you are using R80.30, USFW will generally be enabled, but the rules about when USFW is enabled vs. disabled by default seem to be pretty complex.  I think the complexity concerning USFW being enabled/disabled by default stems from the dual kernels (2.6.18 vs. 3.10) available for R80.30, hopefully this will get clarified in R80.40 which supposedly will require use of the 3.10 kernel.

To sum up, I don't think there is any cause for alarm about fwk0_dev_0 eating lots of CPU, it is just a side effect of USFW being enabled, which causes most firewall CPU utilization to occur in process/user space instead of in the kernel.

Changing the state of USFW requires a reboot, the instructions are here: sk149973: How to enable USFW (User-Space Firewall) on a 23900 appliance

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com
Highlighted
Gold

Re: High CPU utilization during process fwk0_dev_0

I'll be waiting for response from TAC.....

The behaviour is really strange. We had 4 open servers, all installed with R80.30.

2 have USFW on, 2 not. Same hardware model, only different CPU cores.

We did the same install routine on all servers.

Why is it on on 2 of the servers and on the others not ? (I'll hope, TAC has an answer)

Wolfgang

Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

I've seen the same, to date I have not been able to get a clear answer other than "it depends" as far as whether USFW will be enabled by default or not.  What I've been able to figure out so far is that if kernel 2.6.18 is in use, USFW will not be enabled by default upon install or upgrade, but it can be manually enabled.  If the 3.10 kernel is in use it definitely seems to get enabled on firewalls with 40+ cores, and possibly on smaller ones.  Beyond that I'm not sure...so if you could get a straight answer from TAC on this it would be appreciated...

How are the cores different between your firewalls that have USFW enabled vs. those that do not?

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com
Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

According to SK the UMFW is enabled from R80.30 by default. I'm afraid I noticed that a little differently. To confirm this I called a friend (He's a HP dealer.) and asked him if he had a HP DL380 with more then 40 cores in his company:-) Two hours later we were sitting in his LAB and installed R80.30 on this system.

R80.30 kernel 3.10 more then 40 cores -> UMFW is enabled    (checked on HP DL 380 G10 2 * Platinum 8180MProcessor 28 cores = 56 cores)
R80.30 kernel 3.10 less then 40 cores   -> UMFW is disabled    (checked on HP DL 380 G10 1 * Platinum 8180MProcessor 28 cores)

Unfortunately I was not allowed to take the server with 56 cores home with me 🤣.

The installation under R80.30 (UMFW/KMFW) depends on how many cores the system has.
The licensed cores do not play a role for UMFW/KMFW.

R80.30 kernel 2.6                                         -> UMFW is disabled                                                (checked on VMWare with 30 cores and with 46 cores)

Now we have generated traffic on a FW in the UMFW with a traffic generator. Approx. 500 MBit throughput (100 connections) and see there, the process fwk0_dev_0 generates 321% CPU utilization. I see this as problematic, because some cores with 80% CPU load were seen during the tests.

I think the R&D should say something about this topic if it is a problem or not ...so if you could get a straight answer from TAC on this it would be appreciated too...

Tags (1)
Highlighted
Ruby

Tags (1)
Highlighted
Gold

Re: High CPU utilization during process fwk0_dev_0

I agree with @HeikoAnkenbrand . These high CPU utilization could be a problem.

We had around 500 - 700Mbit Throughput on a 8 core machine, all workers are everytime utilized around 40-70%.

If we disable all blades ( ThreatPrevention, APCL, AV, AB etc.) except firewall, changes are only to 35-65%.
Same system with R77.30 and USFW off was on 10-20% CPU utilization.

I‘m really awaiting an answer from TAC.

Wolfgang

Highlighted
Iron

Re: High CPU utilization during process fwk0_dev_0

Same problem:

-HP DL 360 G10

- R80.30 kernel 3.10

- 8 core FW

- 600MBit throughput

- ThreatPrevention, IA, AV, AB, IPS

- USFW enabled

> process fwk0_dev_0 with high CPU  utilization 300%-400%

I‘m really awaiting also an answer from TAC:-)

Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

Right, I don't get how USFW was enabled on your 8-core firewall unless you manually enabled it yourself.  Edit: Are you only licensed for 8 cores but there are more than 40 cores on the system?  The thing is that there WILL be a performance penalty when USFW is enabled due to processing having to come up through the kernel to a fwk process then back down into the kernel, whereas with USFW disabled almost everything except security server process space "trips" are handled down in the kernel which is more efficient.  So unless there are more than 40 cores I don't understand why USFW would get enabled.   If there are more than 40 cores, USFW makes sense as utilizing the extra cores beyond 40 would offset the performance penalty vs. being limited to 40 cores in the kernel with USFW disabled.

Obviously we are waiting for TAC here, but once again in my opinion I don't think the high CPU utilization by fwk0_dev_0 is necessarily a problem.  That process is clearly multi-threaded which explains CPU utilization over 100%.  I've speculated earlier that when USFW is enabled this process may be the liaison between all the fwk Firewall Worker processes, and the state tables which are still stored in the kernel.  When USFW is disabled, everything is down in the kernel and it all just gets lumped in under si or sy CPU cycles and you can't see the individual kernel elements consuming CPU.

When USFW is enabled everything except state table storage and SecureXL is mostly outside the kernel and visible as individual processes, so seeing high CPU utilization by a process may look alarming but I don't think it is an actual problem.  I don't believe the high CPU being consumed by fwk0_dev_0 is a bottleneck, it is just a side effect of fwk process to kernel interaction.  If throughput does seem to be heavily impacted with USFW enabled, there may need to be some usual tuning performed based on the "Super Seven" and such.

Or I could be wrong...

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com
Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

I agree with @Timothy_Hall  here. UMFW should only be enabled automatically if more than 40 cores were active during installation on a R80.30 3.10 kernel.

But maybe your system has more then 40 cores on board.

For a firewall with only 8 active cores (according to license), I would recommend to change the firewall to "Kernel Mode Firewall" otherwise this will produce a performance issue.

Tags (1)
Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

Small note to enable and disable the UMFW!

Switch to Kernel Mode Firewall, do the following:

1) Run the following clish commands:
# cpprod_util FwSetUsFwmachine 0
# cpprod_util FwSetUsermode 0
2) Edit the boot.conf file (vi $FWDIR/boot/boot.conf) with the following: KERN_INSTANCE_NUM 40 3) Reboot. Switch to User Mode Firewall, do the following: 1) Run the following clish commands: # cpprod_util FwSetUsFwmachine 1 # cpprod_util FwSetUsermode 1 2) Edit the boot.conf file (vi$FWDIR/boot/boot.conf) with the following:
KERN_INSTANCE_NUM 62
3) Reboot.

Tags (1)
Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

Quoting @HeikoAnkenbrand here:

> For a firewall with only 8 active cores (according to license), I would recommend to change the firewall to "Kernel Mode Firewall" otherwise this will produce a performance issue.

I'd be careful with that, generally I don't recommend changing whatever the default state of USFW is on your firewall without consulting TAC.  If it is justified to disable USFW because it got mistakenly enabled on an 8-core firewall that is fine, but TAC needs to be involved so they can see how it ended up in that state incorrectly and fix it for future customers.  Same thing if USFW doesn't seem to be enabled when it should be, work with TAC first to make sure you won't do more harm than good by enabling it.

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com
Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

Hi @PhoneBoy@Val_Loukine

Otherwise, I'll open a ticket for tis topic!

Tags (1)
Highlighted
Gold

Re: High CPU utilization during process fwk0_dev_0

I had a ticket open since Monday and I did an update with our findings from here yesterday. Today I pushed the escalation button, but no answer from TAC right now.

Wolfgang

Highlighted

Re: High CPU utilization during process fwk0_dev_0

@HeikoAnkenbrand I do not quite understand what kind of response you expect. UMFW is not supposed to run with less than 40 cores. End of story.

Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

@Val_Loukine @PhoneBoy @Moti @Dorit_Dor

We have many customers who use systems with more than 40 cores. If we generate 300-500% for the mentioned process here at firewalls, we only have 35 cores left. If that's not a problem, I'll give this information to all our customers and write "end of stroy":-)

Very nice solution >>> End of story <<<

Thanks

---

 The original problem

- Customer installed a firewall on a open server with more than 40 cores. This makes the UMFW active during the installation.
- Now he installed the license with 8 cores and use only 8 cores.
- The problem now is, he has a firewall in UMFW (USFM) mode with 8 cores.
- In this situation the process utilization of the process fwk0_dev_0 is 300-500%.

This is a beautiful bug in the installation script for me:-)

When I looked at the customer firewall, I wondered why there is a process with 300-500% CPU usage. Thereupon I have here in the forum asked, what the problem can be? Now @Timothy_Hall answered (thanks to Timothy) and I noticed that the firewall works in User Mode Firewall.Upssssssss!!!

So we had a shredded system, which according to your statement (@Val_Loukine) has no support or is not supported! (Commented and stroke out by VAL: Sorry Heiko, I did not say that)
(Inserted after @Val_Loukine  changes in my text: I actually want to solve the technical problem and not discuss formulations in the forum. To make it clear from my point of view, I was referring to the formulation of VAL: >>>"@HeikoAnkenbrand I do not quite understand what kind of response you expect. UMFW is not supposed to run with less than 40 cores. End of story."<<< Sorry, I accidentally read "not supposed" as "not supported." in VAL's comment. So I've caused some confusion here.)

Unfortunately I did not find this statement in the readme, in the documentation or in a sk of R80.30. Hmmmm!!!

 Solution

Since I didn't get any information from Check Point here in the forum, the customer now has set the firewall to the correct "Kernel Mode FW" in a maintenance window with the following steps:

1) Run the following clish commands:
# cpprod_util FwSetUsFwmachine 0
# cpprod_util FwSetUsermode 0
2) Edit the boot.conf file (vi \$FWDIR/boot/boot.conf) with the following:
KERN_INSTANCE_NUM 40
3) Reboot.

We have composed these commands on the inverted parameters from sk149973.

 But the exciting question is still this

1) Why is the firewall not changed to a Kernel Mode Firewall when a 8 core license is installed. This must be intercepted by the installation script. At least there should be a warning here.

2) Is it possible to use the firewall as UMFW in this situation or is it better to switch the mode to KMFW?

3) If the firewall now works in the wrong mode UMFM "according to your statement unsupported". What should we do at our customers?

4) If we keep running it in the wrong mode! Is that a problem with the prosess fwk0_dev_0.

PS:

- My mistake I should have opened a TAC ticket immediately.

Tags (1)
Highlighted

Re: High CPU utilization during process fwk0_dev_0

@HeikoAnkenbrand Heiko, you state something I did not say.

Please make sure you are either using a correct statement from another comment or a direct quote (doing it for you):

- UMFW is not supposed to run with less than 40 cores

In other words, the system was misconfigured. There is no "not supported" statement in the sentence. BTW, I still do not see any support case here, other than the fact, you are supposed to configure UMFW according to the total amount of licensed cores on you security GW.

Highlighted
Ruby

Re: High CPU utilization during process fwk0_dev_0

@Val_Loukine  sorry, I accidentally read "not supposed" as "not supported." in VAL's comment. So I've caused some confusion here.

Sorry guys! I think I have to bring the beer again.

Highlighted