High CPU utilization during process fwk0_dev_0 (UMFW vs. KMFW)

[HeikoAnkenbrand]

What can be the problem on a 16 core open server R80.30 if the following process fwk0_dev_0 is at 300%-400% high CPU utilization. This prozess "fwk0_dev" indicates VSX in older versions. VSX is not enabled on this system.

What does this prozess do?

Who can help here?

# top

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 19437 admin      0 -20 18.402g 0.015t 538544 S 309.9  1.0   8626:21 fwk0_dev_0
 19831 admin     20   0  755376 186344  41628 S   5.6  0.0 459:42.51 fw_full

 # pstree -l |grep -A 15 -B 15 fwk0_dev_0

-cprid_wd---cprid
     |-cpwd-+-AutoUpdaterServ-+-AutoUpdater---22*[{AutoUpdater}]
     |      |                 `-sleep
     |      |-DAService_scrip-+-DAService---5*[{DAService}]
     |      |                 `-sleep
     |      |-avi_del_tmp_fil---sleep
     |      |-ci_http_server
     |      |-cpd---4*[{cpd}]
     |      |-cphamcset
     |      |-cpview_services
     |      |-cpviewd---{cpviewd}
     |      |-4*[dlpu]
     |      |-fw_full-+-in.acapd---{in.acapd}
     |      |         |-in.asessiond---{in.asessiond}
     |      |         |-in.msd---3*[{in.msd}]
     |      |         |-stormd
     |      |         |-ted---temain---46*[{temain}]
     |      |         |-usrchkd
     |      |         |-vpnd---3*[{vpnd}]
     |      |         `-10*[{fw_full}]
     |      |-fwk_forker---fwk0_dev_0---33*[{fwk0_dev_0}]
     |      |-fwk_wd
     |      |-lpd

[_Val_]

UM FW enabled?

[Timothy_Hall]

Looks like USFW is enabled due to the presence of the fwk_forker process. 

Edit: My guess as to the purpose of the fwk0_dev_0 is that it acts as the liaison between the multiple fwk Firewall Worker processes and the single fwmod kernel driver instance; note that its process priority is jacked up to the maximum possible (-20).

I use the lsmod command to determine if USFW is enabled like this, which also provides insight as to the implementation of USFW:

[HeikoAnkenbrand]

Hi @Timothy_Hall,

thanks for your answer. That's what I think too.

I think someone from R&D should take a look at this.

The problem is that this process generates a very high CPU load. 300%-400% of 1600% (16 cores *100%) are used by this process. This is very high for me!

On the firewall, the CPU load of the firewall workers  is otherwise very low 1-5% utilization per core.

Please can someone from R&D say something about this CPU utilization?

Is that normal for fwk0_dev_0 process?

Alternatively I can also open a TAC case.

[HeikoAnkenbrand]

Hi @Timothy_Hall, @_Val_ 

I was able to clear up some of the issue.

The open server was installed incorrectly. During the installation 64 cores were visible on the gateway because HT was activated. I think the person who installed the FW didn't know that HT is not supported on an open server:-)

Due to the many cores, the FW was activated in "User Mode". When I do this in the lab, I see a very high utilization of the process fwk0_dev_0 in "User Mode". If I switch the same firewall to "Kernel Mode" I can't find this process anymore.

I will set the productive firewall in "Kernel Mode" in the maintenance window.

But the question is , why does the process generate such high CPU utilization in "User Mode FW"?

[_Val_]

Most probably it was configured incorrectly in the first place. I see no case for support here.

[HeikoAnkenbrand]

Correct 👍

Thanks to all

[HeikoAnkenbrand]

Hi @Timothy_Hall,

To make sure that USFW is activated, run the following command:

# cpprod_util FwIsUsermode

1 = User Mode Firewall
0 = Kernel Mode Firewall

If necessary, you can also add it to the third edition of your book.

[Timothy_Hall]

Yeah but I like the lsmod way better. 😁

Book? Not familiar with that obscure word...

[HeikoAnkenbrand]

👍

[Wolfgang]

Hello @all,

very very interesting thread......

I can see the same behaviour on a customer system ( open server )  with 8 cores.

We had a TAC case open  but with no results at the moment. 

"cpprod_util FwIsUsermode" shows "1"

I don't know how about of hyperthreading at the install (customer did this). 

Can we change usermode to kernel mode on the fly with only changing "cpprod_util FwIsUsermode" ?

Does it work on ClusterXL without interruption ?

Wolfgang

[Timothy_Hall]

Generally I don't recommend changing the state of USFW from its default without consulting TAC.  If it is enabled by default I'd leave it enabled as Check Point must feel that it would provide a tangible benefit.  It seems that if the 3.10 kernel is present and you are using R80.30, USFW will generally be enabled, but the rules about when USFW is enabled vs. disabled by default seem to be pretty complex.  I think the complexity concerning USFW being enabled/disabled by default stems from the dual kernels (2.6.18 vs. 3.10) available for R80.30, hopefully this will get clarified in R80.40 which supposedly will require use of the 3.10 kernel.

To sum up, I don't think there is any cause for alarm about fwk0_dev_0 eating lots of CPU, it is just a side effect of USFW being enabled, which causes most firewall CPU utilization to occur in process/user space instead of in the kernel.

Changing the state of USFW requires a reboot, the instructions are here: sk149973: How to enable USFW (User-Space Firewall) on a 23900 appliance

[Wolfgang]

I'll be waiting for response from TAC.....

The behaviour is really strange. We had 4 open servers, all installed with R80.30.

2 have USFW on, 2 not. Same hardware model, only different CPU cores. 

We did the same install routine on all servers.

Why is it on on 2 of the servers and on the others not ? (I'll hope, TAC has an answer)

Wolfgang

[Timothy_Hall]

I've seen the same, to date I have not been able to get a clear answer other than "it depends" as far as whether USFW will be enabled by default or not.  What I've been able to figure out so far is that if kernel 2.6.18 is in use, USFW will not be enabled by default upon install or upgrade, but it can be manually enabled.  If the 3.10 kernel is in use it definitely seems to get enabled on firewalls with 40+ cores, and possibly on smaller ones.  Beyond that I'm not sure...so if you could get a straight answer from TAC on this it would be appreciated...

How are the cores different between your firewalls that have USFW enabled vs. those that do not?

[HeikoAnkenbrand]

According to SK the UMFW is enabled from R80.30 by default. I'm afraid I noticed that a little differently. To confirm this I called a friend (He's a HP dealer.) and asked him if he had a HP DL380 with more then 40 cores in his company:-) Two hours later we were sitting in his LAB and installed R80.30 on this system.

R80.30 kernel 3.10 more then 40 cores -> UMFW is enabled    (checked on HP DL 380 G10 2 * Platinum 8180MProcessor 28 cores = 56 cores)
R80.30 kernel 3.10 less then 40 cores   -> UMFW is disabled    (checked on HP DL 380 G10 1 * Platinum 8180MProcessor 28 cores)

Unfortunately I was not allowed to take the server with 56 cores home with me 🤣.

The installation under R80.30 (UMFW/KMFW) depends on how many cores the system has.
The licensed cores do not play a role for UMFW/KMFW.

R80.30 kernel 2.6                                         -> UMFW is disabled                                                (checked on VMWare with 30 cores and with 46 cores)

Now we have generated traffic on a FW in the UMFW with a traffic generator. Approx. 500 MBit throughput (100 connections) and see there, the process fwk0_dev_0 generates 321% CPU utilization. I see this as problematic, because some cores with 80% CPU load were seen during the tests.

I think the R&D should say something about this topic if it is a problem or not ...so if you could get a straight answer from TAC on this it would be appreciated too...

[HeikoAnkenbrand · Accepted solution]

More read here:

- R80.x - Performance Tuning Tip – User Mode Firewall vs. Kernel Mode Firewall  

[Wolfgang]

I agree with @HeikoAnkenbrand . These high CPU utilization could be a problem.

We had around 500 - 700Mbit Throughput on a 8 core machine, all workers are everytime utilized around 40-70%.

If we disable all blades ( ThreatPrevention, APCL, AV, AB etc.) except firewall, changes are only to 35-65%.
Same system with R77.30 and USFW off was on 10-20% CPU utilization.

I‘m really awaiting an answer from TAC.

Wolfgang

[Jul_Kapendale]

Same problem:

-HP DL 360 G10

- R80.30 kernel 3.10

- 8 core FW

- 600MBit throughput

- ThreatPrevention, IA, AV, AB, IPS

- USFW enabled

> process fwk0_dev_0 with high CPU  utilization 300%-400%

I‘m really awaiting also an answer from TAC:-)

[Timothy_Hall]

Right, I don't get how USFW was enabled on your 8-core firewall unless you manually enabled it yourself.  Edit: Are you only licensed for 8 cores but there are more than 40 cores on the system?  The thing is that there WILL be a performance penalty when USFW is enabled due to processing having to come up through the kernel to a fwk process then back down into the kernel, whereas with USFW disabled almost everything except security server process space "trips" are handled down in the kernel which is more efficient.  So unless there are more than 40 cores I don't understand why USFW would get enabled.   If there are more than 40 cores, USFW makes sense as utilizing the extra cores beyond 40 would offset the performance penalty vs. being limited to 40 cores in the kernel with USFW disabled.

Obviously we are waiting for TAC here, but once again in my opinion I don't think the high CPU utilization by fwk0_dev_0 is necessarily a problem.  That process is clearly multi-threaded which explains CPU utilization over 100%.  I've speculated earlier that when USFW is enabled this process may be the liaison between all the fwk Firewall Worker processes, and the state tables which are still stored in the kernel.  When USFW is disabled, everything is down in the kernel and it all just gets lumped in under si or sy CPU cycles and you can't see the individual kernel elements consuming CPU. 

When USFW is enabled everything except state table storage and SecureXL is mostly outside the kernel and visible as individual processes, so seeing high CPU utilization by a process may look alarming but I don't think it is an actual problem.  I don't believe the high CPU being consumed by fwk0_dev_0 is a bottleneck, it is just a side effect of fwk process to kernel interaction.  If throughput does seem to be heavily impacted with USFW enabled, there may need to be some usual tuning performed based on the "Super Seven" and such.

Or I could be wrong...

[HeikoAnkenbrand]

HI @Jul_Kapendale 

I agree with @Timothy_Hall  here. UMFW should only be enabled automatically if more than 40 cores were active during installation on a R80.30 3.10 kernel.

But maybe your system has more then 40 cores on board.

For a firewall with only 8 active cores (according to license), I would recommend to change the firewall to "Kernel Mode Firewall" otherwise this will produce a performance issue.

[HeikoAnkenbrand]

Small note to enable and disable the UMFW!

Switch to Kernel Mode Firewall, do the following:

1) Run the following clish commands:
# cpprod_util FwSetUsFwmachine 0
# cpprod_util FwSetUsermode 0
2) Edit the boot.conf file (vi $FWDIR/boot/boot.conf) with the following:
KERN_INSTANCE_NUM 40
3) Reboot.

Switch to User Mode Firewall, do the following:

1) Run the following clish commands:
# cpprod_util FwSetUsFwmachine 1
# cpprod_util FwSetUsermode 1
2) Edit the boot.conf file (vi $FWDIR/boot/boot.conf) with the following:
KERN_INSTANCE_NUM 62
3) Reboot.

[Nandhakumar]

I have this doubt, what does KERN_INSTANCE_NUM refer here? How do you set to 40 and 62, why not different number? As we are having same issue with our firewalls created in ESXi host and running in R80.40 version, we need to do the steps suggested. Maximum number of allowed cores is 2. I could see firewall is in user mode.

[Timothy_Hall]

Quoting @HeikoAnkenbrand here:

> For a firewall with only 8 active cores (according to license), I would recommend to change the firewall to "Kernel Mode Firewall" otherwise this will produce a performance issue.

I'd be careful with that, generally I don't recommend changing whatever the default state of USFW is on your firewall without consulting TAC.  If it is justified to disable USFW because it got mistakenly enabled on an 8-core firewall that is fine, but TAC needs to be involved so they can see how it ended up in that state incorrectly and fix it for future customers.  Same thing if USFW doesn't seem to be enabled when it should be, work with TAC first to make sure you won't do more harm than good by enabling it.

[HeikoAnkenbrand]

Hi @PhoneBoy, @_Val_ 

Please  the TAC guys should say something about this issue with high CPU utilization on fwk0_dev_0 process.

Otherwise, I'll open a ticket for tis topic!

Thank you in advance.

[Wolfgang]

@HeikoAnkenbrand ,

I had a ticket open since Monday and I did an update with our findings from here yesterday. Today I pushed the escalation button, but no answer from TAC right now. 

I agree with you that someone should give a statement about this behaviour !

Wolfgang

[_Val_]

@HeikoAnkenbrand I do not quite understand what kind of response you expect. UMFW is not supposed to run with less than 40 cores. End of story.

[HeikoAnkenbrand]

@_Val_ @PhoneBoy @Moti @Dorit_Dor 

We have many customers who use systems with more than 40 cores. If we generate 300-500% for the mentioned process here at firewalls, we only have 35 cores left. If that's not a problem, I'll give this information to all our customers and write "end of stroy":-)

Very nice solution >>> End of story <<<

Thanks

---

The original problem

- Customer installed a firewall on a open server with more than 40 cores. This makes the UMFW active during the installation.
- Now he installed the license with 8 cores and use only 8 cores.
- The problem now is, he has a firewall in UMFW (USFM) mode with 8 cores.
- In this situation the process utilization of the process fwk0_dev_0 is 300-500%.

This is a beautiful bug in the installation script for me:-)

When I looked at the customer firewall, I wondered why there is a process with 300-500% CPU usage. Thereupon I have here in the forum asked, what the problem can be? Now @Timothy_Hall answered (thanks to Timothy) and I noticed that the firewall works in User Mode Firewall.Upssssssss!!!

So we had a shredded system, which according to your statement (@_Val_) has no support or is not supported! (Commented and stroke out by VAL: Sorry Heiko, I did not say that)
(Inserted after @_Val_  changes in my text: I actually want to solve the technical problem and not discuss formulations in the forum. To make it clear from my point of view, I was referring to the formulation of VAL: >>>"@HeikoAnkenbrand I do not quite understand what kind of response you expect. UMFW is not supposed to run with less than 40 cores. End of story."<<< Sorry, I accidentally read "not supposed" as "not supported." in VAL's comment. So I've caused some confusion here.)

Unfortunately I did not find this statement in the readme, in the documentation or in a sk of R80.30. Hmmmm!!!

Solution

Since I didn't get any information from Check Point here in the forum, the customer now has set the firewall to the correct "Kernel Mode FW" in a maintenance window with the following steps:

1) Run the following clish commands:
# cpprod_util FwSetUsFwmachine 0
# cpprod_util FwSetUsermode 0
2) Edit the boot.conf file (vi $FWDIR/boot/boot.conf) with the following:
KERN_INSTANCE_NUM 40
3) Reboot.

We have composed these commands on the inverted parameters from sk149973.

But the exciting question is still this

1) Why is the firewall not changed to a Kernel Mode Firewall when a 8 core license is installed. This must be intercepted by the installation script. At least there should be a warning here.

2) Is it possible to use the firewall as UMFW in this situation or is it better to switch the mode to KMFW?

3) If the firewall now works in the wrong mode UMFM "according to your statement unsupported". What should we do at our customers?

4) If we keep running it in the wrong mode! Is that a problem with the prosess fwk0_dev_0.

PS:

- My mistake I should have opened a TAC ticket immediately.

[_Val_]

@HeikoAnkenbrand Heiko, you state something I did not say.

Please make sure you are either using a correct statement from another comment or a direct quote (doing it for you):

- UMFW is not supposed to run with less than 40 cores

In other words, the system was misconfigured. There is no "not supported" statement in the sentence. BTW, I still do not see any support case here, other than the fact, you are supposed to configure UMFW according to the total amount of licensed cores on you security GW.

[HeikoAnkenbrand]

@_Val_  sorry, I accidentally read "not supposed" as "not supported." in VAL's comment. So I've caused some confusion here.

Sorry guys! I think I have to bring the beer again.

[_Val_]

@HeikoAnkenbrand no objections to beers, obviously 🙂