topic Re: R80.x - Performance Tuning Tip – User Mode Firewall vs. Kernel Mode Firewall in General Topics

R80.x - Performance Tuning Tip – User Mode Firewall vs. Kernel Mode Firewall

HeikoAnkenbrand — Fri, 22 May 2020 17:23:56 GMT

What is a User Mode Firewall?

In “Kernel Mode Firewall” KMFW, the maximum number of running cores is limited to 40 because of the Linux/Intel limitation of 2GB kernel memory, and because CoreXL architecture needs to load a large driver (~42MB) dozens of times (according to the CPU number, and up to 40 times). Newer platforms that contain more than 40 cores e.g., 23900 or open server are not fully utilized.

The solution of the problem is a firewall in the user mode of the Linux operating system.

USFW “User Space Firewall” or UMFW stands for “User Mode Firewall”, and it is based on proven VSX code. This mode was introduced in R80.10.

According to SK the UMFW is enabled from R80.30 by default and is customized via the installation process. To confirm this I called a friend (He's a HP dealer.) and asked him if he had a HP DL380 with more then 40 cores in his company:-) Two hours later we were sitting in his LAB and installed R80.30 on this system. If the info should not be correct, please small info to me, then I change that in the article.

Result:

GAIA version/ Kernel/ Cores	Firewall mode	Check
R80.30 kernel 3.10 more then 35* cores	UMFW is enabled	checked on HP DL 380 G10 2 * Platinum 8180MProcessor 28 cores = 56 cores
R80.30 kernel 3.10 less then 35* cores	KMFW is enabled	checked on HP DL 380 G10 1 * Platinum 8180MProcessor 28 cores
R80.30 kernel 2.6	KMFW is enabled	checked on VMWare with 30 cores and with 46 cores
R80.40 (default 3.10 kernel)	UMFW is enabled by default	checked on VMWare with 4 cores

*) It could be 40 cores. We are in the middle of a discussion on this topic. Read more here:
High CPU utilization during process fwk0_dev_0 (UMFW vs. KMFW)

Threads of process fwk0_dev_0

From a performance point of view I could not see any differences between UMFW and KMFW. I noticed that the process fwk0_dev_0 generates a very high CPU load in the UMFW. My guess as to the purpose of the fwk0_dev_0 is that it acts as the liaison between the multiple fwk firewall worker processes (fw instance thread that takes care for the packet processing) and the single fwmod kernel driver instance and the process for high priority cluster thread.

If you want to change the mode from UMFW to KMFW this can be done by changing the registry parameter FwIsUsermode by cpprod_util command. In UMFW the fw instances are threads of the fwk0_dev_0 so by default the top shows all the threads cpu utilization under the main thread. Top has the option to present the utilization per thread as well.

A small calculation sample for the utilization of process fwk0_dev_0:

                                 max_CoreXL_number            max_CoreXL_number
fwk0_dev_0 =   ∑       fwk0_x    +      ∑         fwk0_dev_x    + fwk0_kissd + fwk0_hp
                                 x=0                                    x=0

Thread from process fwk0_dev_0:

- fwk0_X -> fw instance thread that takes care for the packet processing
- fwk0_dev_X -> the thread that takes care for communication between fw instances and other CP daemons
- fwk0_kissd -> legacy Kernel Infrastructure (obsolete)
- fwk0_hp -> (high priority) cluster thread

Note:
UMFW is not supposed to run with less than 35 cores in R80.10, R80.20 and R80.30

R80.30

In R80.30 kernel 30.10 open servers always load in USFW mode. If the open server has less than 35 fw instances it’s safe to move to kernel mode even on R80.30 with kernel 3.10.

The number of fw instances is derived from the number of cores on the server and the number of core defined by the license.

R80.40+

With R80.40 EA the UMFW is always active by default on kernel 3.10.

Tip

Tip 1 - To make sure that UMFW is activated, run the following command

To make sure that UMFW is activated, run the following command:

# cpprod_util FwIsUsermode

1 = User Mode Firewall
0 = Kernel Mode Firewall

Tip 2 - enable or disable the “User Mode Firewall”

Follow sk149973

Tip 3 - Switch to Kernel Mode Firewall, do the following

Note:
UMFW is not supposed to run with less than 40 cores in R80.10, R80.20 and R80.30

1) Run the following clish commands:
    # cpprod_util FwSetUsFwmachine 0
    # cpprod_util FwSetUsermode 0
2) Edit the boot.conf file (vi $FWDIR/boot/boot.conf) with the following:
    KERN_INSTANCE_NUM 40
3) Reboot.

Tip 4 - Switch to User Mode Firewall, do the following

1) Run the following clish commands:
    # cpprod_util FwSetUsFwmachine 1
    # cpprod_util FwSetUsermode 1
2) Edit the boot.conf file (vi $FWDIR/boot/boot.conf) with the following:
    KERN_INSTANCE_NUM 62
3) Reboot.

Tip 5 - Show thread

utilization of process

fwk0_dev_0

1) search the prozess ID of process fwk0_dev_0

# top

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10219 admin 0 -20 1070m 449m 134m S 2 24.0 0:17.19 fwk0_dev_0

2) Now check the utilization of the threads:

top -Hbn1 -p 10219

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10219 admin 0 -20 1070m 449m 134m S 0 24.0 0:03.49 fwk0_dev_0
10220 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.00 fwk0_kissd
10436 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.57 fwk0_0
10437 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.64 fwk0_1
10438 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.67 fwk0_2
10439 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.80 fwk0_3
10440 admin RT -20 1070m 449m 134m S 0 24.0 0:00.76 fwk0_hp
10441 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.15 fwk0_dev_1
10442 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.09 fwk0_dev_2
10443 admin 0 -20 1070m 449m 134m S 0 24.0 0:00.09 fwk0_dev_3

Chapter

More interesting articles:

- R80.x Architecture and Performance Tuning - Link Collection
- Article list (Heiko Ankenbrand)
- High CPU utilization during process fwk0_dev_0 (UMFW vs. KMFW)

Re: R80.x - Performance Tuning Tip – User Mode Firewall vs. Kernel Mode Firewall

Jul_Kapendale — Thu, 19 Dec 2019 09:26:14 GMT

Nice information.

I have the problem with high CPU usage of the process fwk0_dev_0.

You can read more in this article:

Re: R80.x - Performance Tuning Tip – User Mode Firewall vs. Kernel Mode Firewall

HeikoAnkenbrand — Mon, 06 Jan 2020 07:59:34 GMT

All informations from article High CPU utilization during process fwk0_dev_0 (UMFW vs. KMFW) added.

Re: R80.x - Performance Tuning Tip – User Mode Firewall vs. Kernel Mode Firewall

RamGuy239 — Wed, 15 Apr 2020 06:56:01 GMT

I've been troubleshooting R80.40 installation on Nutanix AHV and we came across this and decided to disable UMFW and test with KMFW and this made things like SecureXL working.

Something that I found confusing according to the SecureKnowledge article and this post is the part with NUMA:

2) Edit the boot.conf file (vi $FWDIR/boot/boot.conf) with the following:
KERN_INSTANCE_NUM 40

sk149973 is specific for 23900 appliance series so these numbers might makes sense for this appliance model. I'm by no means a CPU and NUMA expert but isn't there supposed to be a correlation between the amount of CPU cores and sockets and the NUMA value?

I have tried to verify on different firewalls and management installations running R80.10-R80.40 and every one of them has a KERN_INSTANCE_NUM and KERN6_INSTANCE_NUM that equals the current CoreXL configuration.

On a management server installation there is no KERN_INSTANCE_NUM or KERN6_INSTANCE_NUM values to be found no matter if it's already running in UMFW or KMFW.

I can't see any logical reason why we should edit this value to be 40 regardless of the hardware at hand? What is the reasoning behind this? To me it seems like we are just blindingly following sk149973 and apply the same values regardless eventhough this specific article is for 23900 appliances only? As this model features: 2x CPUs, 36x physical cores, 72x virtual cores (total) it makes sense to ensure such a high NUMA value but this guide makes it seem like you are supposed to set the value of 40 even on a smaller 4-core firewall?

I can't see any reason why we should be doing this? I have also tested the OVF and qcow2 images that are located within sk158292 on VMware ESXi and Nutanix AHV and one thing they all have in common is that they are all running KMFW and not UMFW even on R80.40. And none of them have a KERN_INSTANCE_NUM 40 within the $FWDIR/boot/boot.conf. They all follow the same pattern and logic where the KERN_INSTANCE_NUM equals the number of CoreXL IPv4 firewall instances and the KERN6_INSTANCE_NUM equals the number of CoreXL IPv6 firewall instances.

Re: R80.x - Performance Tuning Tip – User Mode Firewall vs. Kernel Mode Firewall

HeikoAnkenbrand — Thu, 30 Apr 2020 05:14:23 GMT

Now with R80.40 update.

Re: R80.x - Performance Tuning Tip – User Mode Firewall vs. Kernel Mode Firewall

H_W — Sat, 23 May 2020 06:49:29 GMT

👍🏻

Re: R80.x - Performance Tuning Tip – User Mode Firewall vs. Kernel Mode Firewall

B_P — Mon, 01 Jun 2020 17:34:29 GMT

Hi, what is FwSetUsFwmachine and why is it also being changed? This response from Check Point says only changing FwSetUsermode is necessary.

Re: R80.x - Performance Tuning Tip – User Mode Firewall vs. Kernel Mode Firewall

PhoneBoy — Fri, 26 Feb 2021 18:19:03 GMT

Hi, sk149973 doesn't exist anymore as an SK.
Instead, please link to this SK: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk167052
Note that it is only supported on specific appliances, enabled by default on some, can be enabled on others.
It should only be manually enabled on consultation with TAC.