Solved: Re: 23900, R80.30 upgrade, SMT, and CoreXL

phlrnnr · ‎2019-12-17

After upgrading a cluster of 23900s from R80.20/Jumbo 87 to R80.30/Jumbo 111, I noticed some things I found interesting. This is still running the 2.6 kernel code (I don't believe 3.10 is supported on 23900 until R80.40)

1. While SMT was disabled in R80.20 on the 23900, after upgrading to R80.30, it was enabled by default.

2. The 23900 has 36 cores. With SMT enabled, it has 72 'virtual' cores. By default, once SMT turned on, it created 40 firewall workers in CoreXL (20 cores), It had 16 SND threads (8 cores), and 16 threads (8 cores) were completely idle.

Questions;

Does the 40 core limit for CoreXL still apply? I'm assuming so because it created 40 instances after the upgrade?
Is there any way to utilize the 8 cores / 16 threads that are completely idle?
Is the FW more efficient if SMT were turned off as it would be able to utilize those other cores (just with fewer FW worker processes)?

Here is the output of cpview that shows the breakdown:

cores 0-7 SND
cores 8-15 idle / unused
cores 16-35 fw worker
cores 36-43 SND
cores 44-51 idle / unused
cores 52-71 fw worker

# cpview
|-------------------------------------------------------------------------------------------------|
| CPVIEW.CPU 17Dec2019 15:28:19 |
|-------------------------------------------------------------------------------------------------|
| Overview SysInfo Network CPU I/O Software-blades Hardware-Health Advanced |
|-------------------------------------------------------------------------------------------------|
| Overview Top-Protocols Top-Connections |
|-------------------------------------------------------------------------------------------------|
| Host |
|-------------------------------------------------------------------------------------------------|
| CPU: |
| |
| CPU User System Idle I/O wait Interrupts |
| 0 0% 16% 84% 0% 485,534 |
| 1 0% 16% 84% 0% 485,629 |
| 2 0% 16% 85% 0% 485,705 |
| 3 0% 12% 88% 0% 485,811 |
| 4 0% 11% 89% 0% 485,919 |
| 5 0% 15% 86% 0% 486,003 |
| 6 0% 11% 89% 0% 486,095 |
| 7 0% 11% 89% 0% 486,172 |
| 8 0% 0% 100% 0% 486,277 |
| 9 0% 0% 100% 0% 486,382 |
| 10 0% 0% 100% 0% 486,462 |
| 11 0% 0% 100% 0% 486,538 |
| 12 0% 0% 100% 0% 486,620 |
| 13 0% 0% 100% 0% 486,707 |
| 14 0% 0% 100% 0% 486,786 |
| 15 0% 0% 100% 0% 486,854 |
| 16 2% 9% 88% 0% 486,932 |
| 17 3% 8% 89% 0% 487,024 |
| 18 0% 10% 90% 0% 487,097 |
| 19 0% 7% 93% 0% 487,164 |
| 20 0% 9% 90% 0% 487,252 |
| 21 0% 7% 93% 0% 487,343 |
| 22 0% 7% 92% 0% 487,420 |
| 23 0% 8% 92% 0% 487,524 |
| 24 0% 16% 85% 0% 487,618 |
| 25 0% 9% 91% 0% 487,750 |
| 26 0% 10% 90% 0% 487,856 |
| 27 0% 10% 90% 0% 487,939 |
| 28 0% 9% 91% 0% 488,015 |
| 29 0% 9% 91% 0% 488,110 |
| 30 0% 9% 91% 0% 488,200 |
| 31 0% 8% 92% 0% 488,290 |
| 32 0% 10% 90% 0% 488,396 |
| 33 0% 12% 89% 0% 488,538 |
| 34 0% 24% 76% 0% 488,627 |
| 35 5% 19% 76% 0% 977,404 |
| 36 0% 15% 85% 0% 488,790 |
| 37 0% 17% 83% 0% 488,875 |
| 38 0% 15% 86% 0% 488,973 |
| 39 0% 13% 88% 0% 489,055 |
| 40 0% 12% 88% 0% 489,138 |
| 41 0% 11% 89% 0% 489,229 |
| 42 0% 11% 89% 0% 133 |
| 43 0% 14% 86% 0% 236 |
| 44 0% 0% 100% 0% 335 |
| 45 0% 0% 100% 0% 423 |
| 46 0% 0% 100% 0% 375 |
| 47 0% 0% 100% 0% 93 |
| 48 0% 0% 100% 0% 156 |
| 49 0% 0% 100% 0% 240 |
| 50 0% 0% 100% 0% 359 |
| 51 0% 0% 100% 0% 454 |
| 52 1% 10% 90% 0% 1,081 |
| 53 0% 8% 92% 0% 623 |
| 54 0% 11% 89% 0% 707 |
| 55 0% 10% 90% 0% 808 |
| 56 0% 8% 92% 0% 880 |
| 57 0% 7% 93% 0% 960 |
| 58 1% 10% 90% 0% 1,066 |
| 59 1% 12% 88% 0% 1,178 |
| 60 0% 16% 84% 0% 1,244 |
| 61 0% 11% 89% 0% 1,321 |
| 62 0% 11% 89% 0% 1,401 |
| 63 0% 13% 86% 0% 1,480 |
| 64 0% 9% 91% 0% 3,111 |
| 65 0% 9% 91% 0% 1,636 |
| 66 0% 14% 86% 0% 1,695 |
| 67 1% 11% 88% 0% 3,560 |
| 68 0% 9% 91% 0% 1,868 |
| 69 1% 10% 89% 0% 3,926 |
| 70 2% 12% 86% 0% 2,072 |
| 71 2% 15% 83% 0% 2,165

Timothy_Hall · ‎2019-12-18

The 40 core limit for CoreXL will apply regardless of kernel version (2.6.18 vs. 3.10) if USFW is disabled. It appears that USFW will not be enabled in R80.30 by default unless you are using the 3.10 kernel, but it can be manually switched on under the 2.6.18 kernel. We are in a bit of a transition to a new Gaia kernel in R80.30, this should get cleared up by R80.40 which will supposedly require the 3.10 kernel.

SMT must have originally been enabled in the BIOS but disabled by cpconfig, not sure why it would get switched on like that after the upgrade.

My guess is that you had an 8/28 split when SMT was in a disabled state prior to the upgrade. When SMT got enabled by the upgrade process, it tried to double the CoreXL split to 16/56, but hit the 40 core limit due to USFW not being enabled (because you are using the 2.6.18 kernel), resulting in a 16/40 split and 16 unused cores.

You could manually enable USFW to take advantage of all 72 cores under kernel 2.6.18, but in general I do not recommend changing whatever the default state of USFW is without consulting TAC first.

As to whether turning SMT back off will be more efficient that depends. What does fwaccel stats -s show? If more than 70% of your traffic is fully accelerated (not likely), then disabling SMT will definitely help.

In your specific case under R80.30 kernel 2.6.18, USFW did not get enabled by the upgrade due to the use of the 2.6.18 kernel; at least for the moment unless you manually enable USFW (check with TAC first!) it would probably be better to disable SMT.

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com

View solution in original post

HeikoAnkenbrand · ‎2019-12-18

I wrote this in another article: High CPU utilization during process fwk0_dev_0

According to SK the UMFW is enabled from R80.30 by default. I'm afraid I noticed that a little differently. To confirm this I called a friend (He's a HP dealer.) and asked him if he had a HP DL380 with more then 40 cores in his company:-) Two hours later we were sitting in his LAB and installed R80.30 on this system.

R80.30 kernel 3.10 more then 40 cores -> UMFW is enabled (checked on HP DL 380 G10 2 * Platinum 8180MProcessor 28 cores = 56 cores)
R80.30 kernel 3.10 less then 40 cores -> UMFW is disabled (checked on HP DL 380 G10 1 * Platinum 8180MProcessor 28 cores)

Unfortunately I was not allowed to take the server with 56 cores home with me 🤣.

The installation under R80.30 (UMFW/KMFW) depends on how many cores the system has.
The licensed cores do not play a role for UMFW/KMFW.

R80.30 kernel 2.6 -> UMFW is disabled (checked on VMWare with 30 cores and with 46 cores)

...

➜ CCSM Elite, CCME, CCTE ➜ www.checkpoint.tips

View solution in original post

PhoneBoy · ‎2019-12-17

UserMode FW is enabled by default in R80.30.
This lifts the CoreXL 40 core limit and allows SMT to be enabled.
I would confirm boot.conf is configured as described here:
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

phlrnnr · ‎2019-12-17

Apparently the upgrade process enabled SMT, but not USFW:

# cpprod_util FwIsUsermode
0

# cat $FWDIR/boot/boot.conf
CTL_IPFORWARDING 1
DEFAULT_FILTER_PATH /etc/fw.boot/default.bin
KERN_INSTANCE_NUM 40
COREXL_INSTALLED 1
KERN6_INSTANCE_NUM 0
CORE_OVERRIDE 72
IPV6_INSTALLED 0

Suggestions? I don't typically like modifying these types of parameters that could then get over-written or lost during an upgrade process, or need to be re-configured if a box gets RMA'd.

Maarten_Sjouw · ‎2019-12-17

Some of these settings are not updated when you upgrade but are default when you do a clean install, a good example of this would be CCP method, once it has been changed in a previous version ie to broadcast, after the upgrade it will not be changed to Unicast but remains on Broadcast. Default value is now Automatic.

Regards, Maarten

PhoneBoy · ‎2019-12-17

You basically have two options:
1. Disable SMT
2. Fully enable USFW per sk149973.
Like Maarten says, a fresh install of R80.30 should set these parameters correctly.
Why it didn't get fully adjusted on upgrade, not sure.

Timothy_Hall · ‎2019-12-18