Default process affinity on R80.30

[Borut]

Hi

I'm noticing some strange CPU usage on our gateway since upgrade to R80.30 3.10 (currently on JHF T50). The hardware was also changed at the same time (HP DL360 Gen10). We have an 8 core license.

The first thing I noticed was the weird CPU allocation for CoreXL and SecureXL. We have the gateway configured with 6 CoreXL cores, and 2 for SecureXL. In previous version (80.10) CPU's 2-7 were assigned for CoreXL and 0 and 1 to SecureXL. Now, the distribution is something I'm not used to.

# fw ctl affinity -l -r
CPU 0: eth8
CPU 1:
CPU 2:
CPU 3: fw_5
mpdaemon fwd wsdnsd usrchkd in.asessiond in.acapd vpnd pepd lpd rad pdpd topod cprid cpd
CPU 4: fw_3
mpdaemon fwd wsdnsd usrchkd in.asessiond in.acapd vpnd pepd lpd rad pdpd topod cprid cpd
CPU 5: fw_1
mpdaemon fwd wsdnsd usrchkd in.asessiond in.acapd vpnd pepd lpd rad pdpd topod cprid cpd
CPU 6:
CPU 7: eth9 eth4
CPU 8:
CPU 9: fw_4
CPU 10: fw_2
CPU 11: fw_0
All:
The current license permits the use of CPUs 0, 1, 2, 3, 4, 5, 6, 7 only.

What has changed in R80.30 that CoreXL is using cores 3,4,5,9,10,11? Is it trying to evenly distribute load between physical CPU's?  Why was SecureXL affinity set to cores 0 and 7 by default (why not 0 and 6)? I configured manual affinity using the default values.

Also, CoreXL instances don't seem to balance load as nicely as in R80.10.

This was the CPU usage in R80.10:

And this is R80.30

As you can see cores 3-5 are utilized much more than cores 9-11. I presume this is because default affinity for processes is set only to cores 3-5. Why is that? Is that a feature or a bug? 

Is there a simple way to persuade the process affinity to be on all CoreXL cores without manually setting the affinity in fwaffinity.conf for every possible process?

Best regards

[G_W_Albrecht]

- According to sk158112 Unable to assign FWD to a specific core, R80.30 has a CoreXL bug resolved only in Jumbo Hotfix Accumulator for R80.30 from Take 76 !

-  Your screens shows 12 cores, but only 8 are licensed - did you follow Performance Tuning R80.30 Administration Guide ?

- I would also suggest to consult CoreXL section in the sk98348 (Best Practices - Security Gateway Performance) and to look into sk153832: ATRG: SecureXL for R80.20 and above 

- and sk153373: New Multi-Queue Management for Check Point R80.30 with Gaia 3.10 kernel

[Borut]

Hi G_W_Albrecht

- sk158112 talks about machines with SMT enabled. We have Hyperthreading disabled.

- Yes, we have a 12 core (2 physical  CPU's with 6 cores each) machine and a license for 8 cores. Nothing wrong with that. The performance is not the issue, just the weird core allocation for licensed cores and the fact that process affinities are not assigned to all CoreXL cores by default as was the case in versions lower than R80.30 (see my reply to Timothy).

- Thanks for suggesting performance optimization guides. I'm familiar with them but they don't apply with the problem I'm having. I'm aware of the procedures regarding changing process affinities, I'm just a little confused why the defaults have changed (not assigning process affinities to all CoreXL cores by default)

- Multiqueue is disabled on this specific gateway

Thanks for your answer

[Timothy_Hall]

Looks like you have six physical CPUs with Hyperthreading enabled.  So your cores shake out like this:

Core 0/6: SND 1

Core 1/7: SND 2

Core 2/8: Unused?

Core 3/9: 2 workers

Core 4/10: 2 workers

Core 5/11: 2 workers

So the split is 2/6 which falls right into the license limit of 8 cores.  I think your output just looks strange since you have 12 total cores but are licensed for 8.  Normally someone will have 16 cores and be licensed for 8, or have 8 cores and be licensed for 4 and the core numbers are straight multiples of each other.  In that case the output is a little easier to interpret.

Process affinity is on cores 3/4/5.

[Borut]

Hi Timothy

We have an open server with two physical CPU's - Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz. That amounts to 12 physical CPU cores. Hyperthreading is disabled. The license is for 8 cores for budget reasons (and should be enough for what is needed). 

I found another machine with 2x6 physical cores with R80.20 installed with default process affinity settings. SND affinities are set manually.

# fw ctl affinity -l -r
CPU 0:  eth8 eth0 eth1 eth7
CPU 1:  eth9 eth11 eth4 eth5
CPU 2:  fw_5
        rtmd lpd mpdaemon fwd wsdnsd in.asessiond cpd cprid
CPU 3:  fw_4
        rtmd lpd mpdaemon fwd wsdnsd in.asessiond cpd cprid
CPU 4:  fw_3
        rtmd lpd mpdaemon fwd wsdnsd in.asessiond cpd cprid
CPU 5:  fw_2
        rtmd lpd mpdaemon fwd wsdnsd in.asessiond cpd cprid
CPU 6:  fw_1
        rtmd lpd mpdaemon fwd wsdnsd in.asessiond cpd cprid
CPU 7:  fw_0
        rtmd lpd mpdaemon fwd wsdnsd in.asessiond cpd cprid
CPU 8:
CPU 9:
CPU 10:
CPU 11:

 As you can see, the GW is using CPU ID's 0-7 (0-5 are on physical CPU0 and 7-8 are on CPU1) is on and process affinity is distributed among all CoreXL cores by default. fwaffinity.conf file is identical to the one on the R80.30 GW.

Something must have changed in R80.30 regarding which cores the GW is using on a two CPU machines (if licensed cores < available cores). I wonder If that is by design. Maybe they're also trying to share load between physical CPU's or something. I guess assigning process affinity to CPU cores on a second physical CPU is not working by default.

Also, is there a way to change that behavior without manually changing affinity for every single process via the fwaffinity.conf file.

[Martin_Valenta · Accepted solution]

If you have only 8 core license, then disable cores on your physical servers to have only 4 cores per processor.
https://community.checkpoint.com/t5/General-Management-Topics/Open-server-disable-CPU-cores/m-p/2037...

[Borut]

Thank you Martin. We have lowered the active CPU cores to 4 per CPU. The process affinity looks normal now. There is still a difference in CoreXL/SecureXL cores assignment, but that does not bother me 🙂

CPU 0: eth8
CPU 1: fw_5
mpdaemon fwd wsdnsd in.asessiond pdpd vpnd pepd usrchkd in.acapd lpd rad topod cprid cpd
CPU 2: fw_3
mpdaemon fwd wsdnsd in.asessiond pdpd vpnd pepd usrchkd in.acapd lpd rad topod cprid cpd
CPU 3: fw_1
mpdaemon fwd wsdnsd in.asessiond pdpd vpnd pepd usrchkd in.acapd lpd rad topod cprid cpd
CPU 4: eth9 eth4
CPU 5: fw_4
mpdaemon fwd wsdnsd in.asessiond pdpd vpnd pepd usrchkd in.acapd lpd rad topod cprid cpd
CPU 6: fw_2
mpdaemon fwd wsdnsd in.asessiond pdpd vpnd pepd usrchkd in.acapd lpd rad topod cprid cpd
CPU 7: fw_0
mpdaemon fwd wsdnsd in.asessiond pdpd vpnd pepd usrchkd in.acapd lpd rad topod cprid cpd
All: