topic Re: Corexl Problem after Upgrade to R81.20 in Firewall and Security Management

Corexl Problem after Upgrade to R81.20

starmen2000 — Sun, 05 May 2024 11:04:35 GMT

Hi mates,

After upgrading to R81.20, we encountered a following CoreXL issue. Has anybody expereinced like this issue and solved it?

- Despite configuring 4 cores for CoreXL, cpview shows that only one core is utilized for Firewall, while the other 5 cores are used for SND. This results in high CPU usage when this member is set as the active member.

- The output of the fw ctl affinity command indicates that only CPU2 is allocated for all firewall daemons, which is causing the problem. Ideally, all firewall daemons should be distributed across 4 CPUs.

- We attempted to adjust the affinity using the fw ctl affinity -s -k command, which temporarily resolved the issue. However, after a reboot, the configuration reset, and again only 5 CPUs were allocated for SND and 1 for FW.

- We also tried adjusting the fw affinity configuration file to maintain the configuration permanently, but this did not resolve the issue.

Thanks.

Re: Corexl Problem after Upgrade to R81.20

Timothy_Hall — Sun, 05 May 2024 12:57:54 GMT

Are you on a system that does not support Dynamic Split/Balancing? You should be using that if you can, instead of trying to set a static CoreXL split. Only exception would be if you want to do static CPU/core allocations under VSX, outside of that situation trying to introduce manual CoreXL affinities on a modern gateway is quite likely to make performance worse due to how CoreXL, Multi-Queue, and SecureXL are highly interlocked with each other.

Check your licensing, this sounds like the "core crunch" I described in my book:

Re: Corexl Problem after Upgrade to R81.20

the_rock — Sun, 05 May 2024 13:19:52 GMT

I only had this issue happen once and way I solved it was like below:

-run cpconfig

-choose corexl

-disable-

-exit and reboot

-once backup, do same, except re-enable corezl, exit, reboot

Not sure if that may work for you, but there is never need to adjust those things you mentioned in R81.xx, Im positive of that.

I see what @Timothy_Hall is saying, it also could be related to licensing. If what I mentioned fails (if you can doit), maybe try an eval license to see if it makes a difference...just temporarily.

Andy

Re: Corexl Problem after Upgrade to R81.20

starmen2000 — Sun, 05 May 2024 13:25:06 GMT

We resetted the corexl config, but it did not help.

Re: Corexl Problem after Upgrade to R81.20

starmen2000 — Sun, 05 May 2024 13:28:57 GMT

We enabled dynamic balancing, but it seems to be stuck in the initializing status and hasn't progressed. We opened a TAC ticket and are currently awaiting a solution.

By the way, below are the outputs of commands related to licenses and CPU:

grep -c ^processor /proc/cpuinfo
6

fw ctl get int fwlic_num_of_allowed_cores
fwlic_num_of_allowed_cores = 100000

Re: Corexl Problem after Upgrade to R81.20

AmitShmuel — Sun, 05 May 2024 13:38:58 GMT

Any errors observed in $FWDIR/log/dsd.elg or $FWDIR/log/dynamic_split.elg? it should show why it is stuck in initialization.

Re: Corexl Problem after Upgrade to R81.20

genisis__ — Sun, 05 May 2024 14:06:00 GMT

Was the installation in-place, or a clean install? I've found this type of issue when doing in-place upgrades of VSX.

Re: Corexl Problem after Upgrade to R81.20

starmen2000 — Sun, 05 May 2024 14:08:03 GMT

That was update to R81_20 using cpuse, is not vsx.

Re: Corexl Problem after Upgrade to R81.20

the_rock — Sun, 05 May 2024 14:08:36 GMT

Personally, I would work with TAC to get this solved. Seems like pretty serious issue.

Re: Corexl Problem after Upgrade to R81.20

the_rock — Sun, 05 May 2024 14:09:19 GMT

I cant comment on VSX, but never had this problem myself and I upgraded bunch of clusters to R81.20, all of them wth corexl enabled prior to the upgrade.

Re: Corexl Problem after Upgrade to R81.20

starmen2000 — Sun, 05 May 2024 14:11:11 GMT

Already TAC is involved, but still no meaningful progress.

Re: Corexl Problem after Upgrade to R81.20

the_rock — Sun, 05 May 2024 14:13:41 GMT

Can you please send the output of cpconfig for corexl option and also fw ctl affinity -l -r?

Andy

Re: Corexl Problem after Upgrade to R81.20

starmen2000 — Sun, 05 May 2024 14:16:57 GMT

Sure. Secreenshot is below.

Re: Corexl Problem after Upgrade to R81.20

the_rock — Sun, 05 May 2024 14:20:39 GMT

That definitely does not look right, for sure. Can you go through below link see if you tried it already?

Andy

I know its regarding R80.40, but still...

https://community.checkpoint.com/t5/General-Topics/Automatic-sim-affinity-deprecated-in-R80-40/m-p/114698

Re: Corexl Problem after Upgrade to R81.20

starmen2000 — Sun, 05 May 2024 14:23:10 GMT

FWDIR/log/dsd.elg log shows the following.

ds_validate_only_one_instance_per_cpu: instance 0 already affined to CPU 2

ds_get_instances_affinities: there is a cpu with more than one instance affined to it

ds_init_initial_inst_to_cpu: ds_get_instances_affinities failed
ds_init_instances: could not read initial instances to cpus maps
ds_init_mappings: Failed to init instances
ds_init_basic_state: ds_init_mappings failed
ds_init failed

Re: Corexl Problem after Upgrade to R81.20

Chris_Atkinson — Sun, 05 May 2024 14:23:38 GMT

What platform/model is this system, is it supported per sk164155 and which JHF?

Re: Corexl Problem after Upgrade to R81.20

the_rock — Sun, 05 May 2024 14:24:27 GMT

Cluster or single fw?

Andy

Re: Corexl Problem after Upgrade to R81.20

AmitShmuel — Sun, 05 May 2024 14:40:47 GMT

Seems like a CoreXL configuration issue.

Please try:

Remove any lines related to instances affinity in $FWDIR/conf/fwaffinity.conf
Delete $FWDIR/conf/usfw_affinity.conf, if exists
Make sure 'cpprod_util FwIsUsfw' or 'cpprod_util FwIsUserspace' are returning the correct result
Make sure mq_mng is set to auto and try to enable dynamic_split and reboot again

Re: Corexl Problem after Upgrade to R81.20

the_rock — Sun, 05 May 2024 15:27:54 GMT

For the context, @starmen2000 , this is what that file looks like in one of my labs (its master fw, R81.20 jumbo 54)

Andy

[Expert@cpazurecluster1:0]# cat /opt/CPsuite-R81.20/fw1/conf/fwaffinity.conf
# Process / Interface Affinity Settings
# -------------------------------------
#
# Each line shoud contain:
# 1. A type - 1 character. "i" for interface, "n" for process name, "k" for kernel instance.
# 2. An ID - interface name, process name, or kernel instance number.
# a. For interfaces, you can also write "default", and the setting would apply to any interface not
# mentioned in the file.
# 3. The desired affinity. Either:
# a. One or more CPU numbers.
# b. "all" - all CPUs are eligible.
# c. "ignore" - do nothing for this entry.
# d. "auto" - use any free CPU. A free CPU is one that doesn't appear in any line in this file,
# and doesn't run a worker thread.
#
i default auto
[Expert@cpazurecluster1:0]#

Re: Corexl Problem after Upgrade to R81.20

the_rock — Mon, 06 May 2024 12:23:11 GMT

Hey,

Did you manage to get this fixed?

Andy