Corexl Problem after Upgrade to R81.20

[starmen2000]

Hi mates,

After upgrading to R81.20, we encountered a following CoreXL issue. Has anybody expereinced like this issue and solved it?

- Despite configuring 4 cores for CoreXL, cpview shows that only one core is utilized for Firewall, while the other 5 cores are used for SND. This results in high CPU usage when this member is set as the active member.

- The output of the fw ctl affinity command indicates that only CPU2 is allocated for all firewall daemons, which is causing the problem. Ideally, all firewall daemons should be distributed across 4 CPUs.

- We attempted to adjust the affinity using the fw ctl affinity -s -k command, which temporarily resolved the issue. However, after a reboot, the configuration reset, and again only 5 CPUs were allocated for SND and 1 for FW.

- We also tried adjusting the fw affinity configuration file to maintain the configuration permanently, but this did not resolve the issue.

Thanks.

[Timothy_Hall]

Are you on a system that does not support Dynamic Split/Balancing?  You should be using that if you can, instead of trying to set a static CoreXL split.  Only exception would be if you want to do static CPU/core allocations under VSX, outside of that situation trying to introduce manual CoreXL affinities on a modern gateway is quite likely to make performance worse due to how CoreXL, Multi-Queue, and SecureXL are highly interlocked with each other.

Check your licensing, this sounds like the "core crunch" I described in my book:

[starmen2000]

We enabled dynamic balancing, but it seems to be stuck in the initializing status and hasn't progressed. We opened a TAC ticket and are currently awaiting a solution.

By the way, below are the outputs of commands related to licenses and CPU:

grep -c ^processor /proc/cpuinfo
6

fw ctl get int fwlic_num_of_allowed_cores
fwlic_num_of_allowed_cores = 100000

[AmitShmuel]

Any errors observed in $FWDIR/log/dsd.elg or $FWDIR/log/dynamic_split.elg? it should show why it is stuck in initialization.

[starmen2000]

FWDIR/log/dsd.elg log shows the following.

ds_validate_only_one_instance_per_cpu: instance 0 already affined to CPU 2

ds_get_instances_affinities: there is a cpu with more than one instance affined to it

ds_init_initial_inst_to_cpu: ds_get_instances_affinities failed
ds_init_instances: could not read initial instances to cpus maps
ds_init_mappings: Failed to init instances
ds_init_basic_state: ds_init_mappings failed
ds_init failed

[the_rock]

Cluster or single fw?

Andy

[AmitShmuel]

Seems like a CoreXL configuration issue.

Please try:

1. Remove any lines related to instances affinity in $FWDIR/conf/fwaffinity.conf
2. Delete $FWDIR/conf/usfw_affinity.conf, if exists
3. Make sure 'cpprod_util FwIsUsfw' or  'cpprod_util FwIsUserspace' are returning the correct result
4. Make sure mq_mng is set to auto and try to enable dynamic_split and reboot again

[the_rock]

For the context, @starmen2000 , this is what that file looks like in one of my labs (its master fw, R81.20 jumbo 54)

Andy

[Expert@cpazurecluster1:0]# cat /opt/CPsuite-R81.20/fw1/conf/fwaffinity.conf
# Process / Interface Affinity Settings
# -------------------------------------
#
# Each line shoud contain:
# 1. A type - 1 character. "i" for interface, "n" for process name, "k" for kernel instance.
# 2. An ID - interface name, process name, or kernel instance number.
# a. For interfaces, you can also write "default", and the setting would apply to any interface not
# mentioned in the file.
# 3. The desired affinity. Either:
# a. One or more CPU numbers.
# b. "all" - all CPUs are eligible.
# c. "ignore" - do nothing for this entry.
# d. "auto" - use any free CPU. A free CPU is one that doesn't appear in any line in this file,
# and doesn't run a worker thread.
#
i default auto
[Expert@cpazurecluster1:0]#

[Chris_Atkinson]

What platform/model is this system, is it supported per sk164155 and which JHF?

[the_rock]

I only had this issue happen once and way I solved it was like below:

-run cpconfig

-choose corexl

-disable-

-exit and reboot

-once backup, do same, except re-enable corezl, exit, reboot

Not sure if that may work for you, but there is never need to adjust those things you mentioned in R81.xx, Im positive of that.

I see what @Timothy_Hall is saying, it also could be related to licensing. If what I mentioned fails (if you can doit), maybe try an eval license to see if it makes a difference...just temporarily.

Andy

[starmen2000]

We resetted the corexl config, but it did not help.

[genisis__]

Was the installation in-place, or a clean install?  I've found this type of issue when doing in-place upgrades of VSX. 

[starmen2000]

That was update to R81_20 using cpuse, is not vsx.

[the_rock]

I cant comment on VSX, but never had this problem myself and I upgraded bunch of clusters to R81.20, all of them wth corexl enabled prior to the upgrade.

[the_rock]

Personally, I would work with TAC to get this solved. Seems like pretty serious issue.

[starmen2000]

Already TAC is involved, but still no meaningful progress.

[the_rock]

Can you please send the output of cpconfig for corexl option and also fw ctl affinity -l -r?

Andy

[starmen2000]

Sure. Secreenshot is below.

[the_rock]

That definitely does not look right, for sure. Can you go through below link see if you tried it already?

Andy

I know its regarding R80.40, but still...

https://community.checkpoint.com/t5/General-Topics/Automatic-sim-affinity-deprecated-in-R80-40/m-p/1...

[the_rock]

Hey, 

Did you manage to get this fixed?

Andy

[starmen2000 · Accepted solution]

Hey,

I think we found the solution. Due to an incorrect registry value of the fwisusfw register, we reset the CoreXL firewall mode from user mode to kernel mode and then back to user mode. After that, it worked, and dynamic balancing also started to function.

[the_rock]

Good to know, thank you, excellent work!

Andy