Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
starmen2000
Collaborator
Collaborator
Jump to solution

Corexl Problem after Upgrade to R81.20

Hi mates,

 

After upgrading to R81.20, we encountered a following CoreXL issue. Has anybody expereinced like this issue and solved it?
 
- Despite configuring 4 cores for CoreXL, cpview shows that only one core is utilized for Firewall, while the other 5 cores are used for SND. This results in high CPU usage when this member is set as the active member.
- The output of the fw ctl affinity command indicates that only CPU2 is allocated for all firewall daemons, which is causing the problem. Ideally, all firewall daemons should be distributed across 4 CPUs.
- We attempted to adjust the affinity using the fw ctl affinity -s -k command, which temporarily resolved the issue. However, after a reboot, the configuration reset, and again only 5 CPUs were allocated for SND and 1 for FW.
- We also tried adjusting the fw affinity configuration file to maintain the configuration permanently, but this did not resolve the issue.
 
Thanks.
0 Kudos
1 Solution

Accepted Solutions
starmen2000
Collaborator
Collaborator

Hey,

I think we found the solution. Due to an incorrect registry value of the fwisusfw register, we reset the CoreXL firewall mode from user mode to kernel mode and then back to user mode. After that, it worked, and dynamic balancing also started to function.

 

View solution in original post

(1)
21 Replies
Timothy_Hall
Legend Legend
Legend

Are you on a system that does not support Dynamic Split/Balancing?  You should be using that if you can, instead of trying to set a static CoreXL split.  Only exception would be if you want to do static CPU/core allocations under VSX, outside of that situation trying to introduce manual CoreXL affinities on a modern gateway is quite likely to make performance worse due to how CoreXL, Multi-Queue, and SecureXL are highly interlocked with each other.

Check your licensing, this sounds like the "core crunch" I described in my book:

crunch.pngcrunch2.png

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
starmen2000
Collaborator
Collaborator

We enabled dynamic balancing, but it seems to be stuck in the initializing status and hasn't progressed. We opened a TAC ticket and are currently awaiting a solution.

By the way, below are the outputs of commands related to licenses and CPU:

grep -c ^processor /proc/cpuinfo
6

fw ctl get int fwlic_num_of_allowed_cores
fwlic_num_of_allowed_cores = 100000

0 Kudos
AmitShmuel
Employee
Employee

Any errors observed in $FWDIR/log/dsd.elg or $FWDIR/log/dynamic_split.elg? it should show why it is stuck in initialization.

0 Kudos
starmen2000
Collaborator
Collaborator

FWDIR/log/dsd.elg log shows the following.

ds_validate_only_one_instance_per_cpu: instance 0 already affined to CPU 2

ds_get_instances_affinities: there is a cpu with more than one instance affined to it

ds_init_initial_inst_to_cpu: ds_get_instances_affinities failed
ds_init_instances: could not read initial instances to cpus maps
ds_init_mappings: Failed to init instances
ds_init_basic_state: ds_init_mappings failed
ds_init failed

0 Kudos
the_rock
Legend
Legend

Cluster or single fw?

Andy

0 Kudos
AmitShmuel
Employee
Employee

Seems like a CoreXL configuration issue.

Please try:

  1. Remove any lines related to instances affinity in $FWDIR/conf/fwaffinity.conf
  2. Delete $FWDIR/conf/usfw_affinity.conf, if exists
  3. Make sure 'cpprod_util FwIsUsfw' or  'cpprod_util FwIsUserspace' are returning the correct result
  4. Make sure mq_mng is set to auto and try to enable dynamic_split and reboot again
the_rock
Legend
Legend

For the context, @starmen2000 , this is what that file looks like in one of my labs (its master fw, R81.20 jumbo 54)

Andy

 

[Expert@cpazurecluster1:0]# cat /opt/CPsuite-R81.20/fw1/conf/fwaffinity.conf
# Process / Interface Affinity Settings
# -------------------------------------
#
# Each line shoud contain:
# 1. A type - 1 character. "i" for interface, "n" for process name, "k" for kernel instance.
# 2. An ID - interface name, process name, or kernel instance number.
# a. For interfaces, you can also write "default", and the setting would apply to any interface not
# mentioned in the file.
# 3. The desired affinity. Either:
# a. One or more CPU numbers.
# b. "all" - all CPUs are eligible.
# c. "ignore" - do nothing for this entry.
# d. "auto" - use any free CPU. A free CPU is one that doesn't appear in any line in this file,
# and doesn't run a worker thread.
#
i default auto
[Expert@cpazurecluster1:0]#

0 Kudos
Chris_Atkinson
Employee Employee
Employee

What platform/model is this system, is it supported per sk164155 and which JHF?

CCSM R77/R80/ELITE
0 Kudos
the_rock
Legend
Legend

I only had this issue happen once and way I solved it was like below:

-run cpconfig

-choose corexl

-disable-

-exit and reboot

-once backup, do same, except re-enable corezl, exit, reboot

Not sure if that may work for you, but there is never need to adjust those things you mentioned in R81.xx, Im positive of that.

I see what @Timothy_Hall is saying, it also could be related to licensing. If what I mentioned fails (if you can doit), maybe try an eval license to see if it makes a difference...just temporarily.

Andy

0 Kudos
starmen2000
Collaborator
Collaborator

We resetted the corexl config, but it did not help.

0 Kudos
genisis__
Leader Leader
Leader

Was the installation in-place, or a clean install?  I've found this type of issue when doing in-place upgrades of VSX. 

0 Kudos
starmen2000
Collaborator
Collaborator

That was update to R81_20 using cpuse, is not vsx.

0 Kudos
the_rock
Legend
Legend

I cant comment on VSX, but never had this problem myself and I upgraded bunch of clusters to R81.20, all of them wth corexl enabled prior to the upgrade.

0 Kudos
the_rock
Legend
Legend

Personally, I would work with TAC to get this solved. Seems like pretty serious issue.

0 Kudos
starmen2000
Collaborator
Collaborator

Already TAC is involved, but still no meaningful progress.

0 Kudos
the_rock
Legend
Legend

Can you please send the output of cpconfig for corexl option and also fw ctl affinity -l -r?

Andy

0 Kudos
starmen2000
Collaborator
Collaborator
 

Sure. Secreenshot is below.2024-05-05 16_14_48-Greenshot image editor.png

 

0 Kudos
the_rock
Legend
Legend

That definitely does not look right, for sure. Can you go through below link see if you tried it already?

Andy

I know its regarding R80.40, but still...

https://community.checkpoint.com/t5/General-Topics/Automatic-sim-affinity-deprecated-in-R80-40/m-p/1...

0 Kudos
the_rock
Legend
Legend

Hey, 

Did you manage to get this fixed?

Andy

0 Kudos
starmen2000
Collaborator
Collaborator

Hey,

I think we found the solution. Due to an incorrect registry value of the fwisusfw register, we reset the CoreXL firewall mode from user mode to kernel mode and then back to user mode. After that, it worked, and dynamic balancing also started to function.

 

(1)
the_rock
Legend
Legend

Good to know, thank you, excellent work!

Andy

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events