CoreXL option disabled in cpconfig

[Andrew_Kemmy]

Hello Checkmates!

I am hoping that someone has seen this before and can help. 

This was an R77.x cluster and was upgraded "in place". I don't have much history about that but I know the trouble started shortly after the upgrade.

It is now an R80.20 cluster, and still a pair of VMs on VMware ESXi.

cat /proc/cpuinfo tells me each gateway has two virtual CPU cores

I have several issues:

1. No CoreXL option available in cpconfig

2. There is a "per virtual system state" option available in cpconfig

3. CoreXL is running but won't run after a reboot, yet no option in cpconfig to re-enable

4. I have yet to be able to successfully reproduce this state in a lab, and would like to be able to do that before working on the cluster I am describing.

cpconfig
This program will let you re-configure
your Check Point products configuration.

Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State                        <= and note, no option to disable or enable CoreXL
(8) Enable Check Point ClusterXL for Bridge Active/Standby
(9) Automatic start of Check Point Products

(10) Exit

Enter your choice (1-10) :10

Thank You...

See the contents of the file /etc/fw.boot/boot.conf below

cat /etc/fw.boot/boot.conf
CTL_IPFORWARDING 1
DEFAULT_FILTER_PATH /etc/fw.boot/default.bin
KERN_INSTANCE_NUM 1
COREXL_INSTALLED 0
KERN6_INSTANCE_NUM 1
IPV6_INSTALLED 0
CORE_OVERRIDE 2

Also:

> fw ctl multik stat
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 1 | 983 | 33237
1 | Yes | 0 | 6 | 24078

> fw ctl affinity -l -r
CPU 0: eth0
fw_1
CPU 1: eth1 eth2
fw_0
All: mpdaemon lpd fwd in.aclientd in.aftpd in.ahclientd cpd cprid
The current license only permits using CPU 0.

With the configuration file settings  above I cannot reproduce this behaviour in my lab 

- If I make the boot.conf file as above and reboot the "per virtual system state" option appears in cpconfig however in the lab it does not remove the CoreXL option in cpconfig.

I have seen sk62065 however that is for Power Appliance so not sure how much of it applies to VM on Intel.

Value of 

Multik setting in HKLM $CPDIR/registry/HKLM_registry.data

is "[4]1"

I have also gone through most of sk42096, involved 2 TAC cases and consulted our local Check Point SE. 

My questions

1. What configuration options or states for the above system would cause CoreXL option to not exist in cpconfig?

(The system has 2 CPU cores and a perpetual 8 core license so option to enable CoreXL should be available?)

2. If possible, what configuration or state changes need to be made to re-enable CoreXL correctly on this system?

Thanks in advance for any help and apologies if I am missing something obvious!

Cheers

Andrew

[RickHoppe · Accepted solution]

According to this article it might help manually changing the file /etc/fw.boot/boot.conf

Change

COREXL_INSTALLED        0 

into

COREXL_INSTALLED        1

[Andrew_Kemmy]

Hi, thanks for the suggestion and I did see that article before posting.

What I have seen on my test systems is that setting COREXL_INSTALLED to 0 or 1 affects whether CoreXL is enabled or disabled, but it does not affect whether or not the CoreXL option is available in cpconfig.

It seems there is some other factor.

Thanks, Andrew

[Amit_Singh]

Thanks Rick worked for me. @Timothy_Hall  expired licence is issue with VM . In my case I was doing this in lab deleted licence and added a new one changed boot.conf and back in business.  🙂

[PhoneBoy]

How many vCores are assigned to the VMs?

[AK2]

Hi, per original post "cat /proc/cpuinfo" shows two cores. I understand that with one  core there is no Core XL option:)

[PhoneBoy]

The question is what have you assigned in VMware.
In any case, it's debatable how useful CoreXL is with only 2 cores, though it is supported.
I suspect the issue is with your license as Timothy suggests.

[AK2]

For the specific gateway that I have posted about, I don't have access to it in vcenter.

But for other gateways confirmed affected by the same issue, I do, and can confirm they have 2 vCPU's assigned to each gateway VM in vcenter, but no CoreXL option in cpconfig.

So thankfully cpuinfo and vcenter say the same thing - two cores:)

The question of whether two or four or 8 cores is needed for the VM is a whole other ball of wax - it's just that right now we have a set of gateways that we can't use CoreXL on at all.

A colleague suggested increasing the CPU count "to see if it made a difference" however as you say CoreXL is supported and (usually) configurable on two cores.

[Timothy_Hall]

> The current license only permits using CPU 0.

This is the key to your problem, you are only licensed for one core.  Please provide output of cplic print and redact the CK values.

[Andrew_Kemmy]

Hi Tim Hall and thanks,

cplic print output below with CK# removed

It seems to show an 8 core entitlement.

Our local Check Point SE told us that the contracts have expired however I have also been told that this should not prevent CoreXL from working.

cplic print
Host Expiration Features
<IP redacted> never CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT

Contract Coverage:

# ID Expiration SKU
===+===========+============+====================
1 |  | 14May2018 | CPSB-APCL-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
2 |  | 14May2018 | CPSB-AV-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
3 |  | 21May2019 | CPCES-CO-MSP-ADD
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
4 |  | 14May2018 | CPSB-CTNT-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
5 |  | 14May2018 | CPSB-ASPM-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
6 |  | 14May2018 | CPSB-URLF-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
7 |  | 14May2018 | CPSB-IPS-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
8 |  | 14May2018 | CPSB-ABOT-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT

Cheers, Andrew

[AK2]

Hi Tim Hall and thanks,

Info below. Our local Check Point SE told us that the contracts have expired, however I have also been told that an expired contract should not prevent CoreXl from being enabled.

cplic print
Host Expiration Features
<IP redacted> never CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT

Contract Coverage:

# ID Expiration SKU
===+===========+============+====================
1 | | 14May2018 | CPSB-APCL-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
2 | | 14May2018 | CPSB-AV-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
3 | | 21May2019 | CPCES-CO-MSP-ADD
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
4 | | 14May2018 | CPSB-CTNT-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
5 | | 14May2018 | CPSB-ASPM-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
6 | | 14May2018 | CPSB-URLF-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
7 | | 14May2018 | CPSB-IPS-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT
===+===========+============+====================
8 | | 14May2018 | CPSB-ABOT-S-1Y
+-----------+------------+--------------------
|Covers: CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT

Cheers,

Andrew

[AK2]

Hi Tim Hall and thanks,

Info below. Our local Check Point SE told us that the contracts have expired, however I have also been told that an expired contract should not prevent CoreXl from being enabled.

cplic print
Host Expiration Features
<IP redacted> never CPSG-VE+8 CPSB-BASE CPSB-FW CPSM-C-2 CPSB-VPN CPSB-NPM CPSB-LOGS CPSB-IA CPSB-SSLVPN-5 CPSB-ADNC CPSB-IPS-S1 CPSB-URLF CPSB-APCL-S1 CPSB-AV CPSB-ABOT-S CPSB-ASPM CPSB-CTNT

Cheers,

Andrew

[AK2]

For some reason I can't post the contracts. Sorry for delay with this post I had to edit it a few times to get it posted. If you need the contract info let me know

[Timothy_Hall]

Correct, expired contracts will not affect the licensing status of CoreXL.

Why are you using a Virtual Edition license (CPSG-VE+8)?  Are you running in VMWare or bare metal hardware?  You may have the wrong license applied.

You should have something like "CPSG-C-X-U" in the SKU list which is your container and enables CoreXL for X processors.  See sk84761: How to read bladed licenses output.

[AK2]

These gateways are virtual machines running on/in VMware.

We have other gateways which are also virtual machines on the same VMware environment with licences of the form CPSG-VE+ on which the CoreXL option is available in cpconfig.

Thanks for the tip about interpreting the license strings.

[Timothy_Hall]

Are you using site to site VPNs in traditional mode on this gateway?  If so you will not see a VPN column in your firewall/Network Policy Layer, and see Encrypt/Decrypt actions instead.

[AK2]

Hi Tim, there is a VPN column in the firewall/network policy layer. Cheers, Andrew 

[AK2]

I would also like to add that I have been unable to reproduce the issue by doing the following:

1. Take original R77.30 system backup taken before the R80.20 upgrade and restore it to a 2 core R77.30 VM patched to Take 340

2. Do in in-place CPUSE upgrade to R80.20

3. Patch up to Take 87 which is what the problematic system is on.

Resulting system behaves as expected.

What is odd is that 8 gateways across four different clusters which were upgraded at the same time are affected by this issue. 

[AK2]

I have tried a few approaches today.

Mainly what puzzles me is what files are related to this issue

1. installed the completely unsupported strace binary and ran strace on cpconfig

2. Checked vmalloc settings in /boot/grub/grub.conf. The problematic systems have 274M vmalloc - migrated this setting to test system however does not affect cpconfig options

3. Checked a few other files e.g. fwaffinity.conf - nothing unusual in there

4. Did a diff on the $CPDIR/registry/HKLM_registry.data file between the problematic system and my test system and found this:

: (Licensing
:AccountId (redacted)
:PkgDescription ("VEN Gateway")
:ContainerCK (redacted)
:CKSignature (redacted)
:ContainerSKU (CPSG-VEN-NGTP-License)
:SupportLevel ("Collaborative Enterprise Support - MSP Add-on")
:SupportExpiration (redacted)
:HasLicenseActivationStatus (2)
:ActivationStatus (2)

None of the above was on the test system.

When I search "CPSG-VEN-NGTP-License" 

https://sc1.checkpoint.com/uc/pdf/pricelist/Check%20Point%20vSEC%20Elastic%20Licensing.pdf

CPSG-VEN-NGTP-GW         vSEC Gateway for 1 Virtual Core.

Could this be related? 

Cheers,

Andrew

[Timothy_Hall]

Like I said earlier it seems to be related to your license.  Acquire a 30-day evaluation license and apply it, and I can pretty much guarantee that CoreXL will become available after a reboot.

[Wolfgang]

How about changing boot.conf, have you tried:

KERN_INSTANCE_NUM 1
COREXL_INSTALLED 0

to

KERN_INSTANCE_NUM 2
COREXL_INSTALLED 1

I agree with @Timothy_Hall @that the license is your main problem. VE license are for systems like you run, but maybee  something changed with the new release.

Wolfgang

[Robert_Siimon]

Hello Andrew and all others,

this is a bug. Have a look at https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut... and perhaps applying this ongoing jumbo HF will solve the problem.