CoreXL Causes R80.10 VEN GW Policy Installation Fail

[Tim_Ireland]

Hi All,

I have deployed R80.10 HA Gateways on VMWare (Private Cloud). Each have 5 vCPUs assigned and are using ClusterXL.

Everything works perfectly with CoreXL disabled, however when I enable CoreXL for 5vCPU's, policy installation starts to fail with a TCP connectivity failure (port = 18191) error no.10. Disabling CoreXL fixes the policy installation failures.

I've been working with TAC over the past 4 weeks and have completely rebuilt my environment to R80.10 base thinking that it was a corruption issue with the Management Appliance or Firewalls, this has now been proven incorrect. Installing the latest Jumbo Hotfix also had no impact on the issue. A top shows that the CPU utilisation is generally no more than 50% on the active gateway member. Typically I can push policy after hours (sometimes it's intermittent), or on weekends when there is little to no traffic going through them.

I have read that it's best practice for environments with Heavy Logging to assign a dedicated CPU to the FWD process. Can you please advise if this could likely be the way to go here?

I attempted to reduce the CoreXL instances to 3 this morning leaving two free (however not specifically assigning FWD to an available one, refer to attachment, however I started to see alerts on some of my NAT rules and sites published by those rules (not all), stopped working. Disabling CoreXL fixed that issue also.

Thanks for your assistance

[Timothy_Hall]

Bit of a shot in the dark here, but I can't recall ever seeing CoreXL enabled on a system with an odd number of total cores.  Can you reduce vCPUs to 4 or increase to 6?  If not try disabling the Dynamic Dispatcher if you have it enabled.

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com.

[Tim_Ireland]

Hi Tim,

Thanks for your response on this.

I ran only 4 vCPU's for 4 weeks originally with the same issues, with this configuration I have 4 CoreXL instances enabled. 

The thing that I'm struggling to understand is that if I have 4vCPU Cores, the documentation states that I get 3 Kernal Instances, as one Core is used for SND. Does this mean that when I enable CoreXL, that I enable only 3 CoreXL Instances in order to leave 1 for SND, or do I enable 4 instances, and the gateway will automatically make one of the cores SND only.

My assumption from reading the doco was that it automatically assigned the SND to 1 core, which is why I enabled CoreXL for 4 instances, when I was running 4 cores. In the case that I now have now, my logic was to assign 5 as it would still assign one to SND automatically. The documentation states that if you have 8 CPU Cores, then two cores will be used for SND, again I thought this would occur automatically by enabling 8 CoreXL instances.

The configuration that I now am thinking would be optimal for me is for 5 CPU Cores

1 x SND

3 x Kernal Instance

1 x FWD dedicated Core

I'm hoping that I can get clarification on both my approach, but also the exact number of instances to enable through CPConfig.

Thanks again.

Tim

[Timothy_Hall]

With 4 total cores on the system, by default you get 3 Firewall Workers (CPUs 1-3) and one SND/IRQ core (CPU 0).  In the CoreXL screen of cpconfig this is represented by 3 "Firewall kernel instances" or some such verbiage.   The Firewall Workers always take the highest numbered cores, and any left over is/are for SND/IRQ.

In general you should avoid setting the overall number of Firewall Worker instances to the same number of physical cores, as this will cause the same core(s) to handle both SND/IRQ and Firewall Worker core functions which is grossly inefficient but unavoidable on 1-core or 2-core systems.

As far as dedicating a core for FWD heavy logging, I'd recommend against it in this case.  Typically this is only done if you have at least 12 total cores if not many more.  Keep in mind that the firewall logging process can jump on any open core that is not busy in R77.30 to get its work done.   However in a major shift for R80.10 gateway, by default all firewall processes are only allowed to jump on an open Firewall Worker core, and may not use the cores assigned to SND/IRQ at all.  I'm assuming this was done to permit the relatively small SND/IRQ routines to take up increased residence in the CPU fast caches of the SND/IRQ cores, and not have those caches get constantly trashed by some wayward process looking for an open CPU.

So for 5 CPUs I'd recommend initially starting with 4 Firewall Worker instances which leaves 1 SND/IRQ core.  However if a large percentage of traffic is fully accelerated by SecureXL or there is heavy small-packet burstiness on your NICs, the single SND/IRQ core will get overrun.  Symptoms of this will be piling up RX-DRPs in netstat -ni, and extreme CPU utilization on core 0 but nowhere else.  In that case decrease Firewall Worker cores from 4 to 3 in cpconfig to allocate another SND/IRQ core.

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com.

[Tim_Ireland]

Hi Tim,

Thanks for your help with this, thats the most concise answer that I've received from anyone to date.

I'll try setting the CoreXL CPU's to 4 and let you know how I go.

Cheers

[Tim_Ireland]

Hi Tim,

Just wanted to let you know that this resolved the issues that I was having with policy installation by configuring 4 CoreXL instances, and leaving one spare for the SND.

Thanks again for your assistance.

Cheers

Tim

[Timothy_Hall]

Great, thanks for the follow up.

--
My book "Max Power: Check Point Firewall Performance Optimization"
now available via http://maxpowerfirewalls.com.