Commvault Installer :CPU Spikes on Checkpoint

[avisheen_shetty]

Has anyone faced issue CPU spikes on the checkpoint cluster while downloading Commvault packages.

I am currently facing CPU spikes to almost 100% when i try to download Commvault packages using the installer.

This has been observed on a cluster of 5600 appliances and 5900 appliances as well.

These firewalls have all the blades enabled 

[PhoneBoy]

Are there any log messages that show up around this time?

Have you also engaged the TAC with this? Contact Support | Check Point Software 

[avisheen_shetty]

Nothing specific shows in the log messages . I have engaged the tax since

October . They also faced the same problem . However when they disabled all

the blades except firewall, spike wasn’t seen .

Bypassing the IP address on the IPS blade did not solve the problem in my

environment .

I am still working with the TAC . Hope they provide a fix or a solution .

On Sat, Feb 10, 2018 at 7:42 PM Dameon Welch Abernathy <

[PhoneBoy]

Can you please send me the SR number in a private message?

[Timothy_Hall]

> However when they disabled all the blades except firewall, spike wasn’t seen .  Bypassing the IP address on the IPS blade did not solve the problem in my environment .

This is significant.  What does output of enabled_blades show when run on the firewall?  Defining an IPS Exception won't change the path (SXL/PXL/F2F) the Commvault traffic takes through the firewall, but disabling all blades other than Firewall will.  Try this on the gateway:

ips off

(try commvault traffic, and if still causing CPU spike)

fw amw unload

(try commvault traffic again)

ips on

fw amw fetch local

This will at least let you determine if it is one of the Threat Prevention blades causing the CPU spike which is the most likely. 

If the CPU is still spiking after trying these commands there is some other blade causing it, and you'll need to figure out what path the Commvault traffic is being processed in with fwaccel conns.  It is probably F2F which is why you are taking such a big CPU hit.

If the CPU use goes way down during this test first thing would be to define a "null" TP profile (i.e. all TP blades unchecked in the profile) as described in my book and create a TP rule at the top invoking this profile against Commvault traffic.  You can check/enable one TP blade at a time in the "null" profile to figure out which blade is slowing it down and go from there.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

[avisheen_shetty]

enabled_blades
fw vpn urlf av appi ips identityServer anti_bot

fwaccel stats -s
Accelerated conns/Total conns : 2/31302 (0%)
Accelerated pkts/Total pkts : 3400/199296295 (0%)
F2Fed pkts/Total pkts : 8842634/199296295 (4%)
PXL pkts/Total pkts : 190450261/199296295 (95%)
QXL pkts/Total pkts : 0/199296295 (0%)

One of the concern is IP addresses for commvault traffic is widespread . Its mostly 23.0.0.0/8 .Havent been able to pinpoint few IPs.

I havent tried unchecking both IPS and AV blades at the same time. That is something i need to do ;Let me see if creating a null profile and providing exception towards 23.0.0.0/8 causes any relief.

[avisheen_shetty]

I did try unchecking IPS and AV blades and the CPU utilization did come down though it did hit 100% for a fraction of second or more  towards the end of the download. I created a null profile and allowed 23.0.0.0/8  subnet, However that didn't solve the problem. There are lot of IP addresses which come up during each download and different subnets.

Thank you for your suggestions. 

[Soren_Kristense]

Hi i have seen the same issues on a VSX VSLS cluster. 

[PhoneBoy]

Based on the SR you sent me, it looks like this is related to R77.30.

Which Jumbo Hotfix version https://community.checkpoint.com/people/avish7b07c9ff-04a8-4677-991d-d31b9b9dc370‌?

Likewise, which version/jumbo hotfix level are you seeing it on https://community.checkpoint.com/people/SRK65a476d4-f1f4-3f03-89f6-a4e7327d0c44‌?

Also, if you have an SR, please send it to me in a private message.

[avisheen_shetty]

Jumbo hotfix is not applied on the new set of hardwares. However when this

problem surfaced in September , the jumbo fix on the device was latest and

Take 216.

When the problem surfaced , i had two pairs of clusters( former setup on

Dell Edge servers and later on 5600 appliances )

Now hardware's have been changed for both ( Dells replaced with 5600

appliances and 5600s are replaced by 5900 appliances ).

I need to install the latest jumbo fix on the newer hardwares.

[PhoneBoy]

So is this issue resolved for you in the latest jumbo?

And are we talking the latest recommended jumbo or the most recent ongoing?

[avisheen_shetty]

No the issue persists. I haven't yet installed the latest jumbo fix though . I need to install Take 292 and check if that solves the issue

[PhoneBoy]

Do you have Dynamic Dispatcher enabled in this situation?

[avisheen_shetty]

Yes Dynamic dispatcher is enabled .Firewall is able to handle downloads of heavier files. The CPU spike is observed only when Commvault files /packages are downloaded using Commvault installer.

[avisheen_shetty]

I installed the jumbo fix, but problem continued. Creating a null TP profile and applying it to the commvault IP address is likely to solve the problem . But each time i download, i get a new subnet 

[Soren_Kristense]

Dynamic dispatcher is disabled in my VSX cluster. The cluster was R77.30 jumbo take 205

[avisheen_shetty]

I just called the TAC and they informed that it could be case of elephant flow. They are still investigating on the issue before they can confirm this, 

[Niels_van_Sluis]

It seems we ran into this issue also. Did you fix this issue?