Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
avisheen_shetty
Participant

Commvault Installer :CPU Spikes on Checkpoint

Has anyone faced issue CPU spikes on the checkpoint cluster while downloading Commvault packages.

I am currently facing CPU spikes to almost 100% when i try to download Commvault packages using the installer.

This has been observed on a cluster of 5600 appliances and 5900 appliances as well.

These firewalls have all the blades enabled 

17 Replies
PhoneBoy
Admin
Admin

Are there any log messages that show up around this time?

Have you also engaged the TAC with this? Contact Support | Check Point Software 

0 Kudos
avisheen_shetty
Participant

Nothing specific shows in the log messages . I have engaged the tax since

October . They also faced the same problem . However when they disabled all

the blades except firewall, spike wasn’t seen .

Bypassing the IP address on the IPS blade did not solve the problem in my

environment .

I am still working with the TAC . Hope they provide a fix or a solution .

On Sat, Feb 10, 2018 at 7:42 PM Dameon Welch Abernathy <

0 Kudos
PhoneBoy
Admin
Admin

Can you please send me the SR number in a private message?

0 Kudos
Timothy_Hall
Champion
Champion

> However when they disabled all the blades except firewall, spike wasn’t seen .  Bypassing the IP address on the IPS blade did not solve the problem in my environment .

This is significant.  What does output of enabled_blades show when run on the firewall?  Defining an IPS Exception won't change the path (SXL/PXL/F2F) the Commvault traffic takes through the firewall, but disabling all blades other than Firewall will.  Try this on the gateway:

ips off

(try commvault traffic, and if still causing CPU spike)

fw amw unload

(try commvault traffic again)

ips on

fw amw fetch local

This will at least let you determine if it is one of the Threat Prevention blades causing the CPU spike which is the most likely. 

If the CPU is still spiking after trying these commands there is some other blade causing it, and you'll need to figure out what path the Commvault traffic is being processed in with fwaccel conns.  It is probably F2F which is why you are taking such a big CPU hit.

If the CPU use goes way down during this test first thing would be to define a "null" TP profile (i.e. all TP blades unchecked in the profile) as described in my book and create a TP rule at the top invoking this profile against Commvault traffic.  You can check/enable one TP blade at a time in the "null" profile to figure out which blade is slowing it down and go from there.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
0 Kudos
avisheen_shetty
Participant

enabled_blades
fw vpn urlf av appi ips identityServer anti_bot

fwaccel stats -s
Accelerated conns/Total conns : 2/31302 (0%)
Accelerated pkts/Total pkts : 3400/199296295 (0%)
F2Fed pkts/Total pkts : 8842634/199296295 (4%)
PXL pkts/Total pkts : 190450261/199296295 (95%)
QXL pkts/Total pkts : 0/199296295 (0%)

One of the concern is IP addresses for commvault traffic is widespread . Its mostly 23.0.0.0/8 .Havent been able to pinpoint few IPs.

I havent tried unchecking both IPS and AV blades at the same time. That is something i need to do ;Let me see if creating a null profile and providing exception towards 23.0.0.0/8 causes any relief.

0 Kudos
avisheen_shetty
Participant

I did try unchecking IPS and AV blades and the CPU utilization did come down though it did hit 100% for a fraction of second or more  towards the end of the download. I created a null profile and allowed 23.0.0.0/8  subnet, However that didn't solve the problem. There are lot of IP addresses which come up during each download and different subnets.

Thank you for your suggestions. 

0 Kudos
Soren_Kristense
Contributor

Hi i have seen the same issues on a VSX VSLS cluster. 

0 Kudos
PhoneBoy
Admin
Admin

Based on the SR you sent me, it looks like this is related to R77.30.

Which Jumbo Hotfix version https://community.checkpoint.com/people/avish7b07c9ff-04a8-4677-991d-d31b9b9dc370‌?

Likewise, which version/jumbo hotfix level are you seeing it on https://community.checkpoint.com/people/SRK65a476d4-f1f4-3f03-89f6-a4e7327d0c44‌?

Also, if you have an SR, please send it to me in a private message.

0 Kudos
avisheen_shetty
Participant

Jumbo hotfix is not applied on the new set of hardwares. However when this

problem surfaced in September , the jumbo fix on the device was latest and

Take 216.

When the problem surfaced , i had two pairs of clusters( former setup on

Dell Edge servers and later on 5600 appliances )

Now hardware's have been changed for both ( Dells replaced with 5600

appliances and 5600s are replaced by 5900 appliances ).

I need to install the latest jumbo fix on the newer hardwares.

0 Kudos
PhoneBoy
Admin
Admin

So is this issue resolved for you in the latest jumbo?

And are we talking the latest recommended jumbo or the most recent ongoing?

0 Kudos
avisheen_shetty
Participant

No the issue persists. I haven't yet installed the latest jumbo fix though . I need to install Take 292 and check if that solves the issue

0 Kudos
PhoneBoy
Admin
Admin

Do you have Dynamic Dispatcher enabled in this situation?

0 Kudos
avisheen_shetty
Participant

Yes Dynamic dispatcher is enabled .Firewall is able to handle downloads of heavier files. The CPU spike is observed only when Commvault files /packages are downloaded using Commvault installer.

0 Kudos
avisheen_shetty
Participant

I installed the jumbo fix, but problem continued. Creating a null TP profile and applying it to the commvault IP address is likely to solve the problem . But each time i download, i get a new subnet 

0 Kudos
Soren_Kristense
Contributor

Dynamic dispatcher is disabled in my VSX cluster. The cluster was R77.30 jumbo take 205


0 Kudos
avisheen_shetty
Participant

I just called the TAC and they informed that it could be case of elephant flow. They are still investigating on the issue before they can confirm this, 

0 Kudos
Niels_van_Sluis
Contributor

It seems we ran into this issue also. Did you fix this issue?

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events