- CheckMates
- :
- Products
- :
- Quantum
- :
- Threat Prevention
- :
- Re: Commvault Installer :CPU Spikes on Checkpoint
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Are you a member of CheckMates?
×- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Commvault Installer :CPU Spikes on Checkpoint
Has anyone faced issue CPU spikes on the checkpoint cluster while downloading Commvault packages.
I am currently facing CPU spikes to almost 100% when i try to download Commvault packages using the installer.
This has been observed on a cluster of 5600 appliances and 5900 appliances as well.
These firewalls have all the blades enabled
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Are there any log messages that show up around this time?
Have you also engaged the TAC with this? Contact Support | Check Point Software
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Nothing specific shows in the log messages . I have engaged the tax since
October . They also faced the same problem . However when they disabled all
the blades except firewall, spike wasn’t seen .
Bypassing the IP address on the IPS blade did not solve the problem in my
environment .
I am still working with the TAC . Hope they provide a fix or a solution .
On Sat, Feb 10, 2018 at 7:42 PM Dameon Welch Abernathy <
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you please send me the SR number in a private message?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
> However when they disabled all the blades except firewall, spike wasn’t seen . Bypassing the IP address on the IPS blade did not solve the problem in my environment .
This is significant. What does output of enabled_blades show when run on the firewall? Defining an IPS Exception won't change the path (SXL/PXL/F2F) the Commvault traffic takes through the firewall, but disabling all blades other than Firewall will. Try this on the gateway:
ips off
(try commvault traffic, and if still causing CPU spike)
fw amw unload
(try commvault traffic again)
ips on
fw amw fetch local
This will at least let you determine if it is one of the Threat Prevention blades causing the CPU spike which is the most likely.
If the CPU is still spiking after trying these commands there is some other blade causing it, and you'll need to figure out what path the Commvault traffic is being processed in with fwaccel conns. It is probably F2F which is why you are taking such a big CPU hit.
If the CPU use goes way down during this test first thing would be to define a "null" TP profile (i.e. all TP blades unchecked in the profile) as described in my book and create a TP rule at the top invoking this profile against Commvault traffic. You can check/enable one TP blade at a time in the "null" profile to figure out which blade is slowing it down and go from there.
--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com
now available at maxpowerfirewalls.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
enabled_blades
fw vpn urlf av appi ips identityServer anti_bot
fwaccel stats -s
Accelerated conns/Total conns : 2/31302 (0%)
Accelerated pkts/Total pkts : 3400/199296295 (0%)
F2Fed pkts/Total pkts : 8842634/199296295 (4%)
PXL pkts/Total pkts : 190450261/199296295 (95%)
QXL pkts/Total pkts : 0/199296295 (0%)
One of the concern is IP addresses for commvault traffic is widespread . Its mostly 23.0.0.0/8 .Havent been able to pinpoint few IPs.
I havent tried unchecking both IPS and AV blades at the same time. That is something i need to do ;Let me see if creating a null profile and providing exception towards 23.0.0.0/8 causes any relief.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I did try unchecking IPS and AV blades and the CPU utilization did come down though it did hit 100% for a fraction of second or more towards the end of the download. I created a null profile and allowed 23.0.0.0/8 subnet, However that didn't solve the problem. There are lot of IP addresses which come up during each download and different subnets.
Thank you for your suggestions.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi i have seen the same issues on a VSX VSLS cluster.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Based on the SR you sent me, it looks like this is related to R77.30.
Which Jumbo Hotfix version https://community.checkpoint.com/people/avish7b07c9ff-04a8-4677-991d-d31b9b9dc370?
Likewise, which version/jumbo hotfix level are you seeing it on https://community.checkpoint.com/people/SRK65a476d4-f1f4-3f03-89f6-a4e7327d0c44?
Also, if you have an SR, please send it to me in a private message.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Jumbo hotfix is not applied on the new set of hardwares. However when this
problem surfaced in September , the jumbo fix on the device was latest and
Take 216.
When the problem surfaced , i had two pairs of clusters( former setup on
Dell Edge servers and later on 5600 appliances )
Now hardware's have been changed for both ( Dells replaced with 5600
appliances and 5600s are replaced by 5900 appliances ).
I need to install the latest jumbo fix on the newer hardwares.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
So is this issue resolved for you in the latest jumbo?
And are we talking the latest recommended jumbo or the most recent ongoing?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No the issue persists. I haven't yet installed the latest jumbo fix though . I need to install Take 292 and check if that solves the issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Do you have Dynamic Dispatcher enabled in this situation?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes Dynamic dispatcher is enabled .Firewall is able to handle downloads of heavier files. The CPU spike is observed only when Commvault files /packages are downloaded using Commvault installer.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I installed the jumbo fix, but problem continued. Creating a null TP profile and applying it to the commvault IP address is likely to solve the problem . But each time i download, i get a new subnet
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I just called the TAC and they informed that it could be case of elephant flow. They are still investigating on the issue before they can confirm this,
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It seems we ran into this issue also. Did you fix this issue?