High CPU Utlization Troubleshooting Step (SecureXL Fast Accelerator)

[Chinmaya_Naik]

Hello Checkmate Team,

We suddenly see CPU utilization during the morning 5 AM to 8 AM time only.

The multiple fw_worker take high CPU but not impact the SND core utlization

Total Core : 8 core (2 SND and 6 fw_worker)

So we checked the TOP connection by referring CPVIEW utility output

As we get to know that a Auto Backup solution like Backup Server take more CPU because that Auto Backup is start working during the time of issue because during the time it’s take backup file from multiple of devices which connected or like integrated with Backup Server.

This issue we observed when this newly implemented BACKUP Solution is start working

Now find below about Solution :

First we need to know that the TOP connection is going to which path like Accelerated Path , medium Path or Slow Path

For this we run cpkstat Utility during the time of high CPU to know which path that go through.

Few Backup connections are triggered and all are in same Medium Path only (F2P)

Refer : sk103212 (Traffic analysis using the 'CPMonitor' tool) to known the high CPU utlization connection PATH.

The next Solution we planned to implemented that is to put that F2P connection to(SecureXL Fast Accelerator (fw fast_accel)) to reduce the CPU utilisation beacuse in the Fast Accelator no inspection is performed like trusted connections to allow bypassing deep packet inspection

Refer SK : sk156672

Now one query is come that why we put that connection without any Inspection ? : 

Answer : We assured that the connection is legitimate and as its for the backup process only so no need Inspection on this.


If we add the top connections to the SecureXL Fast Accelerator is there going to be any impact on the 2 SND cores, because at the time of High CPU utilization observed the SND cores CPU utilization is around 40-50% ?

Answer : There is going to be no impact on the SNDs due to fastaccel, as there is no inspection for this affected traffic

if some particular connection is already accelerated then can we add those connections in SecureXL Fast Accelerator then is there any impact ?

Answer : No Impact

Hi Team Let me know if some point answer I updated on above is correct or anything wrong ?

My Plan of ACTION :

Base on sk156672 which I refer :

1. fw ctl fast_accel enable (Set feature state to on)

2. fw ctl fast_accel show_table (Display the rules configured by the user)

3. fw ctl fast_accel show_state (Display the current feature state)

4. fw ctl fast_accel add 1.1.1.1 2.2.2.0/24 80 6 (Example IP address and Port number with TCP or UDP protocol)

5. fw ctl fast_accel delete 192.168.0.0/16 any 8080 17 (Example IP address and Port number with TCP or UDP protocol

6. Verify using cpview utility :

So base on our issue I need to add the Backup Server IP address which basically the Destination IP address and also revert the traffic.

Like for example :

Backup Server IP address : x.x.x.x/24
Backup Server Listing PORT : TCP 1667

Command :

fw ctl fast_accel add x.x.x.x/24 1667 6

Kindly suggested that above command is correct or not OR also can I need to add the source IP address also OR source and destination IP address are must be included base on the SK ?

Also I plan to added in only Active gateway for testing so incase if any urganet I will fail-over the gateway so kindly update that is this possible that we can use for Active gateway if we using Cluster then

Also Suggest Any Alternative to resolved the High CPU utlization issue .

Regards

@Chinmaya_Naik 

[AaronCP]

Hi @Chinmaya_Naik,

I think your plan to add the rule to the active gateway for testing, then failing over to the standby if necessary is a sensible approach.

I've just double-checked the command you've listed against the SK and I believe a source and destination IP is required (subnet is optional) in the rule as per section 5 (see attached screenshot), so you will want to tweak your add/delete commands to reflect the correct source IPs/networks.

We saw a similar utilisation issue with our backups, and the fast_accel mechanism worked a treat. As per the SK, connections in the slow path will not be accelerated by this feature, so you will want to confirm the backup connections are being accelerated before implementing this.

[Chinmaya_Naik]

Hi @AaronCP 

Thank you for the update.

Yes I agree that in the sk below details also mention and subnet is optional but if someone implemente without source beacuse in our case source is multiple.

Okay but will add both source and destination and check the result of utlization.

Regards

@Chinmaya_Naik 

[Sorin_Gogean]

hello @Chinmaya_Naik ,

we're using fast_accell rules, and I can tell you that not all the traffic is accelerable 😑

search the community, as there was a post explaining how you can look the connections and see what's accelerable and what's not .

in your command you require source and destination specified, at least we're doing it that way.

why you can't specify the source in your case? don't you have a private range that would cover all your sources ? like 10.0.0.0/8 or similar

the addition of fast_accell rules, will not break anything - at least for us it didn't (we did it 3 - 4 months ago) - and you can easily just delete them, or you can just reboot the node (you have to save them in a file in order to kick in next time.....)

hopefully I clarified some points for you 😁

ty,

PS: one other way to lower the CPU usage - if you concluded that the ack-up is  the one triggering it - you can exclude that traffic from certain inspections - like IPS and others....

[AaronCP]

Hi @Chinmaya_Naik,

I think the SK means the subnet is optional, not the IP. So you can specify a single IP or an entire subnet. It states that all of the parameters must be used to add a rule.

As mentioned by @Sorin_Gogean, you can specify 10.0.0.0/8 as your source, so all of your internal networks to the backup network are covered.

Will be interested to hear if adding the rule helps with your utilisation issue.

[Chinmaya_Naik]

Hi @AaronCP and @Sorin_Gogean Thankyou very much for the suggestion 

As per TAC suggestion and Answer of our query : 

If we add the top connections to the SecureXL Fast Accelerator is there going to be any impact on the 2 SND cores, because at the time of High CPU utilization observed the SND cores CPU utilization is around 40-50% ?

Answer : There is going to be no impact on the SNDs due to fastaccel, as there is no inspection for this affected traffic

So as per my Plane of action also the same BUT Now we face the high CPU utilization in SND core because as we added the high CPU utilization connection into the SecureXL Fast Accelerator.

Now instead of CoreXL_FW now CoreXL_SND take high CPU so to overcome the issue we prepare a next POA :

• Manual configuration of CoreXL Firewall and SND instances so for this 4 for SND and for  firewall workers.

OR : 

• dynamic_split :  On Check Point Appliances, R80.40 added the ability to change the number of CoreXLClosed Firewall and SND instances without reboot (Dynamic Split). Like performance-enhancing daemon that balances the load between CoreXL SNDs and CoreXL Firewalls It dynamically changes the split between CoreXL SNDs and CoreXL Firewalls
• As this dynamic_split disabled by default 
  
• As Check Point does not recommend manual configuration of CoreXL Firewall and SND instances, because such configuration disables the CoreXL Dynamic Split so to enable the CoreXL Dynamic Split again, you must disable it and enable it.
• Supported Models : 5000 series: 5600, 5800 and 5900
  Our GiftCity Appliance : CPAP-SG5800-NGTP which supports Dynamic split .

For SecureXL Fast Accelerator Query :

1. If we added some connections in the  SecureXL Fast Accelerator then is there any chance that only a few connections have already been added but any chance of not going through the SecureXL Fast Accelerator path ?
2. Is it possible to add the overall Subnet into the  SecureXL Fast Accelerator instead of adding any particular source and destination ? (Still not try but any one already try )
3. If reboot the firewall then still the Firewall FAST Accel table connection list exists or removed automatically ?

Kindly consider both and provide the recommande solution Checkmates Team,

Regards

@Chinmaya_Naik 

[AaronCP]

Hi @Chinmaya_Naik,

I would recommend enabling the Dynamic Balancing for CoreXL as per SK164155 if your gateway is compatible. I've personally enabled this feature in our production environment and experienced no negative impact. When monitoring the CPU in CPView, I can see the feature dynamically allocating more SND cores during high load, then reverting to the original split once the load had settled. Looking at your screenshot, it looks like your remaining cores are quite heavily utilised, too. Was your screenshot from a particularly busy period? 

To address your other points:

1. I'm not sure I'm fully understanding the question. Can you elaborate, please?

2. You can add entire networks to rules, but you must still include a source and destination. I've not tried this, but you could try adding a rule that accelerates all internal traffic by adding 10.0.0.0/8 in the source and destination.

3. Rebooting the gateway does retain the entries in the fast_accel table. It's only the hit count that's lost.

Thanks,

Aaron.