FWD proccess high CPU usage

[fabionfsc]

Hello, everyone, how are you?

We have a problem with a client (R81.20 Take 89 ClusterXL) whose log sending process (fwd) is always using a high CPU and the logs take about 10 minutes to be displayed in SmartConsole. The SMS CPU is not high, but the Cluster itself has a LOT of traffic and the CPU is always high during peak hours.

I would like someone to confirm something for me: is it true that, in versions up to R81.20, the fwd process is single-core by design? Is it true that in R82 the fwd process is now multi-threaded?

Personally, I would really like to implement the R82 version in this environment, but I am very worried about the problems that may arise, since there is no Take available yet.

[PhoneBoy]

fwd is typically operating on shared cores. 
Dedicating a core to fwd might be helpful: https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_PerformanceTuning_AdminGuide... 

I'm fairly certain fwd is single core and also heard that was supposed to change in R82.
Don't know for certain if it is, though.

[fabionfsc]

I spent the whole afternoon testing this issue in Lab and came to the following conclusions:

• When the FW has few CPU Cores, the fwd process can be used by any CoreXL or CoreSND, not a specific one.

• When the FW has many CPU Cores, GAiA OS itself seems to create a CPU Core only for the fwd process.

The documentation you sent is to be used in the first case, where the FW has few cores and no specific core is set for the fwd. By doing the documentation, you set a core to be responsible only for the fwd process.

In a Lab (ESXi) I created a VM with 28 cores, look at the behavior in R81.20:

I did the documentation you mentioned, with the intention of adding another core for the fwd process, but it doesn't work, only one core is allowed for the fwd process.

Now, look at how the R82 version looks like. No specific core was set for fwd, I believe that in the R82 version, the fwd process is now multi-threaded, this was said in the EA version of R82:

[PhoneBoy]

Here's what the R82 Release Notes say: 

• Quantum Security Gateway log rate output capacity increased by up to 100% through a new multi-process architecture.

That would imply either fwd is mutli-core or a new process that is multi-core is handling logging.

[Timothy_Hall · Accepted solution]

fwd is indeed single-threaded in R81.20 and earlier, and will not benefit from having more than one core dedicated to it.  On R81.20 and earlier gateways that have 20 or more cores, a core that would otherwise be utilized as a worker core is automatically dedicated to fwd.  So on a 24 core gateway the default split would be 4/19 +1 for fwd.

I would assume there is a new multi-threaded daemon handling logging in R82; my initial assumption was that it is the new uprd daemon but I don't think that is correct. The fwd binaries are very close in size between R81.20 and R82, so I highly doubt fwd is part of this new multi-threaded log mechanism.  Frankly fwd is so old it would be way easier to just write a new daemon to handle this rather than mucking about in such old code.

Would appreciate some insight from R&D on this as I've asked several Check Point SEs about this new enhanced R82 logging mechanism and while they are all aware of it via new-feature presentations and such, none of them had seen any details published internally about it.

[Tomer_Noy · Accepted solution]

Indeed in R81.20 and earlier versions, the fwd process handles most of its processing on a single main thread.
BTW, fwd is often busy handling logs, but it also handles a few more functions such as cluster sync and more.

In R82 we introduced a significant architecture improvement that we call internally "fwd scale-out". Instead of having just one fwd process, we have multiple fwd worker processes that take most of the logging processing away from the main one.

The main benefit is increased log sending throughput, especially on gateways that have many cores and handle a lot of traffic. Another benefit is resiliency since heavy logging is less likely to impact other functionality running on fwd.

The improved log sending capacity goes well with another feature on the Management / Log Server side called "log distribution" in which you can configure a gateway to distribute the log to multiple log servers, which also helps with resiliency and provides improved log ingestion capacity.

[Timothy_Hall]

Makes sense, thanks!

[the_rock]

Thats fantastic info @Tomer_Noy 

[the_rock]

Hey bro, hows life? Hope you are well. Not sure if this might be applicable, but maybe worth checking?

Andy

https://supportcontent.checkpoint.com/solutions?id=sk30908

[HeikoAnkenbrand]

To troubleshoot high CPU usage by the FWD process in Check Point, you can follow these steps:

1) Identify the Cause:
Use the top command to identify which processes are consuming the most CPU.
Check if the FWD process is indeed the one causing high CPU usage.

2) Debug the FWD Process:

Enable debugging for the FWD process to gather more information:

fw debug fwd on TDERROR_ALL_ALL=5
fw debug fwd on DEBUG_LEVEL=3

Replicate the issue and then stop the debug:

fw debug fwd off TDERROR_ALL_ALL=0
fw debug fwd off DEBUG_LEVEL=0

3) Review Logs:

Check the $FWDIR/log/fwd.elg file for any errors or unusual activity that might indicate the cause of the high CPU usage.

4) Check Network and Configuration:

Ensure that there are no network misconfigurations or excessive logging that might be contributing to the high CPU usage.