topic Re: Mangement server R80.10 slowness in Firewall and Security Management

Mangement server R80.10 slowness

Arthur_DENIS1 — Thu, 29 Mar 2018 12:21:42 GMT

Hi,

I experiencing a loooot of slowness issue on my management server.

The management server is on R80.10 and the gateways are running on R77.30.

I keep running to a problem where java process hogs up all the available CPU and I'm unable to do anything at this point. Sometimes, the SmartConsole stops responding and closes. When trying to reconnect, I keep getting an Operation Timeout error. After some time, java process consumption eventually goes down and only after that I'm able to re login.

The management server runs on a VM with 24GB RAM and 4 CPU cores.

Somebody here has already get an issue like this one ?

Thanks for your help

PS: top, sar, and iostat attached.

Re: Mangement server R80.10 slowness

Robert_Decker — Thu, 29 Mar 2018 12:29:46 GMT

try this thread - https://community.checkpoint.com/message/11361-re-management-server-slowness-in-r8010?commentID=11361#comment-11361.

Robert.

Re: Mangement server R80.10 slowness

Arthur_DENIS1 — Thu, 29 Mar 2018 12:49:17 GMT

Hi Robert,

Thanks for you quick answer. However the other thread is not exactly the same as i had also "postgress" process using CPU and not the same take.

But yes, definitively similar, without real solution for now...

Re: Mangement server R80.10 slowness

Robert_Decker — Thu, 29 Mar 2018 12:54:22 GMT

Nevertheless, your feedback is very important, I'll forward it to our management server team.

Robert.

Re: Mangement server R80.10 slowness

Arthur_DENIS1 — Thu, 29 Mar 2018 12:58:51 GMT

thx!
If another logs or informations could be helpful please let me know.

Re: Mangement server R80.10 slowness

Timothy_Hall — Thu, 29 Mar 2018 13:56:18 GMT

Based on your screenshots, the SMS does not appear to be constrained for memory or disk I/O bandwidth.

Keep in mind 4 CPUs is the *minimum* for an R80+ SMS, any chance you can go to 8 discrete CPUs? "Discrete" in this case means 8 separate CPUs, not 4 CPUs w/ 2 cores each. Hyperthreading or having multiple cores per CPU is not recommended on an SMS for performance reasons.

Beyond that we'll need to know which specific java process is chewing CPU, typically there are six of them on an R80+ SMS. Next time you have the problem, identify the PID of the CPU hog with ps or top, then run cpwd_admin list to determine which specific java process (and its associated function) is the issue and we can diagnose further.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Mangement server R80.10 slowness

Arthur_DENIS1 — Thu, 29 Mar 2018 15:15:00 GMT

Thanks for the first confirmation about I/O and memory.

Regarding the CPU, I need to talk with my virtualization team.

regarding the specific java process, as you can see in the new screenshot, SOLR is involved:

Indeed, log indexing is enabled on this SMS. But what is the CPU consumption baseline ?

Re: Mangement server R80.10 slowness

Timothy_Hall — Mon, 30 Sep 2019 14:28:47 GMT

SOLR performs log indexing and does use a lot of CPU (which is expected) but notice that SOLR and the associated log_indexer process show a nice (NI) value of 19 in top, which indicates those processes are running with the lowest CPU priority possible. So if literally any other process needs a CPU, these indexer processes will be immediately thrown off to make way for the other process. Very unlikely that these are causing the SmartConsole hangs you are seeing, unless they are saturating the disk channel to the point of affecting other processes which does not appear to be happening.

Please provide output of ps -efwww | grep 5078, would like to see what that java process is since it isn't showing up in cpwd_admin list.

Looking like the SMS just needs more CPUs as that is the only thing apparently holding you back.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Mangement server R80.10 slowness

Kaspars_Zibarts — Thu, 29 Mar 2018 19:01:37 GMT

Can't agree more. Add CPUs. If you had this HW set up with R77.30 then double it as a minimum for R80.10. it's rather resource hungry. For good reason - you get a lot in return. We quadrupled VM specs after upgrade to have reasonable performance.

Re: Mangement server R80.10 slowness

Kaspars_Zibarts — Fri, 30 Mar 2018 07:35:56 GMT

And hopefully disk access slowness will be resolved in R80.20 with faster and more efficient file system

Re: Mangement server R80.10 slowness

Arthur_DENIS1 — Fri, 30 Mar 2018 07:38:35 GMT

Looks really better after increasing to 8 CPU.

Thanks a lot !

Do you have a sheet or a recommendation based on the utilization ?

Eg: 1 SMS without SmartLog vs with SmartLog, if we have 1 managed firewall vs 15 manged, ...

I mean, real life recommendation, based on your personnal experience.

Thx

Re: Mangement server R80.10 slowness

Arthur_DENIS1 — Fri, 30 Mar 2018 07:39:18 GMT

Nice

Re: Mangement server R80.10 slowness

Timothy_Hall — Mon, 30 Sep 2019 14:29:41 GMT

A lot depends on the logging rate from the managed gateways which can vary considerably between different sites; the Smart-1 datasheets do show the indexed logging rate capacity for the various Smart-1 appliances, and once the number of Smart-1 cores is taken into consideration from this list you can get a general sense of overall capacity. SMS performance is a bit tough to predict or make general recommendations for, other than the following which is my personal opinion:

1) Minimum - 4 discrete (non-hyperthreaded) cores, 8 GB RAM

2) Better - 8 discrete cores, 16GB RAM

I did a bit of testing awhile back, and 8 discrete cores for an R80+ SMS seems to be the sweet spot. As the number of cores was increased from 4 to 8, it was easy to see a fairly linear increase in management performance. Above 8 cores there was still some minor improvement but it seemed to be mostly past the point of diminishing returns.

Allocating more than 16GB of RAM certainly won't hurt, but it will primarily get used for caching disk operations unless you have a very large configuration or are using MDM/Provider-1.

I think the above stems from the fact that there are 6 key java-based processes on an R80+ SMS, which are well-known for trying to soak up all the system resources you can throw at them in an attempt to run faster. When the Linux kernel is updated to 3.10 enabling hyperthreading might help further (and may improve many other areas such as using "huge" memory pages), but the current consensus is to leave hyperthreading off on a 2.6.18 kernel R80/R80.10 SMS due to exacerbating possible bottlenecks in the older I/O channel drivers.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Mangement server R80.10 slowness

Arthur_DENIS1 — Fri, 30 Mar 2018 14:56:05 GMT

Perfect, thanks a lot!

Re: Mangement server R80.10 slowness

Václav_Brožík — Thu, 12 Apr 2018 08:29:13 GMT

Tim Hall wrote:
...there is no way to prioritize disk access, it is just FIFO...

Linux kernel can prioritize IO access (including disk access) for read operations and synchronous write operations. Gaia contains a very old kernel (2.6.18 based on the kernel from RHEL 5.2) but it already supports IO priorities using the CLI tool ionice. If there is a lot of IO operations (check vmstat) it is possible to manually lower the IO priority of a running process to test if it helps. I did not test it yet. There is also a tool "iotop" which can show IO operations of individual processes but I am afraid that the Gaia's kernel lacks support for it.

Newer Linux kernels by default also set the IO priority when you set the CPU priority. So it is possible that in R80.20 some processes will have a lower IO priority. If anyone has EA of the new Gaia you can check it using ionice -p $PID. (Replace $PID by the PID of the process to check.)

Re: Mangement server R80.10 slowness

Timothy_Hall — Thu, 12 Apr 2018 13:24:31 GMT

Thanks for the clarification Václav Brožík‌, for some reason I thought CFQ was available in 2.6.18 but not enabled by default. As you stated it is enabled by default starting in 2.6.18 and follows the CPU nice priority, so processes SOLR and LogCore show via ionice -p PID a lowered I/O priority of 7, while all other processes have the higher priority 0 on a SMS. On an R80.10 SMS, it doesn't look like Check Point has directly tweaked the I/O priorities beyond the two processes mentioned above. The iotop tool will be very helpful on an SMS after the upcoming Gaia kernel update.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Mangement server R80.10 slowness

RickLin — Thu, 12 Apr 2018 13:54:54 GMT

Thanks for provide this kind of information about this new kernel feature.

But do you mean I can toggle i to change priority of Solr or LogCore process?

Re: Mangement server R80.10 slowness

Timothy_Hall — Thu, 12 Apr 2018 16:35:41 GMT

Hi Rick,

You can adjust the I/O priority of any process you want with the ionice command. However the SOLR and LogCore processes are already set to the lowest I/O priority possible by virtue of having their CPU priority ("nice" value) already lowered by Check Point "out of the box". I would most definitely NOT recommend raising the I/O or CPU priority of these two processes on a SMS, Check Point re-niced their priority downwards for a good reason.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Re: Mangement server R80.10 slowness

Václav_Brožík — Thu, 12 Apr 2018 17:49:01 GMT

I have checked ionice in one of my R80.10 (JHFA 70) installations and I have a little bit different results:

1. nice does not change ionice:

[Expert@sms:0]# nice -n19 sleep 100 & ionice -p$!
[1] 21846
none: prio 0

2. SOLR has the basic IO priority:

[Expert@sms:0]# ionice -p"$(pgrep -f CPM_SOLR)"
none: prio 0

3. LogCore has the lowest best-effort IO priority:

[Expert@sms:0]# ionice -p"$(pgrep -f LogCore)"
best-effort: prio 7

Re: Mangement server R80.10 slowness

RickLin — Fri, 13 Apr 2018 00:27:00 GMT

This is in one of my customer installed with R80.10 Jumbo HFA T70 situation.