SMS Server R80.10 Extremely Slow and Cannot login

[mrl_sousa]

Hi ,

I'm running R80.10 on my Gateways and SMS Server ( Physical Appliance).  My SMS server was running very slow for same time  and now i cannot even login in Smart Console. Can someone please help troubleshoot/resolve the issue ?

Find in attach the login error, top,sar, iostat and cpview.

SMS SPECS          :  -2 x CPU ; -8 GB RAM ; 

GATEWAY SPECS :  -2 x CPU ( 16 x cores each) ; -32 GB RAM ;

Note: I'm new to checkpoint.

Regards,

Mauro de sousa

[PhoneBoy]

Are you exporting logs to a SIEM?
If so, which one(s) and are you using Log Exporter or LEA?
What JHF level is your SMS?

[Daniel_Taney]

Can you post the output of cpwd_admin list

[Daniel_Taney]

Per @PhoneBoy's question about installed HFA's, have you seen sk122073 

This seems to be the exact error you are seeing.

[Timothy_Hall · Accepted solution]

@Daniel_Taney's post might hold the solution.

However in looking at your screenshots, the SMS "Waiting for I/O" (wio) percentages are absurdly high which may be causing a timeout when you are attempting to log in.  A few notes:

1) Curiously you don't seem to be swapping much with your 8GB of RAM which is the typical cause of high wio.  This could indicate a high level of disk overhead due to a very high level of logs coming into your SMS.  Please post your logging rate from sk120341: How to monitor the Log Receive Rate on Management Server / Log Server R80 and above. 

2) Also the presence of lea_session processes indicates that you are exporting logs to some kind of SIEM which will further increase hard disk utilization, try disabling this functionality and see if it helps.

3) Finally your hard drive may be experiencing errors or about to fail which is causing long waits for hard drive access as the drive retrys various operations.  Check /var/log/messages* on the SMS carefully, do you see any disk warnings or timeout messages?  If so BACKUP THE SMS IMMEDIATELY and make plans to replace it and/or the hard drive.

Beyond that upgrading RAM beyond 8GB might help here, depending on the number of rules/objects in your configuration.  High wio can also be caused by hardware other than the hard drive that is in the process of failing, but that is fairly unlikely.

[mrl_sousa]

I.   PhoneBoy 1.      Are you exporting logs to a SIEM? If so, which one(s) and are you using Log Exporter or LEA?

• Ans: No we are not exporting logs to a SIEM. From what i know we only export logs from the Gateways and Sandblast to SMS.

2.      What JHF level is your SMS?

• Ans: JHF Take 249.

II.  Daniel_Taney 1.      Can you post the output of cpwd_admin list

2.      Regarding sk122073 

• JHF :Take 103 installed
• Build: 161
• I’m using R80.10
• Regarding the solution in the SK, I’ve opened a case with checkpoint and the local partner before and they did not resolve the issue and I don’t have a clear explanation of why is this happening, so I got tired of trying.

III.    Timothy Hall

1- Curiously you don't seem to be swapping much with your 8GB of RAM which is the typical cause of high wio.  This could indicate a high level of disk overhead due to a very high level of logs coming into your SMS.  Please post your logging rate from sk120341: How to monitor the Log Receive Rate on Management Server / Log Server R80 and above. 

Attach:

• cpstat mg -f log_server
• Log Receive Rate-stattest
• Log Receive Rate Peak-stattest
• Log Receive Rate Average (last 10 min)- stattest
• Connected Gateways Table
• Log Receive Rate Average (last Hour)- stattest
• doctor-log.sh –f

 2- Also the presence of lea_session processes indicates that you are exporting logs to some kind of SIEM which will further increase hard disk utilization, try disabling this functionality and see if it helps.

Ans: can you explain me( wich commands to use) how to disable lea_session?  

3- Finally your hard drive may be experiencing errors or about to fail which is causing long waits for hard drive access as the drive retrys various operations.  Check /var/log/messages* on the SMS carefully, do you see any disk warnings or timeout messages?  If so BACKUP THE SMS IMMEDIATELY and make plans to replace it and/or the hard drive.

Beyond that upgrading RAM beyond 8GB might help here, depending on the number of rules/objects in your configuration.  High wio can also be caused by hardware other than the hard drive that is in the process of failing, but that is fairly unlikely.

Ans: I did no find any error and warning related to HDD, but I will look again.

Regards,

Mauro de Sousa

[Timothy_Hall · Accepted solution]

You are using a Smart-1 210 which has two cores & 8GB of RAM, yet have manually enabled SmartEvent which is not a supported configuration unless you have 16GB RAM, and that is causing high disk utilization.  See page 15 of the R80.10 release notes; this limitation continues into R80.40 so a software upgrade will not help.  You need to disable SmartEvent on your SMS object in the SmartConsole.  Your peak logging rate is also well in excess of the 210's capacity as specified here: sk112797: Smart-1 R80.x Logging Capacity Performance Improvements.

Even if you upgrade your Smart-1 210 with 16GB of RAM, you will just barely be meeting the minimum requirements for a Mgmt/Log/SmartEvent server, and I doubt you will be satisfied with the performance as your next bottleneck will be CPU.  The Smart-1 210 will reach End of Engineering Support later this year, and all support for that model will terminate in 2022. 

I'd strongly suggest replacing your 210 with a Smart-1 410 which has four cores and 32GB of RAM, concurrent with an management software upgrade to at least R80.30.

[Daniel_Schlifka · Accepted solution]

Timothy is right, you're lacking of hardware resources. You need a bigger box.

[mrl_sousa]

Hi All,

Thank you very much for your support, now i will try to disable smartevent ( have to find the procedure).

Regards,

Mauro de Sousa

[Timothy_Hall]

To disable SmartEvent, in the SmartConsole uncheck any boxes under "SmartEvent" on the General Properties...Management screen of your SMS object, then perform an Install Database operation.

[mrl_sousa]

Hi All,

I disable SmartEvent in the begining was slow , but after a restart of SMS, it  started to work nicely but all the Views in "Logs & Monitor" desapeared ( the only one left was "open log View"). So i Re-enable "SmartEvent" and every thing start working nicely ( a lot faster than before) . Now the CPU consumption varies between 7%-100% but always "Up-Down"  and does not stay for more than 3s in 100% . Also one thing that i notice is that the RAM consumption is gone Down ( now Used: 4.6 Gbps to 5 Gbps). I also installed "Check_Point_SmartConsole_R80_10_jumbo_HF_B161_Win" and it is working nicely.

I will monitor the behavior and if there is any  change i will let you know. 

Thank you ALL for your support.

Regards,

Mauro de Sousa