Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
mrl_sousa
Participant
Jump to solution

SMS Server R80.10 Extremely Slow and Cannot login

Hi ,

 

I'm running R80.10 on my Gateways and SMS Server ( Physical Appliance).  My SMS server was running very slow for same time  and now i cannot even login in Smart Console. Can someone please help troubleshoot/resolve the issue ?

Find in attach the login error, top,sar, iostat and cpview.

 

SMS SPECS          :  -2 x CPU ; -8 GB RAM ; 

GATEWAY SPECS :  -2 x CPU ( 16 x cores each) ; -32 GB RAM ;

 

Note: I'm new to checkpoint.

 

Regards,

Mauro de sousa

 

 

 

3 Solutions

Accepted Solutions
Timothy_Hall
Legend Legend
Legend

@Daniel_Taney's post might hold the solution.

However in looking at your screenshots, the SMS "Waiting for I/O" (wio) percentages are absurdly high which may be causing a timeout when you are attempting to log in.  A few notes:

1) Curiously you don't seem to be swapping much with your 8GB of RAM which is the typical cause of high wio.  This could indicate a high level of disk overhead due to a very high level of logs coming into your SMS.  Please post your logging rate from sk120341: How to monitor the Log Receive Rate on Management Server / Log Server R80 and above

2) Also the presence of lea_session processes indicates that you are exporting logs to some kind of SIEM which will further increase hard disk utilization, try disabling this functionality and see if it helps.

3) Finally your hard drive may be experiencing errors or about to fail which is causing long waits for hard drive access as the drive retrys various operations.  Check /var/log/messages* on the SMS carefully, do you see any disk warnings or timeout messages?  If so BACKUP THE SMS IMMEDIATELY and make plans to replace it and/or the hard drive.

Beyond that upgrading RAM beyond 8GB might help here, depending on the number of rules/objects in your configuration.  High wio can also be caused by hardware other than the hard drive that is in the process of failing, but that is fairly unlikely.

 

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com

View solution in original post

0 Kudos
Timothy_Hall
Legend Legend
Legend

You are using a Smart-1 210 which has two cores & 8GB of RAM, yet have manually enabled SmartEvent which is not a supported configuration unless you have 16GB RAM, and that is causing high disk utilization.  See page 15 of the R80.10 release notes; this limitation continues into R80.40 so a software upgrade will not help.  You need to disable SmartEvent on your SMS object in the SmartConsole.  Your peak logging rate is also well in excess of the 210's capacity as specified here: sk112797: Smart-1 R80.x Logging Capacity Performance Improvements.

Even if you upgrade your Smart-1 210 with 16GB of RAM, you will just barely be meeting the minimum requirements for a Mgmt/Log/SmartEvent server, and I doubt you will be satisfied with the performance as your next bottleneck will be CPU.  The Smart-1 210 will reach End of Engineering Support later this year, and all support for that model will terminate in 2022. 

I'd strongly suggest replacing your 210 with a Smart-1 410 which has four cores and 32GB of RAM, concurrent with an management software upgrade to at least R80.30.

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com

View solution in original post

Daniel_Schlifka
Contributor

Timothy is right, you're lacking of hardware resources. You need a bigger box.

View solution in original post

0 Kudos
10 Replies
PhoneBoy
Admin
Admin
Are you exporting logs to a SIEM?
If so, which one(s) and are you using Log Exporter or LEA?
What JHF level is your SMS?
Daniel_Taney
Advisor

Can you post the output of cpwd_admin list

R80 CCSA / CCSE
Daniel_Taney
Advisor

Per @PhoneBoy's question about installed HFA's, have you seen sk122073 

This seems to be the exact error you are seeing.

R80 CCSA / CCSE
Timothy_Hall
Legend Legend
Legend

@Daniel_Taney's post might hold the solution.

However in looking at your screenshots, the SMS "Waiting for I/O" (wio) percentages are absurdly high which may be causing a timeout when you are attempting to log in.  A few notes:

1) Curiously you don't seem to be swapping much with your 8GB of RAM which is the typical cause of high wio.  This could indicate a high level of disk overhead due to a very high level of logs coming into your SMS.  Please post your logging rate from sk120341: How to monitor the Log Receive Rate on Management Server / Log Server R80 and above

2) Also the presence of lea_session processes indicates that you are exporting logs to some kind of SIEM which will further increase hard disk utilization, try disabling this functionality and see if it helps.

3) Finally your hard drive may be experiencing errors or about to fail which is causing long waits for hard drive access as the drive retrys various operations.  Check /var/log/messages* on the SMS carefully, do you see any disk warnings or timeout messages?  If so BACKUP THE SMS IMMEDIATELY and make plans to replace it and/or the hard drive.

Beyond that upgrading RAM beyond 8GB might help here, depending on the number of rules/objects in your configuration.  High wio can also be caused by hardware other than the hard drive that is in the process of failing, but that is fairly unlikely.

 

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
0 Kudos
mrl_sousa
Participant

I.   PhoneBoy 1.      Are you exporting logs to a SIEM? If so, which one(s) and are you using Log Exporter or LEA?

  • Ans: No we are not exporting logs to a SIEM. From what i know we only export logs from the Gateways and Sandblast to SMS.

 

2.      What JHF level is your SMS?

  • Ans: JHF Take 249.

II.  Daniel_Taney 1.      Can you post the output of cpwd_admin list

mrl_sousa_0-1581342333910.png

 

2.      Regarding sk122073 

  • JHF :Take 103 installed
  • Build: 161
  • I’m using R80.10
  • Regarding the solution in the SK, I’ve opened a case with checkpoint and the local partner before and they did not resolve the issue and I don’t have a clear explanation of why is this happening, so I got tired of trying.

 

III.    Timothy Hall

1- Curiously you don't seem to be swapping much with your 8GB of RAM which is the typical cause of high wio.  This could indicate a high level of disk overhead due to a very high level of logs coming into your SMS.  Please post your logging rate from sk120341: How to monitor the Log Receive Rate on Management Server / Log Server R80 and above

Attach:

  • cpstat mg -f log_server
  • Log Receive Rate-stattest
  • Log Receive Rate Peak-stattest
  • Log Receive Rate Average (last 10 min)- stattest
  • Connected Gateways Table
  • Log Receive Rate Average (last Hour)- stattest
  • doctor-log.sh –f

 2- Also the presence of lea_session processes indicates that you are exporting logs to some kind of SIEM which will further increase hard disk utilization, try disabling this functionality and see if it helps.

Ans: can you explain me( wich commands to use) how to disable lea_session?  

 

3- Finally your hard drive may be experiencing errors or about to fail which is causing long waits for hard drive access as the drive retrys various operations.  Check /var/log/messages* on the SMS carefully, do you see any disk warnings or timeout messages?  If so BACKUP THE SMS IMMEDIATELY and make plans to replace it and/or the hard drive.

Beyond that upgrading RAM beyond 8GB might help here, depending on the number of rules/objects in your configuration.  High wio can also be caused by hardware other than the hard drive that is in the process of failing, but that is fairly unlikely.

 

Ans: I did no find any error and warning related to HDD, but I will look again.

 

 

Regards,

Mauro de Sousa

Timothy_Hall
Legend Legend
Legend

You are using a Smart-1 210 which has two cores & 8GB of RAM, yet have manually enabled SmartEvent which is not a supported configuration unless you have 16GB RAM, and that is causing high disk utilization.  See page 15 of the R80.10 release notes; this limitation continues into R80.40 so a software upgrade will not help.  You need to disable SmartEvent on your SMS object in the SmartConsole.  Your peak logging rate is also well in excess of the 210's capacity as specified here: sk112797: Smart-1 R80.x Logging Capacity Performance Improvements.

Even if you upgrade your Smart-1 210 with 16GB of RAM, you will just barely be meeting the minimum requirements for a Mgmt/Log/SmartEvent server, and I doubt you will be satisfied with the performance as your next bottleneck will be CPU.  The Smart-1 210 will reach End of Engineering Support later this year, and all support for that model will terminate in 2022. 

I'd strongly suggest replacing your 210 with a Smart-1 410 which has four cores and 32GB of RAM, concurrent with an management software upgrade to at least R80.30.

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
Daniel_Schlifka
Contributor

Timothy is right, you're lacking of hardware resources. You need a bigger box.

0 Kudos
mrl_sousa
Participant

Hi All,

 

Thank you very much for your support, now i will try to disable smartevent ( have to find the procedure).

 

Regards,

Mauro de Sousa

 

0 Kudos
Timothy_Hall
Legend Legend
Legend

To disable SmartEvent, in the SmartConsole uncheck any boxes under "SmartEvent" on the General Properties...Management screen of your SMS object, then perform an Install Database operation.

 

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
0 Kudos
mrl_sousa
Participant

Hi All,

 

I disable SmartEvent in the begining was slow , but after a restart of SMS, it  started to work nicely but all the Views in "Logs & Monitor" desapeared ( the only one left was "open log View"). So i Re-enable "SmartEvent" and every thing start working nicely ( a lot faster than before) . Now the CPU consumption varies between 7%-100% but always "Up-Down"  and does not stay for more than 3s in 100% . Also one thing that i notice is that the RAM consumption is gone Down ( now Used: 4.6 Gbps to 5 Gbps). I also installed "Check_Point_SmartConsole_R80_10_jumbo_HF_B161_Win" and it is working nicely.

I will monitor the behavior and if there is any  change i will let you know. 

 

Thank you ALL for your support.

 

Regards,

 

Mauro de Sousa

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events