cancel
Showing results for 
Search instead for 
Did you mean: 
Create a Post
Highlighted

SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

Hi ,

 

I'm running R80.10 on my Gateways and SMS Server ( Physical Appliance).  My SMS server was running very slow for same time  and now i cannot even login in Smart Console. Can someone please help troubleshoot/resolve the issue ?

Find in attach the login error, top,sar, iostat and cpview.

 

SMS SPECS          :  -2 x CPU ; -8 GB RAM ; 

GATEWAY SPECS :  -2 x CPU ( 16 x cores each) ; -32 GB RAM ;

 

Note: I'm new to checkpoint.

 

Regards,

Mauro de sousa

 

 

 

3 Solutions

Accepted Solutions
Highlighted

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

@Daniel_Taney's post might hold the solution.

However in looking at your screenshots, the SMS "Waiting for I/O" (wio) percentages are absurdly high which may be causing a timeout when you are attempting to log in.  A few notes:

1) Curiously you don't seem to be swapping much with your 8GB of RAM which is the typical cause of high wio.  This could indicate a high level of disk overhead due to a very high level of logs coming into your SMS.  Please post your logging rate from sk120341: How to monitor the Log Receive Rate on Management Server / Log Server R80 and above

2) Also the presence of lea_session processes indicates that you are exporting logs to some kind of SIEM which will further increase hard disk utilization, try disabling this functionality and see if it helps.

3) Finally your hard drive may be experiencing errors or about to fail which is causing long waits for hard drive access as the drive retrys various operations.  Check /var/log/messages* on the SMS carefully, do you see any disk warnings or timeout messages?  If so BACKUP THE SMS IMMEDIATELY and make plans to replace it and/or the hard drive.

Beyond that upgrading RAM beyond 8GB might help here, depending on the number of rules/objects in your configuration.  High wio can also be caused by hardware other than the hard drive that is in the process of failing, but that is fairly unlikely.

 

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com

View solution in original post

0 Kudos
Highlighted

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

You are using a Smart-1 210 which has two cores & 8GB of RAM, yet have manually enabled SmartEvent which is not a supported configuration unless you have 16GB RAM, and that is causing high disk utilization.  See page 15 of the R80.10 release notes; this limitation continues into R80.40 so a software upgrade will not help.  You need to disable SmartEvent on your SMS object in the SmartConsole.  Your peak logging rate is also well in excess of the 210's capacity as specified here: sk112797: Smart-1 R80.x Logging Capacity Performance Improvements.

Even if you upgrade your Smart-1 210 with 16GB of RAM, you will just barely be meeting the minimum requirements for a Mgmt/Log/SmartEvent server, and I doubt you will be satisfied with the performance as your next bottleneck will be CPU.  The Smart-1 210 will reach End of Engineering Support later this year, and all support for that model will terminate in 2022. 

I'd strongly suggest replacing your 210 with a Smart-1 410 which has four cores and 32GB of RAM, concurrent with an management software upgrade to at least R80.30.

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com

View solution in original post

Highlighted

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

Timothy is right, you're lacking of hardware resources. You need a bigger box.

View solution in original post

0 Kudos
10 Replies
Highlighted
Admin
Admin

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution
Are you exporting logs to a SIEM?
If so, which one(s) and are you using Log Exporter or LEA?
What JHF level is your SMS?
Highlighted

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

Can you post the output of cpwd_admin list

R80 CCSA / CCSE
Highlighted

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

Per @PhoneBoy's question about installed HFA's, have you seen sk122073 

This seems to be the exact error you are seeing.

R80 CCSA / CCSE
Highlighted

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

@Daniel_Taney's post might hold the solution.

However in looking at your screenshots, the SMS "Waiting for I/O" (wio) percentages are absurdly high which may be causing a timeout when you are attempting to log in.  A few notes:

1) Curiously you don't seem to be swapping much with your 8GB of RAM which is the typical cause of high wio.  This could indicate a high level of disk overhead due to a very high level of logs coming into your SMS.  Please post your logging rate from sk120341: How to monitor the Log Receive Rate on Management Server / Log Server R80 and above

2) Also the presence of lea_session processes indicates that you are exporting logs to some kind of SIEM which will further increase hard disk utilization, try disabling this functionality and see if it helps.

3) Finally your hard drive may be experiencing errors or about to fail which is causing long waits for hard drive access as the drive retrys various operations.  Check /var/log/messages* on the SMS carefully, do you see any disk warnings or timeout messages?  If so BACKUP THE SMS IMMEDIATELY and make plans to replace it and/or the hard drive.

Beyond that upgrading RAM beyond 8GB might help here, depending on the number of rules/objects in your configuration.  High wio can also be caused by hardware other than the hard drive that is in the process of failing, but that is fairly unlikely.

 

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com

View solution in original post

0 Kudos

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

I.   PhoneBoy 1.      Are you exporting logs to a SIEM? If so, which one(s) and are you using Log Exporter or LEA?

  • Ans: No we are not exporting logs to a SIEM. From what i know we only export logs from the Gateways and Sandblast to SMS.

 

2.      What JHF level is your SMS?

  • Ans: JHF Take 249.

II.  Daniel_Taney 1.      Can you post the output of cpwd_admin list

mrl_sousa_0-1581342333910.png

 

2.      Regarding sk122073 

  • JHF :Take 103 installed
  • Build: 161
  • I’m using R80.10
  • Regarding the solution in the SK, I’ve opened a case with checkpoint and the local partner before and they did not resolve the issue and I don’t have a clear explanation of why is this happening, so I got tired of trying.

 

III.    Timothy Hall

1- Curiously you don't seem to be swapping much with your 8GB of RAM which is the typical cause of high wio.  This could indicate a high level of disk overhead due to a very high level of logs coming into your SMS.  Please post your logging rate from sk120341: How to monitor the Log Receive Rate on Management Server / Log Server R80 and above

Attach:

  • cpstat mg -f log_server
  • Log Receive Rate-stattest
  • Log Receive Rate Peak-stattest
  • Log Receive Rate Average (last 10 min)- stattest
  • Connected Gateways Table
  • Log Receive Rate Average (last Hour)- stattest
  • doctor-log.sh –f

 2- Also the presence of lea_session processes indicates that you are exporting logs to some kind of SIEM which will further increase hard disk utilization, try disabling this functionality and see if it helps.

Ans: can you explain me( wich commands to use) how to disable lea_session?  

 

3- Finally your hard drive may be experiencing errors or about to fail which is causing long waits for hard drive access as the drive retrys various operations.  Check /var/log/messages* on the SMS carefully, do you see any disk warnings or timeout messages?  If so BACKUP THE SMS IMMEDIATELY and make plans to replace it and/or the hard drive.

Beyond that upgrading RAM beyond 8GB might help here, depending on the number of rules/objects in your configuration.  High wio can also be caused by hardware other than the hard drive that is in the process of failing, but that is fairly unlikely.

 

Ans: I did no find any error and warning related to HDD, but I will look again.

 

 

Regards,

Mauro de Sousa

Highlighted

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

You are using a Smart-1 210 which has two cores & 8GB of RAM, yet have manually enabled SmartEvent which is not a supported configuration unless you have 16GB RAM, and that is causing high disk utilization.  See page 15 of the R80.10 release notes; this limitation continues into R80.40 so a software upgrade will not help.  You need to disable SmartEvent on your SMS object in the SmartConsole.  Your peak logging rate is also well in excess of the 210's capacity as specified here: sk112797: Smart-1 R80.x Logging Capacity Performance Improvements.

Even if you upgrade your Smart-1 210 with 16GB of RAM, you will just barely be meeting the minimum requirements for a Mgmt/Log/SmartEvent server, and I doubt you will be satisfied with the performance as your next bottleneck will be CPU.  The Smart-1 210 will reach End of Engineering Support later this year, and all support for that model will terminate in 2022. 

I'd strongly suggest replacing your 210 with a Smart-1 410 which has four cores and 32GB of RAM, concurrent with an management software upgrade to at least R80.30.

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com

View solution in original post

Highlighted

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

Timothy is right, you're lacking of hardware resources. You need a bigger box.

View solution in original post

0 Kudos
Highlighted

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

Hi All,

 

Thank you very much for your support, now i will try to disable smartevent ( have to find the procedure).

 

Regards,

Mauro de Sousa

 

0 Kudos
Highlighted

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

To disable SmartEvent, in the SmartConsole uncheck any boxes under "SmartEvent" on the General Properties...Management screen of your SMS object, then perform an Install Database operation.

 

Book "Max Power 2020: Check Point Firewall Performance Optimization" Third Edition
Now Available at www.maxpowerfirewalls.com
0 Kudos
Highlighted

Re: SMS Server R80.10 Extremely Slow and Cannot login

Jump to solution

Hi All,

 

I disable SmartEvent in the begining was slow , but after a restart of SMS, it  started to work nicely but all the Views in "Logs & Monitor" desapeared ( the only one left was "open log View"). So i Re-enable "SmartEvent" and every thing start working nicely ( a lot faster than before) . Now the CPU consumption varies between 7%-100% but always "Up-Down"  and does not stay for more than 3s in 100% . Also one thing that i notice is that the RAM consumption is gone Down ( now Used: 4.6 Gbps to 5 Gbps). I also installed "Check_Point_SmartConsole_R80_10_jumbo_HF_B161_Win" and it is working nicely.

I will monitor the behavior and if there is any  change i will let you know. 

 

Thank you ALL for your support.

 

Regards,

 

Mauro de Sousa

0 Kudos