Smart-1 5 CPU high every 4 hours

[Herschel_Liang]

Hi all,

As the title, I used sar cli can see Smart-1 5 CPU high every 4 hours, but I don't know how to trouble and analyse. 

Product version Check Point Gaia R77.30
OS build 204
OS kernel version 2.6.18-92cp
OS edition 32-bit

[Expert@Smart-1:0]# sar
Linux 2.6.18-92cp (Smart-1) 03/25/18

00:00:01 CPU %user %nice %system %iowait %steal %idle
00:10:01 all 0.60 0.00 0.96 0.25 0.00 98.19
00:20:01 all 0.70 0.00 1.12 0.28 0.00 97.89
00:30:01 all 0.67 0.00 1.09 0.25 0.00 97.98
00:40:01 all 0.58 0.00 0.91 0.05 0.00 98.45
00:50:01 all 0.65 0.00 1.09 0.26 0.00 98.00
01:00:01 all 0.69 0.00 1.08 0.26 0.00 97.96
01:10:01 all 0.60 0.00 0.94 0.04 0.00 98.42
01:20:01 all 0.83 0.00 3.46 0.31 0.00 95.39
01:30:01 all 24.27 0.00 67.64 1.06 0.00 7.02
01:40:01 all 0.59 0.00 0.92 0.07 0.00 98.41
01:50:02 all 0.65 0.00 1.07 0.23 0.00 98.04
02:00:01 all 0.85 0.00 1.22 0.32 0.00 97.62
02:10:01 all 0.60 0.00 1.08 11.13 0.00 87.19
02:20:01 all 0.70 0.00 1.14 3.90 0.00 94.25
02:30:01 all 0.63 0.00 1.09 0.27 0.00 98.01
02:40:01 all 0.56 0.00 0.94 0.05 0.00 98.46
02:50:01 all 0.63 0.00 1.08 0.26 0.00 98.02
03:00:01 all 0.67 0.00 1.09 0.28 0.00 97.96
03:10:01 all 0.61 0.00 0.94 0.13 0.00 98.32
03:20:01 all 0.69 0.00 1.10 0.29 0.00 97.93
03:30:01 all 0.66 0.00 1.08 0.27 0.00 97.98
03:40:01 all 0.60 0.00 0.94 0.06 0.00 98.40
03:50:01 all 0.66 0.00 1.09 0.26 0.00 97.99
04:00:01 all 0.86 0.00 1.24 0.32 0.00 97.57
04:10:01 all 0.60 0.00 0.98 2.93 0.00 95.49
04:20:01 all 0.72 0.00 1.11 0.29 0.00 97.88
04:30:01 all 1.23 0.00 4.02 0.43 0.00 94.32
04:40:01 all 25.66 0.00 66.85 1.19 0.00 6.30
04:50:01 all 0.67 0.00 1.07 0.27 0.00 97.98
05:00:02 all 1.24 0.00 1.15 0.21 0.00 97.40
05:10:01 all 0.54 0.00 1.01 6.42 0.00 92.04
05:20:01 all 0.64 0.01 1.04 1.11 0.00 97.20
05:30:01 all 0.51 0.00 0.86 0.05 0.00 98.58
05:40:01 all 0.63 0.00 1.03 0.28 0.00 98.05
05:50:01 all 0.61 0.00 1.01 0.25 0.00 98.12
06:00:01 all 0.68 0.00 1.01 0.13 0.00 98.18
06:10:01 all 0.59 0.00 1.10 6.42 0.00 91.88
06:20:01 all 0.64 0.00 1.10 4.17 0.00 94.09
06:30:01 all 0.49 0.00 0.85 0.05 0.00 98.61
06:40:01 all 0.61 0.00 1.03 0.28 0.00 98.08
06:50:01 all 0.59 0.00 1.03 0.28 0.00 98.10
07:00:01 all 0.51 0.00 0.87 0.06 0.00 98.56
07:10:01 all 0.63 0.00 1.06 0.25 0.00 98.05
07:20:01 all 0.63 0.00 1.08 0.27 0.00 98.02
07:30:01 all 0.49 0.00 0.86 0.06 0.00 98.59
07:40:02 all 1.53 0.00 4.84 0.57 0.00 93.06
07:50:01 all 27.15 0.00 66.46 0.74 0.00 5.64
08:00:01 all 0.71 0.00 1.01 0.10 0.00 98.19
08:10:01 all 0.65 0.00 1.15 5.76 0.00 92.44
08:20:01 all 0.64 0.00 1.10 1.01 0.00 97.24
08:30:01 all 0.52 0.00 0.86 0.06 0.00 98.56
08:40:01 all 0.64 0.00 1.10 0.34 0.00 97.91
08:50:01 all 0.62 0.00 1.06 0.27 0.00 98.06
09:00:01 all 0.55 0.00 0.88 0.07 0.00 98.50

09:00:01 CPU %user %nice %system %iowait %steal %idle
09:10:01 all 0.69 0.00 1.10 0.93 0.00 97.27
09:20:01 all 0.65 0.00 1.07 0.27 0.00 98.00
09:30:01 all 0.53 0.00 0.89 0.04 0.00 98.53
09:40:01 all 0.68 0.00 1.08 0.25 0.00 97.99
09:50:01 all 0.65 0.00 1.07 0.25 0.00 98.04
10:00:01 all 0.76 0.00 1.02 0.10 0.00 98.11
10:10:01 all 0.70 0.00 1.12 0.26 0.00 97.91
10:20:01 all 0.72 0.00 1.09 0.27 0.00 97.92
10:30:01 all 0.59 0.00 0.89 0.11 0.00 98.41
10:40:01 all 0.69 0.00 1.10 0.30 0.00 97.91
10:50:01 all 1.45 0.00 4.45 0.57 0.00 93.53
11:00:01 all 29.75 0.00 67.21 0.72 0.00 2.31
11:10:01 all 0.83 0.00 1.22 4.39 0.00 93.55
11:20:01 all 0.70 0.00 1.10 0.25 0.00 97.96
11:30:01 all 0.60 0.00 0.92 0.05 0.00 98.43
11:40:01 all 0.70 0.00 1.09 0.25 0.00 97.96
11:50:01 all 0.70 0.00 1.10 0.26 0.00 97.94
12:00:01 all 0.78 0.00 1.05 1.08 0.00 97.08
12:10:01 all 0.73 0.00 1.18 5.92 0.00 92.17
12:20:01 all 0.71 0.00 1.09 0.57 0.00 97.63
12:30:01 all 0.60 0.00 0.89 0.04 0.00 98.46
12:40:01 all 0.70 0.00 1.08 0.27 0.00 97.94
12:50:01 all 0.68 0.00 1.09 0.27 0.00 97.96
13:00:01 all 0.58 0.00 0.92 0.05 0.00 98.45
13:10:01 all 0.75 0.00 1.17 4.84 0.00 93.24
13:20:01 all 0.68 0.00 1.08 0.27 0.00 97.96
13:30:01 all 0.60 0.00 0.90 0.07 0.00 98.43
13:40:01 all 0.67 0.00 1.10 0.26 0.00 97.98
13:50:01 all 0.69 0.00 1.07 0.24 0.00 97.99
14:00:01 all 0.90 0.00 3.19 0.11 0.00 95.80
14:10:01 all 31.63 0.01 64.49 1.49 0.00 2.38
14:20:01 all 0.97 0.00 5.26 0.51 0.00 93.26
14:30:01 all 0.59 0.00 0.92 0.08 0.00 98.41
14:40:01 all 0.70 0.00 1.08 0.28 0.00 97.94
14:50:01 all 0.69 0.00 1.08 0.24 0.00 97.99
15:00:01 all 0.59 0.00 0.91 0.07 0.00 98.42
15:10:01 all 0.75 0.00 1.19 5.48 0.00 92.58
15:20:01 all 0.70 0.01 1.12 0.34 0.00 97.84
15:30:01 all 0.60 0.00 0.92 0.06 0.00 98.42
15:40:01 all 0.70 0.00 1.10 0.27 0.00 97.93
15:50:01 all 0.69 0.00 1.09 0.25 0.00 97.97
16:00:01 all 0.78 0.00 1.04 0.10 0.00 98.08
16:10:01 all 0.75 0.00 1.13 0.26 0.00 97.87
16:20:01 all 0.72 0.00 1.08 0.25 0.00 97.94
16:30:01 all 0.63 0.00 0.94 0.06 0.00 98.37
16:40:01 all 0.74 0.00 1.09 0.27 0.00 97.90
16:50:01 all 0.75 0.00 1.14 0.31 0.00 97.81
17:00:01 all 1.18 0.00 0.99 0.19 0.00 97.65
17:10:01 all 0.78 0.00 1.18 4.12 0.00 93.92
17:20:03 all 31.74 0.01 59.97 1.32 0.00 6.96
17:30:01 all 2.73 0.00 12.23 0.32 0.00 84.72
17:40:01 all 0.75 0.00 1.14 0.30 0.00 97.81
17:50:01 all 0.94 0.00 5.09 0.28 0.00 93.69
18:00:01 all 1.53 0.00 12.70 0.17 0.00 85.60

18:00:01 CPU %user %nice %system %iowait %steal %idle
18:10:01 all 0.75 0.00 1.17 5.91 0.00 92.17
18:20:01 all 0.76 0.00 1.15 2.87 0.00 95.21
18:30:01 all 0.61 0.00 0.93 0.06 0.00 98.40
18:40:01 all 0.71 0.00 1.09 0.26 0.00 97.94
18:50:01 all 1.20 0.00 1.15 0.59 0.00 97.06
19:00:01 all 0.59 0.00 0.91 0.05 0.00 98.45
19:10:01 all 0.75 0.00 1.16 5.92 0.00 92.17
19:20:01 all 0.74 0.00 1.15 4.21 0.00 93.90
19:30:01 all 0.61 0.00 0.90 0.06 0.00 98.43
19:40:01 all 0.71 0.00 1.10 0.26 0.00 97.93
19:50:01 all 0.71 0.00 1.09 0.26 0.00 97.94
20:00:01 all 0.80 0.00 1.06 0.13 0.00 98.02
20:10:01 all 0.76 0.00 1.17 4.68 0.00 93.39
20:20:01 all 0.73 0.00 1.11 0.29 0.00 97.87
20:30:01 all 30.91 0.00 53.07 0.88 0.00 15.14
20:40:01 all 5.50 0.00 19.67 0.27 0.00 74.57
20:50:01 all 1.29 0.00 1.16 0.60 0.00 96.95
21:00:01 all 1.41 0.00 2.30 20.83 0.00 75.46
21:10:01 all 0.88 0.00 1.21 5.98 0.00 91.92
21:20:01 all 0.84 0.00 1.71 6.79 0.00 90.66
21:30:01 all 0.62 0.00 0.91 0.05 0.00 98.42
21:40:01 all 0.71 0.00 1.08 0.27 0.00 97.94
21:50:01 all 0.70 0.00 1.08 0.27 0.00 97.94
22:00:01 all 0.76 0.00 1.03 0.08 0.00 98.13
22:10:01 all 0.70 0.00 1.10 0.53 0.00 97.66
22:20:01 all 0.67 0.00 1.10 0.23 0.00 97.99
22:30:01 all 0.54 0.00 0.88 0.05 0.00 98.53
22:40:02 all 0.64 0.00 1.05 0.25 0.00 98.06
22:50:01 all 0.73 0.00 1.08 0.28 0.00 97.92
23:00:01 all 1.56 0.00 1.35 0.25 0.00 96.83
23:10:01 all 0.88 0.00 1.22 0.26 0.00 97.64
23:20:01 all 0.69 0.00 1.06 0.25 0.00 98.00
23:30:01 all 0.55 0.00 0.90 0.04 0.00 98.50
23:40:01 all 12.61 0.00 61.37 1.18 0.00 24.83
23:50:01 all 1.70 0.00 10.39 1.28 0.00 86.63
Average: all 2.24 0.00 5.07 1.13 0.00 91.56

Because I don't found exactly time, I can not find which process lead to this issue. Who can give me some advice? THX!

Best Regards!

[AlekseiShelepov]

Do you have Jumbo Hotfix Accumulator installed? If yes, is it one of recent versions?

I would recommend to look into historic data in cpview to find some additional information:

cpview -t <timestamp>

Shows either the oldest available content, or from a given <timestamp>, where <timestamp> format is: [Jan...Dec] [01...31] [4-digit Year] [hh:mm:ss]

Example: [Expert@HostName]# cpview -t Oct 24 2016 10:54:15

You can go to somewhere close to the time with high CPU usage and use +/- buttons to move in cpview logs.

[Herschel_Liang]

Hi Aleksei,

It seem that it's not a bug and cpview -t <timestamp> is not work in SMS.

Smart-1> cpview -t Mar 25 2018 23:14:23
CPView: Failed to open history database

Best Regards!

[Kaspars_Zibarts]

LikeL like history is disabled, check with

cpview history stat

[Herschel_Liang]

[Expert@Smart-1:0]# cpview history stat
Usage: cpview [ [-c <conf_file>] | [-p] ] | [-b [-t <sec> [-i <count>] [-j] [-l <filesize>] ] | [-s]]
-c Loads configuration from <conf_file>
-p Prints all cpview data
-b Print batch statistics data
-t In the background, data is dumped into logs. Time interval is <sec> seconds before the new dump info is generated. The value of <sec> must be greater than 30. The log is saved to var/log/cpview
-i Limit dump info to <count> times
-j Compresses the generated logs after you run cpview -b -s to stop the instance
-l Change log file size limit to <filesize>MB (default: 1024MB)
-s Stops "cpview -b" instance that runs in the background

[Expert@Smart-1:0]# exit
exit
Smart-1> cpview history stat
Usage: cpview [ [-c <conf_file>] | [-p] ] | [-b [-t <sec> [-i <count>] [-j] [-l <filesize>] ] | [-s]]
-c Loads configuration from <conf_file>
-p Prints all cpview data
-b Print batch statistics data
-t In the background, data is dumped into logs. Time interval is <sec> seconds before the new dump info is generated. The value of <sec> must be greater than 30. The log is saved to var/log/cpview
-i Limit dump info to <count> times
-j Compresses the generated logs after you run cpview -b -s to stop the instance
-l Change log file size limit to <filesize>MB (default: 1024MB)
-s Stops "cpview -b" instance that runs in the background

Smart-1>

[Kaspars_Zibarts]

haha, not used to SMS - only ever worked on MDS that does not have cpview and gateway or standalones that have history option. sorry, have to leave it for someone with SMS/Smart-1 experience

[Maarten_Sjouw]

Have you looked at the messages file for the same timestamps?

Looks like there is a indexing task running every 3 hours and 10 minutes (190 mins), also look at crontab -l to see if anything is scheduled in cron.

[Kaspars_Zibarts]

That is correct - so you should be able to predict next high CPU and then run cpview in the backgound (reading the help you provided)

cpview -b -t 30      to start it

cpview -s          to stop

cpview -p          to print it out when done

worth a shot

[Herschel_Liang]

Yes, I had saw messages and crontab -l find out the problem. thx!

[PhoneBoy]

What did it end up being?

[Herschel_Liang]

crontab -l  show some scheduled task and I delete some useless cron. And CPU is lower than before every 4 hours.

[Garrett_DirSec]

The Smart1-5 was an under-powered platform when it was released in Jun 2009.     It was effectively a laptop platform in a rack-mount chassis.     I doubt has multi-core CPU and memory was minimal.    This means the most simple scheduled system event could cause significant resource consumption and spikes.

A little known fact is that the Smart1-5 license container will successfully attach to open-server build of R77.30 on vmware.   this is not technically supported but has been used by local CP team to connect dots to new renewal and hardware upgrade to Smart1-205 (or 405 now). 

reference:   hardware support policy