Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Herschel_Liang
Collaborator

Smart-1 5 CPU high every 4 hours

Hi all,

As the title, I used sar cli can see Smart-1 5 CPU high every 4 hours, but I don't know how to trouble and analyse. 

Product version Check Point Gaia R77.30
OS build 204
OS kernel version 2.6.18-92cp
OS edition 32-bit

[Expert@Smart-1:0]# sar
Linux 2.6.18-92cp (Smart-1) 03/25/18

00:00:01 CPU %user %nice %system %iowait %steal %idle
00:10:01 all 0.60 0.00 0.96 0.25 0.00 98.19
00:20:01 all 0.70 0.00 1.12 0.28 0.00 97.89
00:30:01 all 0.67 0.00 1.09 0.25 0.00 97.98
00:40:01 all 0.58 0.00 0.91 0.05 0.00 98.45
00:50:01 all 0.65 0.00 1.09 0.26 0.00 98.00
01:00:01 all 0.69 0.00 1.08 0.26 0.00 97.96
01:10:01 all 0.60 0.00 0.94 0.04 0.00 98.42
01:20:01 all 0.83 0.00 3.46 0.31 0.00 95.39
01:30:01 all 24.27 0.00 67.64 1.06 0.00 7.02
01:40:01 all 0.59 0.00 0.92 0.07 0.00 98.41
01:50:02 all 0.65 0.00 1.07 0.23 0.00 98.04
02:00:01 all 0.85 0.00 1.22 0.32 0.00 97.62
02:10:01 all 0.60 0.00 1.08 11.13 0.00 87.19
02:20:01 all 0.70 0.00 1.14 3.90 0.00 94.25
02:30:01 all 0.63 0.00 1.09 0.27 0.00 98.01
02:40:01 all 0.56 0.00 0.94 0.05 0.00 98.46
02:50:01 all 0.63 0.00 1.08 0.26 0.00 98.02
03:00:01 all 0.67 0.00 1.09 0.28 0.00 97.96
03:10:01 all 0.61 0.00 0.94 0.13 0.00 98.32
03:20:01 all 0.69 0.00 1.10 0.29 0.00 97.93
03:30:01 all 0.66 0.00 1.08 0.27 0.00 97.98
03:40:01 all 0.60 0.00 0.94 0.06 0.00 98.40
03:50:01 all 0.66 0.00 1.09 0.26 0.00 97.99
04:00:01 all 0.86 0.00 1.24 0.32 0.00 97.57
04:10:01 all 0.60 0.00 0.98 2.93 0.00 95.49
04:20:01 all 0.72 0.00 1.11 0.29 0.00 97.88
04:30:01 all 1.23 0.00 4.02 0.43 0.00 94.32
04:40:01 all 25.66 0.00 66.85 1.19 0.00 6.30
04:50:01 all 0.67 0.00 1.07 0.27 0.00 97.98
05:00:02 all 1.24 0.00 1.15 0.21 0.00 97.40
05:10:01 all 0.54 0.00 1.01 6.42 0.00 92.04
05:20:01 all 0.64 0.01 1.04 1.11 0.00 97.20
05:30:01 all 0.51 0.00 0.86 0.05 0.00 98.58
05:40:01 all 0.63 0.00 1.03 0.28 0.00 98.05
05:50:01 all 0.61 0.00 1.01 0.25 0.00 98.12
06:00:01 all 0.68 0.00 1.01 0.13 0.00 98.18
06:10:01 all 0.59 0.00 1.10 6.42 0.00 91.88
06:20:01 all 0.64 0.00 1.10 4.17 0.00 94.09
06:30:01 all 0.49 0.00 0.85 0.05 0.00 98.61
06:40:01 all 0.61 0.00 1.03 0.28 0.00 98.08
06:50:01 all 0.59 0.00 1.03 0.28 0.00 98.10
07:00:01 all 0.51 0.00 0.87 0.06 0.00 98.56
07:10:01 all 0.63 0.00 1.06 0.25 0.00 98.05
07:20:01 all 0.63 0.00 1.08 0.27 0.00 98.02
07:30:01 all 0.49 0.00 0.86 0.06 0.00 98.59
07:40:02 all 1.53 0.00 4.84 0.57 0.00 93.06
07:50:01 all 27.15 0.00 66.46 0.74 0.00 5.64
08:00:01 all 0.71 0.00 1.01 0.10 0.00 98.19
08:10:01 all 0.65 0.00 1.15 5.76 0.00 92.44
08:20:01 all 0.64 0.00 1.10 1.01 0.00 97.24
08:30:01 all 0.52 0.00 0.86 0.06 0.00 98.56
08:40:01 all 0.64 0.00 1.10 0.34 0.00 97.91
08:50:01 all 0.62 0.00 1.06 0.27 0.00 98.06
09:00:01 all 0.55 0.00 0.88 0.07 0.00 98.50

09:00:01 CPU %user %nice %system %iowait %steal %idle
09:10:01 all 0.69 0.00 1.10 0.93 0.00 97.27
09:20:01 all 0.65 0.00 1.07 0.27 0.00 98.00
09:30:01 all 0.53 0.00 0.89 0.04 0.00 98.53
09:40:01 all 0.68 0.00 1.08 0.25 0.00 97.99
09:50:01 all 0.65 0.00 1.07 0.25 0.00 98.04
10:00:01 all 0.76 0.00 1.02 0.10 0.00 98.11
10:10:01 all 0.70 0.00 1.12 0.26 0.00 97.91
10:20:01 all 0.72 0.00 1.09 0.27 0.00 97.92
10:30:01 all 0.59 0.00 0.89 0.11 0.00 98.41
10:40:01 all 0.69 0.00 1.10 0.30 0.00 97.91
10:50:01 all 1.45 0.00 4.45 0.57 0.00 93.53
11:00:01 all 29.75 0.00 67.21 0.72 0.00 2.31
11:10:01 all 0.83 0.00 1.22 4.39 0.00 93.55
11:20:01 all 0.70 0.00 1.10 0.25 0.00 97.96
11:30:01 all 0.60 0.00 0.92 0.05 0.00 98.43
11:40:01 all 0.70 0.00 1.09 0.25 0.00 97.96
11:50:01 all 0.70 0.00 1.10 0.26 0.00 97.94
12:00:01 all 0.78 0.00 1.05 1.08 0.00 97.08
12:10:01 all 0.73 0.00 1.18 5.92 0.00 92.17
12:20:01 all 0.71 0.00 1.09 0.57 0.00 97.63
12:30:01 all 0.60 0.00 0.89 0.04 0.00 98.46
12:40:01 all 0.70 0.00 1.08 0.27 0.00 97.94
12:50:01 all 0.68 0.00 1.09 0.27 0.00 97.96
13:00:01 all 0.58 0.00 0.92 0.05 0.00 98.45
13:10:01 all 0.75 0.00 1.17 4.84 0.00 93.24
13:20:01 all 0.68 0.00 1.08 0.27 0.00 97.96
13:30:01 all 0.60 0.00 0.90 0.07 0.00 98.43
13:40:01 all 0.67 0.00 1.10 0.26 0.00 97.98
13:50:01 all 0.69 0.00 1.07 0.24 0.00 97.99
14:00:01 all 0.90 0.00 3.19 0.11 0.00 95.80
14:10:01 all 31.63 0.01 64.49 1.49 0.00 2.38
14:20:01 all 0.97 0.00 5.26 0.51 0.00 93.26
14:30:01 all 0.59 0.00 0.92 0.08 0.00 98.41
14:40:01 all 0.70 0.00 1.08 0.28 0.00 97.94
14:50:01 all 0.69 0.00 1.08 0.24 0.00 97.99
15:00:01 all 0.59 0.00 0.91 0.07 0.00 98.42
15:10:01 all 0.75 0.00 1.19 5.48 0.00 92.58
15:20:01 all 0.70 0.01 1.12 0.34 0.00 97.84
15:30:01 all 0.60 0.00 0.92 0.06 0.00 98.42
15:40:01 all 0.70 0.00 1.10 0.27 0.00 97.93
15:50:01 all 0.69 0.00 1.09 0.25 0.00 97.97
16:00:01 all 0.78 0.00 1.04 0.10 0.00 98.08
16:10:01 all 0.75 0.00 1.13 0.26 0.00 97.87
16:20:01 all 0.72 0.00 1.08 0.25 0.00 97.94
16:30:01 all 0.63 0.00 0.94 0.06 0.00 98.37
16:40:01 all 0.74 0.00 1.09 0.27 0.00 97.90
16:50:01 all 0.75 0.00 1.14 0.31 0.00 97.81
17:00:01 all 1.18 0.00 0.99 0.19 0.00 97.65
17:10:01 all 0.78 0.00 1.18 4.12 0.00 93.92
17:20:03 all 31.74 0.01 59.97 1.32 0.00 6.96
17:30:01 all 2.73 0.00 12.23 0.32 0.00 84.72
17:40:01 all 0.75 0.00 1.14 0.30 0.00 97.81
17:50:01 all 0.94 0.00 5.09 0.28 0.00 93.69
18:00:01 all 1.53 0.00 12.70 0.17 0.00 85.60

18:00:01 CPU %user %nice %system %iowait %steal %idle
18:10:01 all 0.75 0.00 1.17 5.91 0.00 92.17
18:20:01 all 0.76 0.00 1.15 2.87 0.00 95.21
18:30:01 all 0.61 0.00 0.93 0.06 0.00 98.40
18:40:01 all 0.71 0.00 1.09 0.26 0.00 97.94
18:50:01 all 1.20 0.00 1.15 0.59 0.00 97.06
19:00:01 all 0.59 0.00 0.91 0.05 0.00 98.45
19:10:01 all 0.75 0.00 1.16 5.92 0.00 92.17
19:20:01 all 0.74 0.00 1.15 4.21 0.00 93.90
19:30:01 all 0.61 0.00 0.90 0.06 0.00 98.43
19:40:01 all 0.71 0.00 1.10 0.26 0.00 97.93
19:50:01 all 0.71 0.00 1.09 0.26 0.00 97.94
20:00:01 all 0.80 0.00 1.06 0.13 0.00 98.02
20:10:01 all 0.76 0.00 1.17 4.68 0.00 93.39
20:20:01 all 0.73 0.00 1.11 0.29 0.00 97.87
20:30:01 all 30.91 0.00 53.07 0.88 0.00 15.14
20:40:01 all 5.50 0.00 19.67 0.27 0.00 74.57
20:50:01 all 1.29 0.00 1.16 0.60 0.00 96.95
21:00:01 all 1.41 0.00 2.30 20.83 0.00 75.46
21:10:01 all 0.88 0.00 1.21 5.98 0.00 91.92
21:20:01 all 0.84 0.00 1.71 6.79 0.00 90.66
21:30:01 all 0.62 0.00 0.91 0.05 0.00 98.42
21:40:01 all 0.71 0.00 1.08 0.27 0.00 97.94
21:50:01 all 0.70 0.00 1.08 0.27 0.00 97.94
22:00:01 all 0.76 0.00 1.03 0.08 0.00 98.13
22:10:01 all 0.70 0.00 1.10 0.53 0.00 97.66
22:20:01 all 0.67 0.00 1.10 0.23 0.00 97.99
22:30:01 all 0.54 0.00 0.88 0.05 0.00 98.53
22:40:02 all 0.64 0.00 1.05 0.25 0.00 98.06
22:50:01 all 0.73 0.00 1.08 0.28 0.00 97.92
23:00:01 all 1.56 0.00 1.35 0.25 0.00 96.83
23:10:01 all 0.88 0.00 1.22 0.26 0.00 97.64
23:20:01 all 0.69 0.00 1.06 0.25 0.00 98.00
23:30:01 all 0.55 0.00 0.90 0.04 0.00 98.50
23:40:01 all 12.61 0.00 61.37 1.18 0.00 24.83
23:50:01 all 1.70 0.00 10.39 1.28 0.00 86.63
Average: all 2.24 0.00 5.07 1.13 0.00 91.56

Because I don't found exactly time, I can not find which process lead to this issue. Who can give me some advice? THX!

Best Regards!

11 Replies
AlekseiShelepov
Advisor

Do you have Jumbo Hotfix Accumulator installed? If yes, is it one of recent versions?

I would recommend to look into historic data in cpview to find some additional information:

cpview -t <timestamp>

Shows either the oldest available content, or from a given <timestamp>, where <timestamp> format is: [Jan...Dec] [01...31] [4-digit Year] [hh:mm:ss]

Example: [Expert@HostName]# cpview -t Oct 24 2016 10:54:15

You can go to somewhere close to the time with high CPU usage and use +/- buttons to move in cpview logs.

0 Kudos
Herschel_Liang
Collaborator

Hi Aleksei,

It seem that it's not a bug and cpview -t <timestamp> is not work in SMS.

Smart-1> cpview -t Mar 25 2018 23:14:23
CPView: Failed to open history database

Best Regards!

0 Kudos
Kaspars_Zibarts
Authority
Authority

LikeL like history is disabled, check with

cpview history stat

0 Kudos
Herschel_Liang
Collaborator

[Expert@Smart-1:0]# cpview history stat
Usage: cpview [ [-c <conf_file>] | [-p] ] | [-b [-t <sec> [-i <count>] [-j] [-l <filesize>] ] | [-s]]
-c Loads configuration from <conf_file>
-p Prints all cpview data
-b Print batch statistics data
-t In the background, data is dumped into logs. Time interval is <sec> seconds before the new dump info is generated. The value of <sec> must be greater than 30. The log is saved to var/log/cpview
-i Limit dump info to <count> times
-j Compresses the generated logs after you run cpview -b -s to stop the instance
-l Change log file size limit to <filesize>MB (default: 1024MB)
-s Stops "cpview -b" instance that runs in the background

[Expert@Smart-1:0]# exit
exit
Smart-1> cpview history stat
Usage: cpview [ [-c <conf_file>] | [-p] ] | [-b [-t <sec> [-i <count>] [-j] [-l <filesize>] ] | [-s]]
-c Loads configuration from <conf_file>
-p Prints all cpview data
-b Print batch statistics data
-t In the background, data is dumped into logs. Time interval is <sec> seconds before the new dump info is generated. The value of <sec> must be greater than 30. The log is saved to var/log/cpview
-i Limit dump info to <count> times
-j Compresses the generated logs after you run cpview -b -s to stop the instance
-l Change log file size limit to <filesize>MB (default: 1024MB)
-s Stops "cpview -b" instance that runs in the background

Smart-1>

0 Kudos
Kaspars_Zibarts
Authority
Authority

haha, not used to SMS - only ever worked on MDS that does not have cpview and gateway or standalones that have history option. sorry, have to leave it for someone with SMS/Smart-1 experience

0 Kudos
Maarten_Sjouw
Champion
Champion

Have you looked at the messages file for the same timestamps?

Looks like there is a indexing task running every 3 hours and 10 minutes (190 mins), also look at crontab -l to see if anything is scheduled in cron.

Regards, Maarten
0 Kudos
Kaspars_Zibarts
Authority
Authority

That is correct - so you should be able to predict next high CPU and then run cpview in the backgound (reading the help you provided)

cpview -b -t 30      to start it

cpview -s          to stop

cpview -p          to print it out when done

worth a shot

0 Kudos
Herschel_Liang
Collaborator

Yes, I had saw messages and crontab -l find out the problem. thx!

0 Kudos
PhoneBoy
Admin
Admin

What did it end up being?

0 Kudos
Herschel_Liang
Collaborator

crontab -l  show some scheduled task and I delete some useless cron. And CPU is lower than before every 4 hours.

0 Kudos
Garrett_Anderso
Advisor

The Smart1-5 was an under-powered platform when it was released in Jun 2009.     It was effectively a laptop platform in a rack-mount chassis.     I doubt has multi-core CPU and memory was minimal.    This means the most simple scheduled system event could cause significant resource consumption and spikes.

A little known fact is that the Smart1-5 license container will successfully attach to open-server build of R77.30 on vmware.   this is not technically supported but has been used by local CP team to connect dots to new renewal and hardware upgrade to Smart1-205 (or 405 now). 

reference:   hardware support policy