Postgres consuming %100 percent of CPU on Smart-1 210 management server

[Champika_Gamage]

The manager is consistently at 96-100% CPU all the time. When checking the processes below came up. 

Check Point Security Management - R80.20

Kernel:                 3.10.0-693cpx86_64

Edition:                 64-bit

Build Number:  117

Uptime:               74 days 2 hours 57 minutes

Updates:             no new recommended updates detected

Top shows these 4 postgres processes consuming all CPU:

top - 16:39:35 up 74 days,  2:21,  1 user,  load average: 4.13, 4.36, 4.46

Tasks: 197 total,   6 running, 191 sleeping,   0 stopped,   0 zombie

%Cpu0  : 99.0 us,  1.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

%Cpu1  : 99.7 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

KiB Mem :  7924680 total,   568584 free,  4673312 used,  2682784 buff/cache

KiB Swap: 16185480 total, 15603176 free,   582304 used.  2663028 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

11406 cp_post+  20   0  471580 320288  39056 R  49.8  4.0  51034:31 postgres

10157 cp_post+  20   0  383168 270828  39264 R  48.8  3.4  33562:41 postgres

13914 cp_post+  20   0  476088 282916  39052 R  48.8  3.6  51043:12 postgres

10207 cp_post+  20   0  396312 296220  39260 R  48.5  3.7  31803:52 postgres

Checking to see what these four processes are doing:

datid |  datname   |  pid  | usesysid | usename  | application_name | client_addr | client_hostname | client_port |         backend_start         |          xact_start           |          query_start          |         state_change

      | waiting |        state        |

                                                                           query

-------+------------+-------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+---------+---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

16385 | cpm        | 10157 |    16384 | postgres |                  | 127.0.0.1   |                 |       34966 | 2020-06-25 07:56:44.398143+03 | 2020-09-07 09:12:11.605852+03 | 2020-09-07 10:03:03.397844+03 | 2020-09-07 10:03:03.3978

48+03 | f       | active              | SELECT objid AS a1, DTYPE AS a2, ADDITIONALINFO AS a3, ADMINID AS a4, ADMINNAME AS a5, APPLICATIONNAME AS a6, AUDITSTATUS AS a7, DISPLAYADDITIONALINFO AS a8, DISPLAYEDOBJCLASSNAME AS a9, DISPLAYEDOBJCOLOR AS a10, DISPLAYEDOBJFIELDSCHANGES AS a11, DISPLAYEDOBJICON AS a12, DISPLAYEDOBJID AS a13, displayedobjname AS a14, DOMAINID AS a15, DOMAINVERSION AS a16, IPADDRESS AS a17, LOGID AS a18, LOGSEQUENCEID AS a19, MACHINE AS a20, PARENTAUDITLOGID AS a21, PUBLICSESSION AS a22, PUBLISHDYNAMICATTRIBUTES AS a23, SYNCTRANSACTIONID AS a24, TIME AS a25, TRANSACTIONID AS a26, CONFIGCHANGEAUDITLOG AS a27, DOMAINWORKSESSIONID AS a28, TRIGGERPHASE AS a29, WORKSESSIONID AS a30, BLADEID AS a31, TARGETOBJID AS a32, TARGETOBJIDS AS a33, WORKSESSION AS a34, NUMBEROFPUBLISHEDCHANGES AS a35, RULEBASEID AS a36, ENABLED AS a37, ENTITYID AS a38, RULEBASEENTITYTYPE AS a39, INTERNALFILE AS a40, REQUEST AS a41, OBJECTNAME AS a42, SECONDARYMANAGEMENT AS a43, SICNAME AS a44, TYPE AS a45, PARENTID AS a46, POSITION AS a47, RULEBASESECTION A

16385 | cpm        | 10207 |    16384 | postgres |                  | 127.0.0.1   |                 |       35004 | 2020-06-25 07:56:52.946237+03 | 2020-09-07 08:24:28.192655+03 | 2020-09-07 10:01:16.025558+03 | 2020-09-07 10:01:16.0255

61+03 | f       | active              | SELECT objid AS a1, DTYPE AS a2, ADDITIONALINFO AS a3, ADMINID AS a4, ADMINNAME AS a5, APPLICATIONNAME AS a6, AUDITSTATUS AS a7, DISPLAYADDITIONALINFO AS a8, DISPLAYEDOBJCLASSNAME AS a9, DISPLAYEDOBJCOLOR AS a10, DISPLAYEDOBJFIELDSCHANGES AS a11, DISPLAYEDOBJICON AS a12, DISPLAYEDOBJID AS a13, displayedobjname AS a14, DOMAINID AS a15, DOMAINVERSION AS a16, IPADDRESS AS a17, LOGID AS a18, LOGSEQUENCEID AS a19, MACHINE AS a20, PAREN

TAUDITLOGID AS a21, PUBLICSESSION AS a22, PUBLISHDYNAMICATTRIBUTES AS a23, SYNCTRANSACTIONID AS a24, TIME AS a25, TRANSACTIONID AS a26, CONFIGCHANGEAUDITLOG AS a27, DOMAINWORKSESSIONID AS a28, TRIGGERPHASE AS a29, WORKSESSIONID AS a30, BLADEID AS a31, TARGETOBJID AS a32, TARGETOBJIDS AS a33, WORKSESSION AS a34, NUMBEROFPUBLISHEDCHANGES AS a35, RULEBASEID AS a36, ENABLED AS a37, ENTITYID AS a38, RULEBASEENTITYTYPE AS a39, INTERNALFILE AS a40, REQUEST AS a41, OBJECTNAME A

S a42, SECONDARYMANAGEMENT AS a43, SICNAME AS a44, TYPE AS a45, PARENTID AS a46, POSITION AS a47, RULEBASESECTION A

16385 | cpm        | 11406 |    16384 | postgres |                  | 127.0.0.1   |                 |       35138 | 2020-06-25 08:01:58.448874+03 | 2020-09-07 08:55:10.086094+03 | 2020-09-07 09:59:25.705632+03 | 2020-09-07 09:59:25.7056

35+03 | f       | active              | SELECT objid AS a1, DTYPE AS a2, ADDITIONALINFO AS a3, ADMINID AS a4, ADMINNAME AS a5, APPLICATIONNAME AS a6, AUDITSTATUS AS a7, DISPLAYADDITIONALINFO AS a8, DISPLAYEDOBJCLASSNAME AS a9, DISPLAYEDOBJCOLOR AS a10, DISPLAYEDOBJFIELDSCHANGES AS a11, DISPLAYEDOBJICON AS a12, DISPLAYEDOBJID AS a13, displayedobjname AS a14, DOMAINID AS a15, DOMAINVERSION AS a16, IPADDRESS AS a17, LOGID AS a18, LOGSEQUENCEID AS a19, MACHINE AS a20, PARENTAUDITLOGID AS a21, PUBLICSESSION AS a22, PUBLISHDYNAMICATTRIBUTES AS a23, SYNCTRANSACTIONID AS a24, TIME AS a25, TRANSACTIONID AS a26, CONFIGCHANGEAUDITLOG AS a27, DOMAINWORKSESSIONID AS a28, TRIGGERPHASE AS a29, WORKSESSIONID AS a30, BLADEID AS a31, TARGETOBJID AS a32, TARGETOBJIDS AS a33, WORKSESSION AS a34, NUMBEROFPUBLISHEDCHANGES AS a35, RULEBASEID AS a36, ENABLED AS a37, ENTITYID AS a38, RULEBASEENTITYTYPE AS a39, INTERNALFILE AS a40, REQUEST AS a41, OBJECTNAME AS a42, SECONDARYMANAGEMENT AS a43, SICNAME AS a44, TYPE AS a45, PARENTID AS a46, POSITION AS a47, RULEBASESECTION A

16385 | cpm        | 13914 |    16384 | postgres |                  | 127.0.0.1   |                 |       38804 | 2020-06-25 08:14:28.519961+03 | 2020-09-07 09:44:48.436818+03 | 2020-09-07 09:58:55.704017+03 | 2020-09-07 09:58:55.7040

2+03  | f       | active              | SELECT objid AS a1, DTYPE AS a2, ADDITIONALINFO AS a3, ADMINID AS a4, ADMINNAME AS a5, APPLICATIONNAME AS a6, AUDITSTATUS AS a7, DISPLAYADDITIONALINFO AS a8, DISPLAYEDOBJCLASSNAME AS a9, DISPLAYEDOBJCOLOR AS a10, DISPLAYEDOBJFIELDSCHANGES AS a11, DISPLAYEDOBJICON AS a12, DISPLAYEDOBJID AS a13, displayedobjname AS a14, DOMAINID AS a15, DOMAINVERSION AS a16, IPADDRESS AS a17, LOGID AS a18, LOGSEQUENCEID AS a19, MACHINE AS a20, PARENTAUDITLOGID AS a21, PUBLICSESSION AS a22, PUBLISHDYNAMICATTRIBUTES AS a23, SYNCTRANSACTIONID AS a24, TIME AS a25, TRANSACTIONID AS a26, CONFIGCHANGEAUDITLOG AS a27, DOMAINWORKSESSIONID AS a28, TRIGGERPHASE AS a29, WORKSESSIONID AS a30, BLADEID AS a31, TARGETOBJID AS a32, TARGETOBJIDS AS a33, WORKSESSION AS a34, NUMBEROFPUBLISHEDCHANGES AS a35, RULEBASEID AS a36, ENABLED AS a37, ENTITYID AS a38, RULEBASEENTITYTYPE AS a39, INTERNALFILE AS a40, REQUEST AS a41, OBJECTNAME AS a42, SECONDARYMANAGEMENT AS a43, SICNAME AS a44, TYPE AS a45, PARENTID AS a46, POSITION AS a47, RULEBASESECTION A

Any ideas why this device might be running 100% CPU 24/7?  

Those 4 listed processes seem to be there all the time – non-stop. Are these four processes normal?

[PhoneBoy]

Postgres processes are normal.
Probably a good idea to get the TAC involved here.

[Martin_Oles]

Hi,

I am observing the same issue MultiDomain R80.30 HFA 227. Postgres process is taking full 5 CPUs (from 12 available):

top - 11:44:36 up 4 days, 14:00, 1 user, load average: 4.72, 4.99, 5.02
Tasks: 463 total, 6 running, 457 sleeping, 0 stopped, 0 zombie
%Cpu0 : 89.7 us, 0.3 sy, 0.3 ni, 9.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 1.0 us, 0.3 sy, 0.3 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu2 : 99.7 us, 0.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 64.9 us, 1.3 sy, 0.3 ni, 33.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 10.0 us, 1.7 sy, 1.0 ni, 85.3 id, 2.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 6.4 us, 0.7 sy, 1.0 ni, 91.6 id, 0.3 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 25.3 us, 0.3 sy, 1.0 ni, 72.7 id, 0.7 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 10.3 us, 0.3 sy, 0.3 ni, 89.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 64.7 us, 1.0 sy, 0.0 ni, 34.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 14.0 us, 1.0 sy, 0.7 ni, 83.7 id, 0.7 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 25.9 us, 0.3 sy, 0.0 ni, 72.8 id, 1.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 99.7 us, 0.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 65940488 total, 15140260 free, 13446256 used, 37353972 buff/cache
KiB Swap: 33029636 total, 32975876 free, 53760 used. 50116468 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21548 cp_post+ 20 0 1630856 1.503g 1.296g R 100.0 2.4 6539:22 postgres
21479 cp_post+ 20 0 1544560 1.423g 1.296g R 100.0 2.3 5929:01 postgres
21535 cp_post+ 20 0 1472688 1.371g 1.297g R 100.0 2.2 2196:09 postgres
23331 cp_post+ 20 0 1558468 1.426g 1.292g R 99.7 2.3 6539:23 postgres
20940 cp_post+ 20 0 1596296 1.462g 1.290g R 99.3 2.3 6528:01 postgres

I am having Check point service request opened, but apart "update to latest HFA", no real solution or advice has been given.

Command psql_client cpm postgres -c "SELECT * FROM pg_stat_activity;" give the same output for all affected processes:

16385 | cpm | 23331 | 16384 | postgres | | 127.0.0.1 | | 61370 | 202
1-02-03 21:47:30.104155+02 | 2021-02-08 10:17:18.455634+02 | 2021-02-08 11:43:57.043566+02 | 2021-02-08 11:43:57.043568+
02 | f | active | SELECT objid AS a1, DTYPE AS a2, ADDITIONALINFO AS a3, ADMINID AS a4, ADMINNAME AS
a5, APPLICATIONNAME AS a6, AUDITSTATUS AS a7, DISPLAYADDITIONALINFO AS a8, DISPLAYEDOBJCLASSNAME AS a9, DISPLAYEDOBJCOLO
R AS a10, DISPLAYEDOBJFIELDSCHANGES AS a11, DISPLAYEDOBJICON AS a12, DISPLAYEDOBJID AS a13, displayedobjname AS a14, DOM
AINID AS a15, DOMAINVERSION AS a16, IPADDRESS AS a17, LOGID AS a18, LOGSEQUENCEID AS a19, MACHINE AS a20, PARENTAUDITLOG
ID AS a21, PUBLICSESSION AS a22, PUBLISHDYNAMICATTRIBUTES AS a23, SYNCTRANSACTIONID AS a24, TIME AS a25, TRANSACTIONID A
S a26, CONFIGCHANGEAUDITLOG AS a27, DOMAINWORKSESSIONID AS a28, TRIGGERPHASE AS a29, WORKSESSIONID AS a30, BLADEID AS a3
1, RULEBASEID AS a32, FIELDNAME AS a33, PARENTRULEID AS a34, CHILDRULEBASEID AS a35, OBJECTNAME AS a36, SECONDARYMANAGEM
ENT AS a37, SICNAME AS a38, TYPE AS a39, TARGETOBJID AS a40, TARGETOBJIDS AS a41, CREATEDDOMAINID AS a42, FORCEUPDATE AS
a43, INSTRUCTIONS AS a44, GLOBALPOLICYCHANGE AS a45, GLOBALPOLICYID AS a46, LOCALPOLICYID AS a47, P

Any idea or suggestion, what might cause this behavior, or where to look to receive more information, logs, whatever ...?

I believe that I have been re-create behavior in lab environment (from backups), but high cpu there has disappeared after some time, leaving me still clueless.

Thank you.