topic Re: Management Server is stuck - user is unable to run any command, seen many times! in Firewall and Security Management

Management Server is stuck - user is unable to run any command, seen many times!

Thomas_Eichelbu — Tue, 11 Jun 2024 10:41:46 GMT

Hello team,

recently we stumbled over three issues on three totaly independet customer who run into this issue:

The MGMT server stopps to execute all kind of Check Point commands.
only "cpwd_admin list" worked and showed all processes as "E" not "T".

Even "reboot" or "init6" stop to work.
only a power cycle via VmWare or similar is possible to regain control.

if the MGMT is down, the Check Point CA is down, which is very unheathly for all VPN tunnels from the same MGMT based on certificates.
"invalid certificate" messages are then shown in the log.

cpm.elg says:
08/06/24 08:09:44,534 ERROR tracker.dataLog.TrackerDataSenderSvcImp [taskExecutor-31]: AuditLogsToTrackerSender: Unable to connect fwm (down), Exception: Could not receive Message.

and it throws a ton on java errors in cpm.elg.

we saw this on three different customers, all on R81.20 HFA 53/65

and yes there is that sk:
https://support.checkpoint.com/results/sk/sk173405

it did not work for me to kill "autoupdater" ...

anybody noticed the same?
we have two CP cases ongoing!

best regards

Re: Management Server is stuck - user is unable to run any command, seen many times!

Lesley — Tue, 11 Jun 2024 12:26:23 GMT

You could consider to disable the CRL check (less secure). It is a workaround during the time you figure it out with TAC.

At least it will maybe give you some rest if you are not able to power cycle the unit right away.

https://support.checkpoint.com/results/sk/sk21156

Re: Management Server is stuck - user is unable to run any command, seen many times!

Thomas_Eichelbu — Tue, 11 Jun 2024 12:46:01 GMT

Hello Lesley,

well yes i know that, but thats not the problem itself ...

the problem is, the MGMT becomes unusable, since no services run and no CP commands can be executed.
Thats the root issue that lead to the outcome of an unreachable CRL ...
And when the MGMT stops working you have no more chance to apply your SK sk21156 since, you cannot connect to MGMT database anymore to push policy 🙂

Re: Management Server is stuck - user is unable to run any command, seen many times!

Lesley — Tue, 11 Jun 2024 12:50:17 GMT

I understand that. And SK can be done before the problem occurs. Atleast the tunnels will stay up

Re: Management Server is stuck - user is unable to run any command, seen many times!

the_rock — Tue, 11 Jun 2024 14:11:27 GMT

Does guidbedit load or that does not work either? I assume rebooting the mgmt does not make a difference?

Re: Management Server is stuck - user is unable to run any command, seen many times!

Duane_Toler — Tue, 11 Jun 2024 15:08:33 GMT

Check your cp_mgmt SIC certificate. The "fwm" is down and the certificate errors you stated indicate you may have a problem there.

cpca_client lscert -kind SIC |grep cp_mgmt -A 2

Re: Management Server is stuck - user is unable to run any command, seen many times!

the_rock — Tue, 11 Jun 2024 15:11:33 GMT

Good command!

Re: Management Server is stuck - user is unable to run any command, seen many times!

Thomas_Eichelbu — Tue, 11 Jun 2024 15:27:42 GMT

well i see some expired certificates, but the majority of the certificates is still valid.
sorry i cannot post much of the output since it all contains personal data and so on ...

[Expert@ABCDEF:0]# cpca_client lscert -kind SIC |grep cp_mgmt -A 2
Subject = CN=cp_mgmt,O=ABCDEF.X.X.X.X.Y.Y.Y.4cn4gu
Status = Valid Kind = SIC Serial = 7622 DP = 0
Not_Before: Tue May 25 14:22:35 2021 Not_After: Mon May 25 14:22:35 2026

But if the SIC certificate is expired or revoked or anything but valid the MGMT would not stay completely down. And i still could run any CP commands on cli.
And if the SIC certificate is invalid a reboot would not help here ...

As i said the MGMT server is not running nor any services nor running any Check Point CLI commands works.
Iam pretty sure it will be stuck by tomorrow again ... we will see.

Re: Management Server is stuck - user is unable to run any command, seen many times!

Pauli — Tue, 11 Jun 2024 15:55:49 GMT

@Thomas_Eichelbu

Hi,
are there a lot of zombie processes running on the server? Our server currently has this error and the server is therefore unusable. A restart will temporarily fix the error

best regards

Re: Management Server is stuck - user is unable to run any command, seen many times!

Duane_Toler — Tue, 11 Jun 2024 15:56:42 GMT

Ok that's good to rule out. Have you done the handful of sanity checks as well? I would expect you have, but again just to rule them out:

check disk space
check OS logs to make sure nothing weird is there
Since it's a VM, make sure the hypervisor host is ok:
- datastore disk space
- datastore access to the SAN or local storage
- hypervisor RAM
Run CPM doctor ($FWDIR/scripts/run_cpmdoc.sh) when the host is functioning normally

If you are able to run OS commands, but not Check Point commands, then that does sound like an issue with the registry file like the SK indicated.

Try this, too: If the host is functioning now, do a controlled reboot just to see how it behaves. Since you have a pattern of the host misbehaving on an interval, see if this controlled reboot "buys" you more time for that interval before the next occurrence of the issue. Then look at the CPM debug topics and enable debug for Solr and webservices. These may give you an additional clue while you wait on TAC.

https://support.checkpoint.com/results/sk/sk115557

Likewise, you may also want to do a separate debug of CPD:
https://support.checkpoint.com/results/sk/sk86320

If it exhibits the issue again, a close examination of the CPM debug *should* point to the issue at the moment it occurs.

As a heads-up: TAC may give you the solr_cure process as part of the troubleshooting (sk140394, but it's a TAC internal SK).

Re: Management Server is stuck - user is unable to run any command, seen many times!

the_rock — Tue, 11 Jun 2024 16:05:49 GMT

What did TAC come back with?

Re: Management Server is stuck - user is unable to run any command, seen many times!

Thomas_Eichelbu — Tue, 11 Jun 2024 16:56:32 GMT

Hello, oh yes many Zombies ...

HCP from customer B

grep for Zombies on Customer A, here HCP doesnt run, it dies on licence check, a different story ... maybe?

endless rows of:
[Expert@ABCDEF:0]# ps aux | grep Z | more
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
admin 4653 0.0 0.0 0 0 ? Z 08:38 0:00 [cpd] <defunct>
admin 4654 0.0 0.0 0 0 ? Z 08:38 0:00 [cpd] <defunct>
admin 4655 0.0 0.0 0 0 ? Z 08:38 0:00 [cpd] <defunct>
admin 4657 0.0 0.0 0 0 ? Z 08:38 0:00 [cpd] <defunct>
admin 4658 0.0 0.0 0 0 ? Z 08:38 0:00 [cpd] <defunct>
admin 4659 0.0 0.0 0 0 ? Z 08:38 0:00 [cpd] <defunct>
admin 4661 0.0 0.0 0 0 ? Z 08:38 0:00 [cpd] <defunct>

customer C, the lucky guy!
no Zombies,

[Expert@HIJKLMNO:0]# ps aux | grep Z | more
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
admin 24765 0.0 0.0 2652 572 pts/1 S+ 18:44 0:00 grep --color=auto Z

https://support.checkpoint.com/results/sk/sk182370 is about Zombis on CPD ...

maybe an license issues, on customer A & B is see licensed issued for different IP´s on the SMS ...
(usage of aliases and so on )
Customer C has licensees issued only for its own real IP.

Re: Management Server is stuck - user is unable to run any command, seen many times!

Thomas_Eichelbu — Tue, 11 Jun 2024 16:28:36 GMT

nothing so far ...
iam still waiting.

Re: Management Server is stuck - user is unable to run any command, seen many times!

Thomas_Eichelbu — Tue, 11 Jun 2024 16:33:04 GMT

this are all good points.

CPM Doctor did not show any negative things, all green.
disk space is all good.
didnt run SOLR Cure yet
since iam not controlling the VMware infrastructure, i rely on third party to check this.

Re: Management Server is stuck - user is unable to run any command, seen many times!

the_rock — Tue, 11 Jun 2024 16:36:22 GMT

Ok, fair enough. As far as zombie processes, I know that usually fixed by doing cpstop; cpstart or reboot, but does not sound that would do much here. And since you said it does happen on R81.20 jumbo 65 as well, they cant really ask you to install any other jumbo hotfix. Now, to comment for cpm doctor, if that does not show any errors, tells me most likely database is clean.

Just wondering, how much ram is there on these servers, any idea?

Andy

Re: Management Server is stuck - user is unable to run any command, seen many times!

Duane_Toler — Tue, 11 Jun 2024 21:15:09 GMT

Incredibly curious, indeed. Have you been able to get the hotfix mentioned in that SK?

I happened to check one of my customers, and I also see them with numerous defunct CPD processes. Theirs is a CloudGuard management server, but I have many other customers with the same deployment (with same Azure template and VM size). I went through a bunch of logs and didn't find any smoking guns. I found some concerning logs, but other customers have the same, without issue.

I'm going to request that hotfix from TAC for my one customer, like yours. Looks like we have the same bug. 😔

Re: Management Server is stuck - user is unable to run any command, seen many times!

Thomas_Eichelbu — Wed, 12 Jun 2024 06:34:14 GMT

Well i had a long phone call with TAC to summarize all things ...
But he didn't said much about the zombies. Maybe they are not so horrible as they sound. At least he didn't paid much attention to it.

And i got no Hotfix for the CDP zombies as mention in SK sk182370.
Honestly i didn't requested one.

So i have opened two cases for two customer, they are ongoing. lets see what TAC will find out.

Re: Management Server is stuck - user is unable to run any command, seen many times!

the_rock — Wed, 12 Jun 2024 17:50:59 GMT

Hey, any new updates or nothing yet?

Andy

Re: Management Server is stuck - user is unable to run any command, seen many times!

Chris_Wilson — Mon, 17 Jun 2024 16:30:30 GMT

We have the same thing with 1 of our CMA's on our MDS. At least for us, a reboot of the MDS will fix things and we will be fine for another 6-10 days until we have the CPD problem. Working with TAC, the one engineer said that T150 was supposed to contain a fix for CPD with defunct status. I assume this might be the same hotfix that sk182370 mentions. We are currently running T141.

The first time we had this problem was May 30, before we installed T141. T141 was recommended at the time the hotfixes for the vpn problem came out, so we went with that

With CPD in the defunct status, we had the SIC issues too and we started to have the vpn tunnels start failing as I would say it would have been over the 24 hours that the remote firewall had checked in with the mgmt server to check SIC.

Re: Management Server is stuck - user is unable to run any command, seen many times!

Duane_Toler — Tue, 18 Jun 2024 02:24:12 GMT

My customer with this issue hasn't had the CPD defunct process situation escalate to total server outage yet. I have Nagios monitoring the system process counts frequently, so I am able to get to it and restart CPD with the "cpwd_admin" commands in a controlled state.

If you're desperate, make yourself a cron job to do it, too.

EDIT: I made a real script today that will do everything we need (MDS top-level, MDS per-domain, SMS, EPM, SME) and posted it in the ToolBox:

https://community.checkpoint.com/t5/Scripts/Restart-CPD-script/m-p/217862/highlight/true#M1159

[Expert@cpmgmt01:0]# ./cpd_restart.sh -h cpd_restart.sh: Restart CPD process on Multi-Domain server and Security/Endpoint management Usage: ./cpd_restart.sh [ -d [ ALL | <specific domain server> ] | [ -h ] Options: d Specify a single domain management server (CMA) or special word ALL for all domain servers listed in "mdsstat" output (Optional; only relevant for MDS) h This help If no argument is given, then the top level CPD process is restarted (for the MDS itself, Security Management server, or Endpoint Management server)

Run it with a "-d ..." to restart CPD on a given domain server if that's your troublesome one, or "-d ALL" to restart CPD on all domain servers. This only restarts CPD and leaves the other processes alone, so there's no outage. It uses the same methods that Check Point's own scripts use (shameless stole the commands out of $MDSDIR/scripts/cpshared). This ensures CPD restart is done the correct way and gets re-attached to CPWD for monitoring.

If you just have a single Security Management server, then don't give any arguments and it'll just restart the one process, or the MDS root CPD process.

Put that script in /home/admin, chmod 755, then set a job in CLISH:

> add cron job CPD_Restart command "/home/admin/cpd_restart.sh" recurrence hourly hours all at 00 > show cron job CPD_Restart recurrence Every day at every hour at the 00 minutes.

or for MDS:

> add cron job CPD_Restart command "/home/admin/cpd_restart.sh" recurrence hourly hours all at 00 > add cron job CPD_Restart_domains command "/home/admin/cpd_restart.sh -d ALL" recurrence hourly hours all at 05 > show cron job CPD_Restart recurrence Every day at every hour at the 00 minutes. > show cron job CPD_Restart_domains recurrence Every day at every hour at the 05 minutes.