topic Re: State: Connection with 'fw-vsx-n01' is lost in General Topics

State: Connection with 'fw-vsx-n01' is lost

Teddy_Brewski — Mon, 27 Jan 2025 11:19:53 GMT

Hello,

One of the VSs (out of 4), started reporting today:

State: Connection with 'fw-vsx-n01' is lost

There are no issues reported from the SSH session -- this node is active and handles the load.

cphaprob state, cphaprob -a if, cphaprob -ia list, revealed nothing wrong.

Tried to reboot the management but it didn't help.

Management: R81.20 Take 65

CP VSX: R81.20 Take 90

Thank you.

Re: State: Connection with 'fw-vsx-n01' is lost

Chris_Atkinson — Mon, 27 Jan 2025 12:04:11 GMT

What does 'vsx stat -v' show?

Re: State: Connection with 'fw-vsx-n01' is lost

Teddy_Brewski — Mon, 27 Jan 2025 12:21:51 GMT

Looks healthy too.

The one that is complaining about one of the nodes being down is VS5 (fw-vs-cloud):

VSX Gateway Status
==================
Name: fw-vsx-ext-n01
Access Control Policy: fw-vsx-external-vsx
Installed at: 14Jan2025 15:27:58
Threat Prevention Policy: <No Policy>
SIC Status: Trust

Number of Virtual Systems allowed by license: 6
Virtual Systems [active / configured]: 3 / 3
Virtual Routers and Switches [active / configured]: 2 / 2
Total connections [current / limit]: 32393 / 96500

Virtual Devices Status
======================

Type: S - Virtual System, B - Virtual System in Bridge mode,
R - Virtual Router, W - Virtual Switch.

Re: State: Connection with 'fw-vsx-n01' is lost

AkosBakos — Mon, 27 Jan 2025 12:38:43 GMT

Hi @Teddy_Brewski

Does it happen periodically

Maybe the cpd process crashes.

Have a look at on this: https://support.checkpoint.com/results/sk/sk101484

What does cpwd.elg say?

Akos

Re: State: Connection with 'fw-vsx-n01' is lost

Teddy_Brewski — Mon, 27 Jan 2025 13:18:07 GMT

Hi @AkosBakos

No, this is the first time it happened.

I checked cpwd.elg on the affected node, and although I see 'did not send keep-alive message for 1 number of times' error messages, none of them are related to CPD, but rather MSGD:

[cpWatchDog 19785 4133372096]@fw-vsx-n01[27 Jan 11:47:49] [ERROR] MSGD (pid=30558) did not send keep-alive message for 1 number of times
[cpWatchDog 19785 4133372096]@fw-vsx-n01[27 Jan 11:49:34] [ERROR] MSGD (pid=30661) did not send keep-alive message for 1 number of times

Re: State: Connection with 'fw-vsx-n01' is lost

AkosBakos — Mon, 27 Jan 2025 13:18:00 GMT

Hi @Teddy_Brewski

The https://support.checkpoint.com/results/sk/sk101484 said:

In some scenarios cpwd.elg shows repeatedly:

[ERROR] CPD (pid_of_cpd) did not send keep-alive message for x number of times

Re: State: Connection with 'fw-vsx-n01' is lost

Teddy_Brewski — Mon, 27 Jan 2025 13:44:23 GMT

Hi @AkosBakos

Yes, but SK mentions that it's CPD daemon that is reporting the error. In my case it's MSGD -- do you think it's the same?

There are no traces of core dumps in the logs and no high CPU observed. SIC is also fine. I haven't tried to push the policy on the affected VS though.

Re: State: Connection with 'fw-vsx-n01' is lost

AkosBakos — Mon, 27 Jan 2025 15:41:11 GMT

Hi @Teddy_Brewski

Because the lack of information, I can't say this is the same or not, but there are symptomes which are the same.

In this case, the best thing what you can do to ask the TAC about this issue.

Akos

Re: State: Connection with 'fw-vsx-n01' is lost

Chris_Atkinson — Mon, 27 Jan 2025 23:42:31 GMT

Concur, anything of note flagged in HCP ?

Otherwise some suggestions:

- Attempt policy install

- Restart cpd per sk97638

- Failover / Reboot gateways

- Patch with latest recommended JHF

- Open a TAC case (attach HCP & CPinfo)

Re: State: Connection with 'fw-vsx-n01' is lost

Teddy_Brewski — Mon, 03 Feb 2025 14:50:23 GMT

Thank you for the hint @AkosBakos !

With the help of our CP partner the issue has been identified. CPD crashed and was failing to restart since then. No CPEPS database corruption has been observed, so killing the stale process and stopping/starting CPD manually in the context of affected VS fixed the issue.