Connection with 'xxxxx' is lost

[ED]

Hi,

This is the situation:

When i hover over the x sign it says 'Connection with xxxx is lost'. I can do the following:

-open up all the gateway object properties

-install policy

-SIC is communicating on the GW's objects

-SIC on SMS is greyed out

What i did was to poweroff the SMS to take a snapshot with VMware. After bootup I got these red x signs. Any ideas what could have happened?

[Nick_Doropoulos]

Hi Ed,

I do occasionally notice similar strange 'behaviour' on SmartConsole myself. Sometimes it takes a while before SmartConsole updates the information correctly. 

Below are some of the things I would do in an attempt to cause SmartConsole to display the correct status of the gateways:

1) With a gateway/cluster selected, click on the Monitor option at the top and refresh the page a few times.

2) Toggle the "status" column off and toggle it back on.

3) Initiate some traffic from the gateways to the manager in the form of pings or fetching policy.

4) Reset SIC.

I hope this helps.

[ED]

Hi @Nick_Doropoulos 

I have tried what you suggested except resetting SIC. Still the same problem. Before I try resetting SIC to the security gateways, shouldn't the status of SMS be marked as green? I find it strange that everything seems to work fine except the status column in the SmartConsole. There is not so much help to find on usercenter either. 

[Nick_Doropoulos]

Hi Ed, 

Another thing worth trying is to use another version of SmartConsole. That has fixed similar issues for me in the past. 

Give that a go too if you can and let us know of the result. 

[ED]

@Nick_Doropoulos 

I am trying with these two different versions of SmartConsole:

Should I try to uninstall one of these on the management server?

[Nick_Doropoulos]

Hi Ed,

Build 122 is the latest one I believe so we can rule this out as well.

I would do the following as well if I were you:

1) Double-click on of the gateways, navigate to Network Management and select the "get interfaces without topology" option. This option will get all interfaces without changing your existing topology. This might just be what you need for the SMS to 'see' there are no issues with its connection to the gateway.

2) In expert mode, try cpstop ; cpstart in case there is a daemon responsible for this false status and needs restarting. Providing you use PSK-based VPNs and not certificate-based ones, there shouldn't be any service disruption to your environment.

3) If neither of the above works, try a failover of one of your clusters to see if that does it.

Once again, resetting SIC is another troubleshooting step worth doing I believe.

Let us know of the results.

[Timothy_Hall]

Try clearing the monitoring database as specified in this SK:

sk112058: Gateways & Servers view in R80 SmartConsole does not show statuses

This procedure seems to be the R80+ equivalent of clearing the SmartConsole cache files CPMILinksMgr.db* and applications.C* for the R7* SmartView Monitor when it displays incorrect status information about gateways as documented here:

sk100507: SmartConsole problems with Security Management Server / Multi-Domain Security Management S...

[ED]

@Timothy_Hall 

I tried to run the script but the status is still red x and the hit counts wasn't cleared either. 

[ED]

Can someone from CheckPoint tell me what check is being processed in the background to determine the Status of an object in SmartConsole? 

[Timothy_Hall]

On the gateway, the cpd daemon is the one responding to the status query on TCP port 18192 (CPD_amon). This traffic is allowed by an implied rule on the gateway so it shouldn't be blocked.   If the cpd daemon dies or is impaired on a gateway it won't report a status at all; the log file for this daemon is located in $CPDIR/log/cpd.elg.   In the R77.30 and earlier SmartView Monitor I'm pretty sure the SmartConsole GUI system would initiate the CPD_amon connection directly to the gateway itself to pull status.  Not sure if this is still the case in the R80+ SmartConsole, the SMS may well be the one maintaining this connection, I would assume via the corresponding cpd daemon on the SMS although it could be the cpstat_monitor (CPSM) process, not sure.  Might be worth checking out log file $FWDIR/log/cpstat_monitor.elg as well.

[ED]

Hi @Timothy_Hall 

I ran the command: netstat -an | grep 18192 on the SG.

This shows that the connection between security gateway and SMS is established. 

How can it then say that the connection is lost in SmartConsole?

[Timothy_Hall]

OK so that confirms that the SMS is maintaining that monitoring connection but the status simply isn't getting reported to your SmartConsole GUI for some reason.  Try this:

1) Anything interesting in $FWDIR/log/cpstat_monitor.elg?

2) Open the SmartView Monitor.  To do this from SmartConsole: Logs & Monitor tab...New tab (+)...Tunnel & User Monitoring (lower left corner).  Does the Smartview Monitor report the same status for those gateways as the SmartConsole?

3) As a last resort, kill the cpstat_monitor daemon and let it restart.

[ED]

@Timothy_Hall 

The cpstat_monitor.elg file is empty. SmartView Monitor shows status as disconnected.

I ran these commands:

cpwd_admin stop -name CPSM

cpwd_admin start -name CPSM -path "$FWDIR/bin/cpstat_monitor" -command "cpstat_monitor"

No change. 

[Timothy_Hall]

Sounds like it is time for some debugging of the problem with TAC.  Can take a shot yourself if you want, these are probably the daemons you should look at:

sk108177: How to debug the "cpstat_monitor" daemon

sk86320: How to debug the CPD daemon

[Sergei_Karpovit]

I have similar monitoring issue but only with one Gateway, which is using bond interface (VPC config).

I have used step 2 and in Smartview Monitor it was reporting the same status for gateway as the SmartConsole - DOWN. When i click refresh it went green, then become green on SmartConsole as well, but not for long? From gateway CLI i use netstat command to confirm Management is talking to GW on correct port and connectivity wise there are no issue. I'm wondering if the issue could be due to the asymmetric path as we use bond interface (VPC) with 2 sub-interfaces, can someone advice how to test to prove if that's the issue?

[PhoneBoy]

Don't think that sort of asymmetry would cause an issues since, logically, they're coming from the same interface. (Where the bond is the logical interface).

[Timothy_Hall]

Run cpwd_admin list and make sure cpd is stable and not being restarted by cpwd.  Whenever an alarm is displayed in the SmartConsole and SmartView Monitor, it will "stick" for a period of time even if everything is OK again in order to ensure that you noticed there was a problem at some point.  Anything interesting in $CPDIR/log/cpd.elg?

[daemonkenji]

Hello Timothy,

I'm having same issue. Looking at $CPDIR/log/cpd.elg, we got below

[CPD 13456 4133075328]@UXFRLB-CKPSMS1P[9 Dec 20:45:38] SIC Error for amon: received bad message length from peer
[CPD 13456 4133075328]@UXFRLB-CKPSMS1P[9 Dec 20:45:43] SIC Error for amon: received bad message length from peer
[CPD 13456 4133075328]@UXFRLB-CKPSMS1P[9 Dec 20:45:48] SIC Error for amon: received bad message length from peer
[CPD 13456 4133075328]@UXFRLB-CKPSMS1P[9 Dec 20:45:54] SIC Error for amon: received bad message length from peer
[CPD 13456 4133075328]@UXFRLB-CKPSMS1P[9 Dec 20:45:59] SIC Error for amon: received bad message length from peer

Do you have any advice please?

Regards

[Ryan_Ryan]

Had the same issue, following sk113744 resolved it immediately with no impact or downtime.

[daemonkenji]

Hello Ryan,

I'm having same issue. Could you confirm that you did follow sk20905 for resolving the issue?

Actually, the certificate on our SMS showing no Revoked or Expired, it's still Valid for all the managed CKP gateway. How about your case?

Thank you in advance.

Regards

[Ryan_Ryan]

yes followed sk113744 as part of the process, ours werent expired either (from memory)

[starmen2000]

@ED  have you solved that problem? I have same issue at the moment.  81912 port is established with the SMS. but still monitors that gateway connection is lost. 

[Tal_Paz-Fridman]

As mentioned above, have you tried sk112058 - "Gateways & Servers view in R80.x SmartConsole does not show statuses of servers":

https://support.checkpoint.com/results/sk/sk112058

Also - you have SIC established with the Security Gateway? Install Policy works properly?