SMS lost its marbles =- SmartDashboard component failed to connect to server x.x.x.x

[supruzer1]

Hello Gurus.

Our production SMS (R80.40) lost communication to all the gateways including Eventia server. Now we cant open gateways in Smartconsole, push policy and logs not populating from itself and Eventia server. 

When logging into Smartconsole the error "SmartDashboard component failed to connect to server x.x.x.x (IP of the SMS)

Happened without reason - but noticed it when we shutdown a couple of pair of old gateways. Note the issue isn't because of routing or traffic.

1. API Status - API readiness test SUCCESSFUL. The server is up and ready to receive connections

2. cpwd_admin list - all services up and running

Have restarted service, have rebooted the manager, communication between it and the devices it manages are seen in the logs and no communication breaks. 

TAC says to upgrade the Server but I don't think its the reason. They don't know the cause because I think its rare. We will upgrade the server and continue tbshooting with TAC. However if you fine guys have come across such a thing in your many years of expertise please let me know. 

Cheers!

S

[the_rock]

In my 15+ years dealing with CP, I had only seen that error once and reboot fixed it. Now, here is the thing. Based on your excellent troubleshooting and all you had done so far, I would be shocked if upgrading fixes it, but I been wrong before (many times actually lol), so lets hope that does work.

By the way, just to sort of break this down logically, any changes made recently? When was last time this worked? Have you maybe tried different admin to see if any luck? Different PC?

Only thing I found about it is below:

https://support.checkpoint.com/results/sk/sk165272

Andy

[the_rock]

You know what...I know this may sound sort of silly, but I know one of my colleagues had similar issue like this recently and after some testing, we realized it was CPMI that was the issue, specifically this line, in $FWDIR/lib/implied_rules.def file on mgmt server, so maybe worth checking. Make sure those lines are EXACTLY how they are in the screenshot I attached.

Andy

 

[supruzer1]

Thank you Andy for quick answer. 
1. For https://support.checkpoint.com/results/sk/sk165272. I think once the upgrade to latest hotfix is done we will know if we can go further. Otherwise I think we will have to rebuild from our old snapshot/backups and add firewall policy changes that were done after. 

2. The file matches exactly like yours so doesn't seem like something happened to it in my case.

cat implied_rules.def
/*
* (c) Copyright 1993-2008 Check Point Software Technologies Ltd.
* All rights reserved.
*
* This is proprietary information of Check Point Software Technologies
* Ltd., which is provided for informational purposes only and for use
* solely in conjunction with the authorized use of Check Point Software
* Technologies Ltd. products. The viewing and use of this information is
* subject, to the extent appropriate, to the terms and conditions of the
* license agreement that authorizes the use of the relevant product.
*
* $RCSfile: implied_rules.def,v $ $Revision: 1.2.4.5.2.1.4.11.26.1 $ $Date: 200 5/07/12 14:06:28 $
*/

#ifndef __implied_rules_def__
#define __implied_rules_def__

#include "office_dhcp.def"

#define ENABLE_FWD_TOPO
#define ENABLE_FWD_SVC
#define ENABLE_CPMI

We are upgrading tomorrow. 

Cheers! Many thanks!

[the_rock]

Please let us know how it goes. I truly hope that fixes the issue.

Andy

[supruzer1 · Accepted solution]

Hey Team,

Reboot, restarting services, upgrading to latest R80.40 HFA didn't work. Resolved by following sk123597 and then to renew Certificate sk20905

- Not able to open any GW object from Smartconsole and All GW's are showing down status.
- "Please wait while loading..." when trying to open or editany Security Gateway objects
- Error "SmartDashboard component failed to connect to server x.x.x.x (IP of the SMS)
- No logs on and after the date/time of issue, only previous logs seen
- Error - Certificate Revoked if you updated your Smartconsole Client.
- tail -n 20 $RTDIR/log/solr.log shows certificate error: ERROR [qtp-1879661199-14909] com.checkpoint.java_sic.remote.SicRemoteTrustManager.checkClientTrusted:25 - Failed to validate Client certificate [1]

Cheers!

S

[the_rock]

Awesome job!!