SMS server showing error and can't see policy

[Gombodorj]

Hello,

I have Smart-1 S600 SMS running R81.10 with take 79. Everything was working perfectly last week. Today I just tried to click on policy on Smart Console, it says "Could not load the selected policy". Also SmartView seems to be not working because I cant monitor the license informations. SMS gaia portal is also not working, not connecting from web browser. 

Also, it shows in the logs that bunch of Monitored processes restarting frequently or going down on SMS origin. (Smartview,cpsead etc...)

However, I can SSH into SMS and CPU,RAM,STORAGE usage seems to be perfectly fine. Storage at 50%.

What is the issue?

[G_W_Albrecht]

Tried a reboot yet ?

[the_rock]

I second what @G_W_Albrecht told you, reboot seems best in this case. But, BEFORE you do that, if you can wait a little bit, can you please send output of below commands?

evconfig

top

ps -auxw

free -m

api status

Andy

[G_W_Albrecht]

# Andy

did not work for me 😉

[the_rock]

You lost me there @G_W_Albrecht lol. What did not work?

[G_W_Albrecht]

The last command:

# Andy
-bash: Andy: command not found

 😂

[the_rock]

HAHAHAHAHA...I dont know man, worked fine for me!! Its layer 8 problem on your end brother...

[Gombodorj]

Hi, I ran those commands. 

[Tal_Paz-Fridman]

Hi - this seems to be a problem affecting many important processes. 

As G_W_Albrecht suggested, if possible, try rebooting the machine. A few minutes after it comes up, run the following command to see that all the processes are running properly:
cpwd_admin list

If all the processes are up, try repeating the actions you listed and see if everything is working properly.

If not, I suggest opened a ticket with TAC but we could also continue investigating it directly.

Thanks

[Chris_Atkinson]

What troubleshooting has been attempted so far?

cpstop;cpstart etc ...

[Gombodorj]

I tried rebooting, after that I ran cpwd_admin list and everything seems to be in executing state. Problem still exactly the same. Also tried cpstop, cpstart as well.

[G_W_Albrecht]

Restore last update and redo recent changes ?

[Gombodorj]

I do have snapshot and thinking of reverting it, but it's fairly old one so it will be real hassle to do vsx configs again

[G_W_Albrecht]

Is TAC involved already ? I would strongly suggest that with VSX managed...

[the_rock]

K, can you confirm the following for us please? What is currently the status if you run api status command from expert mode? Also, can you send output of cpwd_admin list?

I actually had a weird issue couple of years ago with customer on R80.30 I believe and their mgmt server one day just "decided" to stop working and no matter what we did, we could never make smart console come up at all, even after working with TAC for few days. Finally, we decided not to spend more time on it and thank God they had working backup and after restore, we just loaded latest jumbo and all worked fine afterwards. I never recall seeing anything about policy not loading, first time I ever seen that was in my R81.20 lab, but it was standalone, which I could not make work after 5 tries.

[Gombodorj]

Here are the results. SMS just decided to not work as you said. Can't click the Security Policies tab so I can't make new policies or even see them.

[the_rock]

Ok, so we can see that cpm and fwm processes are up and running, so thats good, but whats NOT good is that api status shows failed. Can you also run this

cd $FWDIR/scripts

./cpm_status.sh

Send the output of that script please. At this point, I wont waste your time and ask you to try another PC, as I know 100% it wont work, as long as api status shows failed, that has to show successful.

[Gombodorj]

[Expert@mn-dc1-r1c1-sec-fw.sms1:0]# cd $FWDIR/scripts
[Expert@mn-dc1-r1c1-sec-fw.sms1:0]# ./cpm_status.sh
Check Point Security Management Server is running and ready
[Expert@mn-dc1-r1c1-sec-fw.sms1:0]#

this is the output.

Seems like apache in API is not running but I dont know much about api status troubleshooting

[the_rock]

As @G_W_Albrecht said, that looks good. If I were you, I would pick up the phone and call TAC and work on this right away. There is something seriously wrong here, what it is, Im not sure myself. For any movement here, you need to see api status as successful, so one thing you could try is maybe api restart, see if it does anything.

[G_W_Albrecht]

Looks fine. TAC needed.

[Shira]

Hi Gombo,

What was the solution? can you please share it here, i am having exact issue.

WR,

Shira

[biskit · Accepted solution]

I had the problem today too.  I was in, working with no problems, then suddenly the rules page went blank, and I got an error.

Symptoms:

• SmartConsole rulebase would not load - I get "Could not load the selected policy" error.
• Logs would not open.
• Gaia WebUI refused connection
• #API status gave:   API readiness test FAILED. The server is down and unable to receive connections!
• Reboot did not fix it

I found sk180382  No access to Gaia Portal on the Security Management Server (checkpoint.com).

Most of the symptoms matched (HTTPD seemed to be screwed and wouldn't reload).  Except I did not get the "sic_cert.pem" error.

TAC directed me to sk179589  "Could not load selected policy" in SmartConsole (checkpoint.com)

I found that my /web/templates/httpd-ssl.conf.templ file was completely empty!

There was a .bac version, so I copied this back over the top of /web/templates/httpd-ssl.conf.templ.

HTTPD was still screwed (no reboot yet)

I rebooted....

Bingo - everything works perfectly again after the reboot.

So, try checking you /web/templates/httpd-ssl.conf.templ file and see if it's empty?  That could be the problem.  Restore from the backup file (or from another working machine) and reboot.  Hopefully that'll fix the issue.

[the_rock]

Wow, thats quite an issue you had...thanks for sharing, it can definitely help others if they encounter the same situation.

Andy

[Shira]

Hi,

In our case, issue got addressed after performing sk180829.

WR,

Shira

[the_rock]

I remember customer doing this before Shira. No idea how it happened in the first place, but thank God it only happened once...

[biskit]

Ah good.  That's kinda the same thing I did, but under a different SK.  Glad it's all working again now!

[Fire_Verse]

I can confirm that this fix works on R81.20 Jumbo 41 for an Open Server SMS.

Had the exact same error message:"Could not load the selected policy"

Copied the /web/templates/httpd-ssl.conf.templ over the .bak file. Rebooted. Policy was visible again. Fixed.

Before applying the fix I noticed the following:

• Install database through SmartConsole would complete 
• I could install the policy through SmartConsole despite not being able to view it
• I tried cloning the existing policy to see if that would clear things up: still could not see the policy in the cloned version
• WebUI was unresponsive
• 'https://*SMS IP address*/smartconsole' was unavailable
• 'HealthCheck Point' would not work via SmartConsole

We had a power outage and the SMS was not shut down gracefully. When the SMS was manually started back up, the "Could not load the selected policy" issue appeared in the SmartConsole.