SMS server showing error and can't see policy

[Gombodorj]

Hello,

I have Smart-1 S600 SMS running R81.10 with take 79. Everything was working perfectly last week. Today I just tried to click on policy on Smart Console, it says "Could not load the selected policy". Also SmartView seems to be not working because I cant monitor the license informations. SMS gaia portal is also not working, not connecting from web browser. 

Also, it shows in the logs that bunch of Monitored processes restarting frequently or going down on SMS origin. (Smartview,cpsead etc...)

However, I can SSH into SMS and CPU,RAM,STORAGE usage seems to be perfectly fine. Storage at 50%.

What is the issue?

[G_W_Albrecht]

Tried a reboot yet ?

[the_rock]

I second what @G_W_Albrecht told you, reboot seems best in this case. But, BEFORE you do that, if you can wait a little bit, can you please send output of below commands?

evconfig

top

ps -auxw

free -m

api status

Andy

[G_W_Albrecht]

# Andy

did not work for me 😉

[the_rock]

You lost me there @G_W_Albrecht lol. What did not work?

[G_W_Albrecht]

The last command:

# Andy
-bash: Andy: command not found

 😂

[the_rock]

HAHAHAHAHA...I dont know man, worked fine for me!! Its layer 8 problem on your end brother...

[Gombodorj]

Hi, I ran those commands. 

[Tal_Paz-Fridman]

Hi - this seems to be a problem affecting many important processes. 

As G_W_Albrecht suggested, if possible, try rebooting the machine. A few minutes after it comes up, run the following command to see that all the processes are running properly:
cpwd_admin list

If all the processes are up, try repeating the actions you listed and see if everything is working properly.

If not, I suggest opened a ticket with TAC but we could also continue investigating it directly.

Thanks

[Chris_Atkinson]

What troubleshooting has been attempted so far?

cpstop;cpstart etc ...

[Gombodorj]

I tried rebooting, after that I ran cpwd_admin list and everything seems to be in executing state. Problem still exactly the same. Also tried cpstop, cpstart as well.

[G_W_Albrecht]

Restore last update and redo recent changes ?

[Gombodorj]

I do have snapshot and thinking of reverting it, but it's fairly old one so it will be real hassle to do vsx configs again

[G_W_Albrecht]

Is TAC involved already ? I would strongly suggest that with VSX managed...

[the_rock]

K, can you confirm the following for us please? What is currently the status if you run api status command from expert mode? Also, can you send output of cpwd_admin list?

I actually had a weird issue couple of years ago with customer on R80.30 I believe and their mgmt server one day just "decided" to stop working and no matter what we did, we could never make smart console come up at all, even after working with TAC for few days. Finally, we decided not to spend more time on it and thank God they had working backup and after restore, we just loaded latest jumbo and all worked fine afterwards. I never recall seeing anything about policy not loading, first time I ever seen that was in my R81.20 lab, but it was standalone, which I could not make work after 5 tries.

[Gombodorj]

Here are the results. SMS just decided to not work as you said. Can't click the Security Policies tab so I can't make new policies or even see them.

[the_rock]

Ok, so we can see that cpm and fwm processes are up and running, so thats good, but whats NOT good is that api status shows failed. Can you also run this

cd $FWDIR/scripts

./cpm_status.sh

Send the output of that script please. At this point, I wont waste your time and ask you to try another PC, as I know 100% it wont work, as long as api status shows failed, that has to show successful.

[Gombodorj]

[Expert@mn-dc1-r1c1-sec-fw.sms1:0]# cd $FWDIR/scripts
[Expert@mn-dc1-r1c1-sec-fw.sms1:0]# ./cpm_status.sh
Check Point Security Management Server is running and ready
[Expert@mn-dc1-r1c1-sec-fw.sms1:0]#

this is the output.

Seems like apache in API is not running but I dont know much about api status troubleshooting

[the_rock]

As @G_W_Albrecht said, that looks good. If I were you, I would pick up the phone and call TAC and work on this right away. There is something seriously wrong here, what it is, Im not sure myself. For any movement here, you need to see api status as successful, so one thing you could try is maybe api restart, see if it does anything.

[G_W_Albrecht]

Looks fine. TAC needed.