FWM dies quietly on CMA R0.20

[Kaspars_Zibarts]

Just wondering if anyone else has noticed issues with FWM on CMA - shows as UP on mdsstat but actually is not responding. Then you do mdsstop_customer and that particular FWM still shows in UP state. Kill manually and start CMA, then all starts working again. I simply haven't had time to run any debugs yet but would be interesting to know if we are alone with this

[Maarten_Sjouw]

I have a case open for this issue on my 2 MDS setups in R80.30
The only way to get the CMA going again is by killing the process and issuing a mdsstart_customer for the specific CMA.

[Kaspars_Zibarts]

Ah! So we are not alone then! 🙂 Makes me lazy.. maybe I should sit and wait till your case is resolved 🙂 What take you are on BTW?

[Maarten_Sjouw]

Please also open a case and you can refer to mine, ask them to assign the case to Asaf and mention my name.
My take on the R80.30 is JHF 111

[Maarten_Sjouw]

This week we received a fix for our version and I installed it on our 4 MDS servers.
So far I did not see any FWM lockups yet.
Found out also earlier this week that when you run a mdsbackup script, the backup script will report those locked up FWM CMA's as they cannot be backed up.
So be warned about that!

[Francesco-P]

Hi all,

Is this issue occasional or it's replicable?

I ask this because, I opened a case for a similar strangness on our SMS R80.20 T118 with HA configured, where debug is not usefull at all, also because it was not replicable.

I noted this for the first time on our primary manager: i was not able to loggin with smartconsole, but the cpm/fwm was up as reported on cpwd_admin list.
After a cpstop/cpstart all works fine...and the issue has never occurred again.

But after a few, i noticed the same behaviour on the secondary manager where the fwm process was report as up on watchdog, with a 100% of core utilization for a long time.

In this case i stopped the fwm process and started it again to "solve".

[Maarten_Sjouw]

It just happens from time to time on a CMA, there is nothing we can see that is anything like a regularity.

[the_rock]

Hey guys,

I am also having similar issue on R80.30 Provider-1. What I tried doing was actually importing separate migrate export from R80.30 regular mgmt server and to my surprise, seems like there is no way to actually NOT start new CMA from dashboard (like you could back in R77,xx versions), so I had to actually follow below link to create new cma from command line without starting it

https://community.checkpoint.com/t5/Multi-Domain-Management/Create-a-Domain-without-startup/td-p/191...

Now, that all works great and I can do cma_migrate to import the config, BUT, once thats successful, I can NEVER start the cma itself, fwm always shows down and rebooting it, doing mdsstop actually seems to make it worse and though mdsstart_customer cmanname does say fwm is started, mdsstat still shows it as down.

If anyone has any suggestions, I would greatly appreciate it.

Andy

[Maarten_Sjouw]

These 2 items are not related, but certainly involve TAC here.
May I also ask how long you have been waiting to see FWM started in mdsstat? The first time it can take up to an hour depending on how big you database is.

Regarding the creation of a CMA without starting, you are correct only through the API you can create and not start a new domain.

[the_rock]

Thanks Maarten for your reply, appreciated. Well, first time, I waited maybe 10-15 minutes, so this time, will wait much longer. Actually, I used to work for Check Point for a while, but this was before R80 days, so Provider-1 has changed for sure :). Let me do this...I actually deleted the old Provider-1, since this is in the lab, so just finished installing a brand new one and will reboot after jumbo 111 install and then simply add new cma through cli and wait for an hour to see if it starts after importing the mgmt backup.

Thanks again. If that fails, I will reach out to my good friend who works for escalation team at CP and see if he might be able to help me out.

Have a nice weekend!

Andy

[Maarten_Sjouw]

Remember that when you have the export of an SMS, this will NOT work, for that you need to install R80.40 as soon as it comes out. Or join the EA.

[the_rock]

Actually, I am not doing this for myself or a customer, just testing to see what result I get, since my other friend I used to work with in the past asked me about it the other day, so I wanted to test it to see if I get the same issue. Appears that Check Point professional services has some type of internal script that fixes this problem, but its not available publicly. I did this many times back in R77.x versions and never had a problem. I cant actually believe that they took away the option to not start cma via the dashboard or import it directly via dashboard too...that was so convenient. I really do hope they bring that back in future versions.

Thanks again and have a great weekend!

Andy

[the_rock]

Marteen, turns out that problem why cma does not start is the actual license, since after the import of R80.30 mgmt server, it brings over license with different IP and even when you do mdsenv cma_ip and then run fwm, gives license violation error. Once I can get eval license and start cma, I will check what happens and update.

Cheers,

Andy

[Maarten_Sjouw]

I did see similar issues but only on the MDS itself related to the license, fwm did not start on the MDS and indeed that was due to an expired license but I did see something about this being fixed in a jumbo.

[Kaspars_Zibarts]

I have some update from R&D that we got during our CPX user group meeting. Even though actual root cause is still unknown, CP seems to know how to fix it. It is suppose to come out some time soon

[the_rock]

I saw similar issue on cma R80.30 and license was indeed a problem...I really wish CP would fix license stuff, its so annoying. In all my years working there, it was exactly same stuff every day...customers would cry foul about licences and everyone knew it was terrible, but they chose to do absolutely nothing about it. Pretty sad, in my opinion.

[Maarten_Sjouw]

2 weeks ago I received a fix for the dying FWM on CMA's and MDS's.
It has been running since and we had no issues with them anymore.

[Kaspars_Zibarts]

Was it custom fix or available in ongoing take?

[Maarten_Sjouw]

Was custom, to test and see if it would help, when all ok it would be moved to general fixes...