FWM in pending on mds after install JHF

[Maarten_Sjouw]

Hey guys and galls,

I have a problem with an R81 MDS where I installed the latest jumbo and after the reboot all FWM processes remain in pending state, at least for half an hour already. All SK's are only mentioning older versions.

Anyone seen this? 

[Naama_Specktor]

Hi 🙂

My name is naama specktor and I am from check point.

I will aparecaite it if you will share the TAC SR# with me , so we can get more details about this case.

thanks in advanced,

Naama 

[Maarten_Sjouw]

I got word from my regular contact in Tel Aviv, who was able to tell me this is normal behaviour when you do an upgrade from take 36 to anything above 42, the issue is there is a kind of a rebuild and cleanup of the database, which takes many hours, in our case we are already past the 10 hours mark and we don't expect it to finish in the next couple of hours.

You can reference sk176304 for the issue.

Output from the api status command shows this line:

CPM Starting 18469 Check Point Security Management Server is during initialization

mdsstat shows the same line on top.

[Ran_Kopelman]

Hi Maarten,

My name is Ran and I’m a manager in Check Point RnD, responsible for the quality of the Management product.

I would like to emphasize that this is definitely NOT the expected behavior following any Jumbo installation.

There is indeed an issue that is specific to your database which caused this problem.  

As you know, we have fixed your environment manually and we are working to provide you also a code-based fix.

My team will update you via your SR

Thanks,
Ran

[genisis__]

Is this issue present in R81.10 with the latest JHFA?

[Maarten_Sjouw]

I don't think R81.10 has issues with this, the problem here is very specific to R81 with JHF 36 going to a higher version.

[Maarten_Sjouw]

Sorry, Ran, 

I cannot quite agree to what you're saying, in november we did the exact same thing with a other single MDS then from 36 to 44 and there we ran into the exact same issue, it needed time to get everything back in order again and CPM started.

In our current situation we are still struggling with residue of all the attempts getting things back online, one of the 3 systems has all its objects duplicated in SmartConsole, all Multi Domain Servers show twice, all domains on that MDS contain all of the objects duplicated, so gateways, networks, you name it.

We will be uploading backups tomorrow and see what you guys can come up with.

[Ran_Kopelman]

Maarten,

I understand that while your Management  Server is UP, you see duplicated objects in Smart Console.

Unfortunately, this is a known issue related to our internal server architecture that is composed of two different databases – Solr and Postgres. It seems that they are uncorrelated and that is the reason for this cosmetic issue.

The “solr-cure” procedure should have fixed that, I know that you have already executed it but it seems that you are missing a fix in order for it to work properly in your environment.

I asked my team to provide you this extra fix.   

It is worth mentioning that in R81.10 we are no longer using Solr and such issues should not happen! (Including the original issue @genisis__ )

Regarding what happened to you in November, I would appreciate it if you could share with me privately the SR number so I can look at it.

Thanks,
Ran  

[Tommy_Forrest]

Count me in as to having the issue of the CPM  process not starting in a timely manner after an upgrade to JHT68 from 23.

1h40 minutes in after reboot on a 5150 and it is still not back up.  However, I'm not seeing the errors noted in sk176304. 

I just see it walking through incrementing revisions.  Connecting to revision 1833, Connecting to revision 1834, so on and so forth.

Throw in some java errors for good effort.

Opened a TAC case, waiting for call back, 30 minutes into that wait.

**UPDATE: The primary MDS finished the upgrade in a hair under 10 hours.

Kicked off the backup MDS and it's doing the same thing.

***Update-to-the-update - Discovered why it took so long.  We do/did not have database purging active on the CMA's.  Each CMA had several thousand revisions the JHT had to update during the upgrade.

Apparently, it is best practice to have database purging enabled.  For that see sk170059,