Disconnected sessions preventing upgrade from R80.20 to R80.30

[Gilles_Lerat]

Hello Checkmates, 

I am upgrading a Check Point Management Server from R80.20 to R80.30 

Everything works fine during upgrade. The Webui is restarted

But we can't connect to the Management Server. Turns out that CPM has not initialized properly.

[Expert@DCTSMS:0]# /opt/CPsuite-R80.30/fw1/scripts/cpm_status.sh
Check Point Security Management Server is during initialization

We see that in the $FWDIR/log/cpm.elg file, that there are several logs worth investigating.

One of them : 

ERROR fts.solr.Jpa2SolrManagerImpl [main]: SOLR is completely out of sync!!! more than 5000 jpa2FtsRecords are out of sync.

... leads us to sk116014 : CPM process initialization is slow after backup restore

But this time, it's not slow,  it's super slow. 

3 hours and no progress (of the size of the cpm.elg file). 

We find that in this file, there are lines like : 

Caused by: CpmGeneralException{base='com.checkpoint.management.is.exceptions.CpmGeneralException: java.lang.SecurityException: Tried to open non existing session with id d16200d0-e68e-42b5-ad37-1a4da8f3b5de', errorCode='CP_ERR_UNSPECIFIED', errorFamily='null', messageForUser='null', message='java.lang.SecurityException: Tried to open non existing session with id d16200d0-e68e-42b5-ad37-1a4da8f3b5de'}
        at com.checkpoint.management.object_store.fts.solr.Jpa2SolrManagerImpl.syncJpaDbWithFtsIndex(Jpa2SolrManagerImpl.java:688)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl.syncJpaDbWithFtsIndex_aroundBody194(ObjectStoreSessionImpl.java:3600)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl$AjcClosure195.run(ObjectStoreSessionImpl.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.checkpoint.management.dleserver.coresvc.internal.TransactionRetrySvcImpl.proceed(TransactionRetrySvcImpl.java:79)
        at com.checkpoint.management.dle.aspects.TransactionRetryAspect.aroundOperation(TransactionRetryAspect.java:7)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl.syncJpaDbWithFtsIndex(ObjectStoreSessionImpl.java:2500)
        at com.checkpoint.management.object_store.ObjectStoreImpl.syncJpaDbWithFtsIndex_aroundBody14(ObjectStoreImpl.java:56)
        at com.checkpoint.management.object_store.ObjectStoreImpl$AjcClosure15.run(ObjectStoreImpl.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.checkpoint.management.dleserver.coresvc.internal.TransactionRetrySvcImpl.proceed(TransactionRetrySvcImpl.java:79)
        at com.checkpoint.management.dle.aspects.TransactionRetryAspect.aroundOperation(TransactionRetryAspect.java:7)
        at com.checkpoint.management.object_store.ObjectStoreImpl.syncJpaDbWithFtsIndex(ObjectStoreImpl.java:83)
        ... 32 more
Caused by: java.lang.SecurityException: Tried to open non existing session with id d16200d0-e68e-42b5-ad37-1a4da8f3b5de
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl.isPublished_aroundBody192(ObjectStoreSessionImpl.java:542)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl$AjcClosure193.run(ObjectStoreSessionImpl.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.checkpoint.management.dleserver.coresvc.internal.TransactionRetrySvcImpl.proceed(TransactionRetrySvcImpl.java:79)
        at com.checkpoint.management.dle.aspects.TransactionRetryAspect.aroundOperation(TransactionRetryAspect.java:7)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl.isPublished(ObjectStoreSessionImpl.java:1010)
        at com.checkpoint.management.object_store.fts.solr.Jpa2SolrManagerImpl.syncJpaDbWithFtsIndex(Jpa2SolrManagerImpl.java:304)

========================

So it seems that session ID d16200d0-e68e-42b5-ad37-1a4da8f3b5de is non existent and causing problems regarding CPM initialization. 

I try to suppress this session ID using the method I have seen on one of the forums :

mgmt_cli discard --port 443 uid d16200d0-e68e-42b5-ad37-1a4da8f3b5de
Username: sc-admin
Password:
code: "generic_server_error"
message: "Management server failed to execute command"

============================================================

It doesn't work. 

Meanwhile, I have noticed that, indeed, there is a ghost session in the Smartcenter that we can't suppress using Smartconsole (or even GUIDBedit).  See attached file.

I have tried to remove ghost session using the psql_client command...  But I don't know how to proceed.

Any help ? 

Thanks,

                               Gilles

[Maarten_Sjouw]

First of all open a TAC case, second do not reboot or cpstop anything else like that, just let it run, just keep an eye on Top and see if it runs out of memory (is it using swap space?).

[Gilles_Lerat]

Thanks. 

A TAC number has been escalated to Check Point (SR#6-0001795556) .

Best regards, 

                            Gilles

[Tomer_Noy]

TAC should be able to help you clear the sessions, and if needed, involve R&D.

Regarding the attempt to remove the sessions yourself using SQL commands, that is highly not recommended. The DB structure in R80 is not trivial and making direct modifications to the DB data is very risky. You could damage the data integrity in a way that would be very difficult to recover from.

Direct DB manipulation should be done rarely, and only using formal SKs with scripts that perform the modification in a safe and contained way.

[cespinoza]

Hello,

I have the same issue, How did you manage to solve this

01/04/20 22:06:59,369 ERROR scheduling.support.TaskUtils$LoggingErrorHandler [taskScheduler-9]: Unexpected error occurred in scheduled task.
java.lang.SecurityException: Tried to open non existing session with id f98f9b12-a342-48e9-908a-d922e247fff7

Currently there is also a session that  has the lock  the mgmt that cant be discarted

Expert@CP-MGMT:0]# psql_client cpm postgres -c "select applicationname,objid,creator,state,numberoflocks,numberofoperations,creationtime,lastmodifytime from worksession where state != 'PUBLISHED' and state != 'DISCARDED' and (numberoflocks != '0' or numberofoperations != '0');"
applicationname | objid | creator | state | numberoflocks | numberofoperations | creationtime | lastmodifytime
-----------------+--------------------------------------+---------+-------+---------------+--------------------+-------------------------+-------------------------
SmartConsole | 84f2d4d2-0d64-4158-807f-ae9af5e642d6 | admin | OPEN | 5 | 6 | 2020-03-16 12:27:33.778 | 2020-04-01 21:48:53.276
(1 row)

Best regards

[zsigmondrichard]

Hi,

We have the same issue. Were you able to solve this?

Thanks

[Gilles_Lerat]

As for us, we were not able to solve this. 

We had to have TAC do a remote session on the management and turn it back to R80.20. 

The problem was finally solved by upgrading directly to R80.40. 

[zsigmondrichard]

Thank you for the response!

Currently, we are on R80.40 T77 and wanted to hotfix to Take 94. After the hotfix installation I wasn't able to log in to SmartConsole. Using psql_client command we found, that I have a stuck session, but we can not delete it due to "generic_server_error - CP_ERR_UNSPECIFIED".

We revert from snapshot, but the issue still persist as it shows, that my user has stuck for 2 weeks...Also, before hotfixing, I did not face this issue and my user was stuck for 2 weeks according to the output of the psql command.

We have a TAC case opened.

Thank you!