Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Gilles_Lerat
Participant

Disconnected sessions preventing upgrade from R80.20 to R80.30

 

Hello Checkmates, 

 

I am upgrading a Check Point Management Server from R80.20 to R80.30 

Everything works fine during upgrade. The Webui is restarted

 

But we can't connect to the Management Server. Turns out that CPM has not initialized properly.

 

[Expert@DCTSMS:0]# /opt/CPsuite-R80.30/fw1/scripts/cpm_status.sh
Check Point Security Management Server is during initialization

 

We see that in the $FWDIR/log/cpm.elg file, that there are several logs worth investigating.

One of them : 

ERROR fts.solr.Jpa2SolrManagerImpl [main]: SOLR is completely out of sync!!! more than 5000 jpa2FtsRecords are out of sync.

 

... leads us to sk116014 : CPM process initialization is slow after backup restore

But this time, it's not slow,  it's super slow. 

3 hours and no progress (of the size of the cpm.elg file). 

We find that in this file, there are lines like : 

Caused by: CpmGeneralException{base='com.checkpoint.management.is.exceptions.CpmGeneralException: java.lang.SecurityException: Tried to open non existing session with id d16200d0-e68e-42b5-ad37-1a4da8f3b5de', errorCode='CP_ERR_UNSPECIFIED', errorFamily='null', messageForUser='null', message='java.lang.SecurityException: Tried to open non existing session with id d16200d0-e68e-42b5-ad37-1a4da8f3b5de'}
        at com.checkpoint.management.object_store.fts.solr.Jpa2SolrManagerImpl.syncJpaDbWithFtsIndex(Jpa2SolrManagerImpl.java:688)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl.syncJpaDbWithFtsIndex_aroundBody194(ObjectStoreSessionImpl.java:3600)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl$AjcClosure195.run(ObjectStoreSessionImpl.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.checkpoint.management.dleserver.coresvc.internal.TransactionRetrySvcImpl.proceed(TransactionRetrySvcImpl.java:79)
        at com.checkpoint.management.dle.aspects.TransactionRetryAspect.aroundOperation(TransactionRetryAspect.java:7)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl.syncJpaDbWithFtsIndex(ObjectStoreSessionImpl.java:2500)
        at com.checkpoint.management.object_store.ObjectStoreImpl.syncJpaDbWithFtsIndex_aroundBody14(ObjectStoreImpl.java:56)
        at com.checkpoint.management.object_store.ObjectStoreImpl$AjcClosure15.run(ObjectStoreImpl.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.checkpoint.management.dleserver.coresvc.internal.TransactionRetrySvcImpl.proceed(TransactionRetrySvcImpl.java:79)
        at com.checkpoint.management.dle.aspects.TransactionRetryAspect.aroundOperation(TransactionRetryAspect.java:7)
        at com.checkpoint.management.object_store.ObjectStoreImpl.syncJpaDbWithFtsIndex(ObjectStoreImpl.java:83)
        ... 32 more
Caused by: java.lang.SecurityException: Tried to open non existing session with id d16200d0-e68e-42b5-ad37-1a4da8f3b5de
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl.isPublished_aroundBody192(ObjectStoreSessionImpl.java:542)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl$AjcClosure193.run(ObjectStoreSessionImpl.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.checkpoint.management.dleserver.coresvc.internal.TransactionRetrySvcImpl.proceed(TransactionRetrySvcImpl.java:79)
        at com.checkpoint.management.dle.aspects.TransactionRetryAspect.aroundOperation(TransactionRetryAspect.java:7)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl.isPublished(ObjectStoreSessionImpl.java:1010)
        at com.checkpoint.management.object_store.fts.solr.Jpa2SolrManagerImpl.syncJpaDbWithFtsIndex(Jpa2SolrManagerImpl.java:304)

========================

So it seems that session ID d16200d0-e68e-42b5-ad37-1a4da8f3b5de is non existent and causing problems regarding CPM initialization. 

 

I try to suppress this session ID using the method I have seen on one of the forums :

mgmt_cli discard --port 443 uid d16200d0-e68e-42b5-ad37-1a4da8f3b5de
Username: sc-admin
Password:
code: "generic_server_error"
message: "Management server failed to execute command"

============================================================

It doesn't work. 

Meanwhile, I have noticed that, indeed, there is a ghost session in the Smartcenter that we can't suppress using Smartconsole (or even GUIDBedit).  See attached file.

 

I have tried to remove ghost session using the psql_client command...  But I don't know how to proceed.

Any help ? 

 

Thanks,

                               Gilles

 

 

0 Kudos
7 Replies
Maarten_Sjouw
Champion
Champion

First of all open a TAC case, second do not reboot or cpstop anything else like that, just let it run, just keep an eye on Top and see if it runs out of memory (is it using swap space?).
Regards, Maarten
0 Kudos
Gilles_Lerat
Participant

 

Thanks. 

A TAC number has been escalated to Check Point (SR#6-0001795556) .

 

Best regards, 

                            Gilles

 

 

0 Kudos
Tomer_Noy
Employee
Employee

TAC should be able to help you clear the sessions, and if needed, involve R&D.

Regarding the attempt to remove the sessions yourself using SQL commands, that is highly not recommended. The DB structure in R80 is not trivial and making direct modifications to the DB data is very risky. You could damage the data integrity in a way that would be very difficult to recover from.

Direct DB manipulation should be done rarely, and only using formal SKs with scripts that perform the modification in a safe and contained way.
cespinoza
Explorer

Hello,

 

I have the same issue, How did you manage to solve this

01/04/20 22:06:59,369 ERROR scheduling.support.TaskUtils$LoggingErrorHandler [taskScheduler-9]: Unexpected error occurred in scheduled task.
java.lang.SecurityException: Tried to open non existing session with id f98f9b12-a342-48e9-908a-d922e247fff7

Currently there is also a session that  has the lock  the mgmt that cant be discarted

Expert@CP-MGMT:0]# psql_client cpm postgres -c "select applicationname,objid,creator,state,numberoflocks,numberofoperations,creationtime,lastmodifytime from worksession where state != 'PUBLISHED' and state != 'DISCARDED' and (numberoflocks != '0' or numberofoperations != '0');"
applicationname | objid | creator | state | numberoflocks | numberofoperations | creationtime | lastmodifytime
-----------------+--------------------------------------+---------+-------+---------------+--------------------+-------------------------+-------------------------
SmartConsole | 84f2d4d2-0d64-4158-807f-ae9af5e642d6 | admin | OPEN | 5 | 6 | 2020-03-16 12:27:33.778 | 2020-04-01 21:48:53.276
(1 row)

 

Best regards

0 Kudos
zsigmondrichard
Participant

Hi,

 

We have the same issue. Were you able to solve this?

 

Thanks

0 Kudos
Gilles_Lerat
Participant

 

As for us, we were not able to solve this. 

We had to have TAC do a remote session on the management and turn it back to R80.20. 

The problem was finally solved by upgrading directly to R80.40. 

 

0 Kudos
zsigmondrichard
Participant

Thank you for the response!

 

Currently, we are on R80.40 T77 and wanted to hotfix to Take 94. After the hotfix installation I wasn't able to log in to SmartConsole. Using psql_client command we found, that I have a stuck session, but we can not delete it due to "generic_server_error - CP_ERR_UNSPECIFIED".

We revert from snapshot, but the issue still persist as it shows, that my user has stuck for 2 weeks...Also, before hotfixing, I did not face this issue and my user was stuck for 2 weeks according to the output of the psql command.

We have a TAC case opened.

 

Thank you!

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events