Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted

Disconnected sessions preventing upgrade from R80.20 to R80.30

 

Hello Checkmates, 

 

I am upgrading a Check Point Management Server from R80.20 to R80.30 

Everything works fine during upgrade. The Webui is restarted

 

But we can't connect to the Management Server. Turns out that CPM has not initialized properly.

 

[Expert@DCTSMS:0]# /opt/CPsuite-R80.30/fw1/scripts/cpm_status.sh
Check Point Security Management Server is during initialization

 

We see that in the $FWDIR/log/cpm.elg file, that there are several logs worth investigating.

One of them : 

ERROR fts.solr.Jpa2SolrManagerImpl [main]: SOLR is completely out of sync!!! more than 5000 jpa2FtsRecords are out of sync.

 

... leads us to sk116014 : CPM process initialization is slow after backup restore

But this time, it's not slow,  it's super slow. 

3 hours and no progress (of the size of the cpm.elg file). 

We find that in this file, there are lines like : 

Caused by: CpmGeneralException{base='com.checkpoint.management.is.exceptions.CpmGeneralException: java.lang.SecurityException: Tried to open non existing session with id d16200d0-e68e-42b5-ad37-1a4da8f3b5de', errorCode='CP_ERR_UNSPECIFIED', errorFamily='null', messageForUser='null', message='java.lang.SecurityException: Tried to open non existing session with id d16200d0-e68e-42b5-ad37-1a4da8f3b5de'}
        at com.checkpoint.management.object_store.fts.solr.Jpa2SolrManagerImpl.syncJpaDbWithFtsIndex(Jpa2SolrManagerImpl.java:688)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl.syncJpaDbWithFtsIndex_aroundBody194(ObjectStoreSessionImpl.java:3600)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl$AjcClosure195.run(ObjectStoreSessionImpl.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.checkpoint.management.dleserver.coresvc.internal.TransactionRetrySvcImpl.proceed(TransactionRetrySvcImpl.java:79)
        at com.checkpoint.management.dle.aspects.TransactionRetryAspect.aroundOperation(TransactionRetryAspect.java:7)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl.syncJpaDbWithFtsIndex(ObjectStoreSessionImpl.java:2500)
        at com.checkpoint.management.object_store.ObjectStoreImpl.syncJpaDbWithFtsIndex_aroundBody14(ObjectStoreImpl.java:56)
        at com.checkpoint.management.object_store.ObjectStoreImpl$AjcClosure15.run(ObjectStoreImpl.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.checkpoint.management.dleserver.coresvc.internal.TransactionRetrySvcImpl.proceed(TransactionRetrySvcImpl.java:79)
        at com.checkpoint.management.dle.aspects.TransactionRetryAspect.aroundOperation(TransactionRetryAspect.java:7)
        at com.checkpoint.management.object_store.ObjectStoreImpl.syncJpaDbWithFtsIndex(ObjectStoreImpl.java:83)
        ... 32 more
Caused by: java.lang.SecurityException: Tried to open non existing session with id d16200d0-e68e-42b5-ad37-1a4da8f3b5de
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl.isPublished_aroundBody192(ObjectStoreSessionImpl.java:542)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl$AjcClosure193.run(ObjectStoreSessionImpl.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at com.checkpoint.management.dleserver.coresvc.internal.TransactionRetrySvcImpl.proceed(TransactionRetrySvcImpl.java:79)
        at com.checkpoint.management.dle.aspects.TransactionRetryAspect.aroundOperation(TransactionRetryAspect.java:7)
        at com.checkpoint.management.object_store.ObjectStoreSessionImpl.isPublished(ObjectStoreSessionImpl.java:1010)
        at com.checkpoint.management.object_store.fts.solr.Jpa2SolrManagerImpl.syncJpaDbWithFtsIndex(Jpa2SolrManagerImpl.java:304)

========================

So it seems that session ID d16200d0-e68e-42b5-ad37-1a4da8f3b5de is non existent and causing problems regarding CPM initialization. 

 

I try to suppress this session ID using the method I have seen on one of the forums :

mgmt_cli discard --port 443 uid d16200d0-e68e-42b5-ad37-1a4da8f3b5de
Username: sc-admin
Password:
code: "generic_server_error"
message: "Management server failed to execute command"

============================================================

It doesn't work. 

Meanwhile, I have noticed that, indeed, there is a ghost session in the Smartcenter that we can't suppress using Smartconsole (or even GUIDBedit).  See attached file.

 

I have tried to remove ghost session using the psql_client command...  But I don't know how to proceed.

Any help ? 

 

Thanks,

                               Gilles

 

 

0 Kudos
4 Replies
Highlighted

First of all open a TAC case, second do not reboot or cpstop anything else like that, just let it run, just keep an eye on Top and see if it runs out of memory (is it using swap space?).
Regards, Maarten
0 Kudos
Highlighted

 

Thanks. 

A TAC number has been escalated to Check Point (SR#6-0001795556) .

 

Best regards, 

                            Gilles

 

 

0 Kudos
Highlighted
Employee+
Employee+

TAC should be able to help you clear the sessions, and if needed, involve R&D.

Regarding the attempt to remove the sessions yourself using SQL commands, that is highly not recommended. The DB structure in R80 is not trivial and making direct modifications to the DB data is very risky. You could damage the data integrity in a way that would be very difficult to recover from.

Direct DB manipulation should be done rarely, and only using formal SKs with scripts that perform the modification in a safe and contained way.
Ivory

Hello,

 

I have the same issue, How did you manage to solve this

01/04/20 22:06:59,369 ERROR scheduling.support.TaskUtils$LoggingErrorHandler [taskScheduler-9]: Unexpected error occurred in scheduled task.
java.lang.SecurityException: Tried to open non existing session with id f98f9b12-a342-48e9-908a-d922e247fff7

Currently there is also a session that  has the lock  the mgmt that cant be discarted

Expert@CP-MGMT:0]# psql_client cpm postgres -c "select applicationname,objid,creator,state,numberoflocks,numberofoperations,creationtime,lastmodifytime from worksession where state != 'PUBLISHED' and state != 'DISCARDED' and (numberoflocks != '0' or numberofoperations != '0');"
applicationname | objid | creator | state | numberoflocks | numberofoperations | creationtime | lastmodifytime
-----------------+--------------------------------------+---------+-------+---------------+--------------------+-------------------------+-------------------------
SmartConsole | 84f2d4d2-0d64-4158-807f-ae9af5e642d6 | admin | OPEN | 5 | 6 | 2020-03-16 12:27:33.778 | 2020-04-01 21:48:53.276
(1 row)

 

Best regards

0 Kudos