Security Management Server is not running (after migration 77.30->80.30)

[Jerry]

any clues what steps to take in order to bring CPM/FWM live again?

few facts:

Product Name: Check Point Security Management Server
Major version: 6
Minor version: 0
Build number: 993000001
Is started: 0
Active status: active
Status: Security Management Server is not running

[Expert@cpm:0]# cpwd_admin list
APP PID STAT #START START_TIME MON COMMAND
CPVIEWD 21947 E 1 [10:56:04] 11/11/2019 N cpviewd
CPVIEWS 21950 E 1 [10:56:04] 11/11/2019 N cpview_services
CPD 21965 E 1 [10:56:04] 11/11/2019 N cpd
FWD 22054 E 1 [10:56:05] 11/11/2019 N fwd -n
FWM 0 T 1 [10:56:05] 11/11/2019 N fwm
STPR 22071 E 1 [10:56:05] 11/11/2019 N status_proxy
CLOUDGUARD 22104 E 1 [10:56:05] 11/11/2019 N vsec_controller_start
CPM 22451 E 1 [10:56:06] 11/11/2019 N /opt/CPsuite-R80.30/fw1/scripts/cpm.sh -s
SOLR 22505 E 1 [10:56:06] 11/11/2019 N java_solr /opt/CPrt-R80.30/conf/jetty.xml
RFL 22560 E 1 [10:56:06] 11/11/2019 N LogCore
SMARTVIEW 22608 E 1 [10:56:06] 11/11/2019 N SmartView
INDEXER 22684 E 1 [10:56:06] 11/11/2019 N /opt/CPrt-R80.30/log_indexer/log_indexer
SMARTLOG_SERVER 22730 E 1 [10:56:06] 11/11/2019 N /opt/CPSmartLog-R80.30/smartlog_server
DASERVICE 23082 E 1 [10:56:07] 11/11/2019 N DAService_script

1. migration from 77.30 to 80.30 was done based on https://community.checkpoint.com/t5/General-Management-Topics/R77-30-to-R80-10-SMS-Migration/td-p/36...

2. new SMS is on different IP address than the old one - we need to remain with new SMS on new IP address as the old one is still up&running and serves 77.30 clusters

3. goal is to have new SMS with content from old one with new Cluster.

4. wanted to reach out to TAC but 1st I believe is the so called "best practice" to ask your mates ... so I did 🙂

thanks for all your hints

Jerry

[Maarten_Sjouw]

Did you replace the license? FWM will not start without a valid license.

[Jerry]

oh no ... I think there is only a trial one on that new SMS, can I apply the trial one?

also I've got following in cpm.elg:

11/11/19 11:31:52,293 ERROR java_sic.remote.SicRemoteTrustManager [dbsyncTaskExecutor-1]: Failed to validate server certificate [-6]
11/11/19 11:31:52,294 WARN db_sync.server.CpmSession [dbsyncTaskExecutor-1]: Login failure to a0eebc99-afed-4ef8-bb6d-fedfedfedfed on a.b.c.d. message: Marshalling Error: java.security.cert.CertificateException: Failed to validate server certificate -6

[PhoneBoy]

Depending on the vintage of the initial installation of the SMS, the old ICA may be signed using algorithms we no longer consider valid in R80.x.
There is a fix for this that the TAC should be able to provide.

[Jerry]

there is a valid license (trial) but valid (cplic print checked).
fwm indeed does not run 😞 ...

APP PID STAT #START START_TIME MON COMMAND
CPVIEWD 21947 E 1 [10:56:04] 11/11/2019 N cpviewd
CPVIEWS 21950 E 1 [10:56:04] 11/11/2019 N cpview_services
CPD 21965 E 1 [10:56:04] 11/11/2019 Y cpd
FWD 22054 E 1 [10:56:05] 11/11/2019 N fwd -n
FWM 0 T 0 [11:00:07] 11/11/2019 N fwm

[Maarten_Sjouw]

The problem is that your original lic (original IP) comes along with the import and is the only one used. The trial might not even be there anymore.

[Jerry]

[11 Nov 12:47:47] [PreupgradeVerifierRunner::exec] ERR: Preupgrade verifier found errors
[11 Nov 12:47:47] [PreupgradeVerifierRunner::exec] Preupgrade verifier's output:

...

[11 Nov 12:47:47] .<-- PreupgradeVerifierRunner::exec
[11 Nov 12:47:47] <-- ConditionalExecutor::exec
[11 Nov 12:47:47] [ActivitiesManager::exec] ERR: Activity 'ConditionalExecutor' failed
[11 Nov 12:47:47] [ProgressUpdater::UpdateProgressToGaia] Progress Updated to '28.5714
[11 Nov 12:47:47] [ActivitiesManager::exec] WRN: Activities execution finished with errors
[11 Nov 12:47:47] [ActivitiesManager::exec] WRN: Activities 'ConditionalExecutor' have failed
[11 Nov 12:47:47] [ActivitiesManager::exec] Designated exit code is 1

so clearly it points to the SOURCE not being ready for migration ... zonk 😞

[Jerry]

it turns out ... to be sk114739 😞 

none-Unicode ch-ters ... 340 objects with "_" ... now local admin is "renaming" all of them with emergency RFC.

then export should go as expected so do the import on R80.30

cheers Marteen and thanks for heads up!

J.

[Maarten_Sjouw]

Yeah that will make a difference.

[Jerry]

problem is that is barely possible to get a rid of all "_" within names of objects as some of them are "unchangeable" like LocalHostAll_Interfaces etc. We found today with my Customer that we need TAC SR to get this sorted, I'm not feeling confident any longer running this:

./migrate export /tmp/db/export

and receiving this all over again (after all the "rename's"

================================
Action items before upgrade:
================================

Errors found! To create a working environment, the errors must be corrected.
==============================================================================


Title: Objects with non-Unicode characters
-----
* Description: The database contains objects with non-Unicode characters. Remove the non-Unicode characters or follow the instructions in sk114739 before running the upgrade process.

These tables contain objects with non-Unicode characters:

fw_policies

--- so what does it mean? it means that this DB isn't yet "READY" to be migrated to R80 and require lots of work in order to prep it up and clean up all the inherited and obsolete most of the time objects. whatta nightmare it is ... and cutover approaching just tomorrow eve ... 😞

[Jerry]

and then I found sk109795 and ... will tell you more when R80.30 starts up 🙂 (if it will!)

[Jerry]

DB imported well (270MB instead of 23GB!) but:

SMS is by its interface on .7 but object in SMS is as it was on the old name with IP in topo as .6 (will play with the object tomorrow when on site);

SMS has old obsolte objects - I will remove them as we as remove them from rulebases

new SMS R80.30 seems stable so far but whole objects/policies need a proper housekeeping so this is my task for tomorrow 

also new Cluster and its SIC's need to be done from scratch so new Cluster need to be created but thats piece of cake 😛

will give you more findings when happening.

so far so good !

J.

ps. no SR was needed 😄 

[Maarten_Sjouw]

Good to hear it finally worked out, except for the IP of the SMS object, I know there is an SK about chnging it but after 10 minutes search I gave up finding it, sorry.

[Jerry]

precisely ... that's what I'm looking for now Marteen, my SMS network-wise is on .7 but in SMS it is still on the OLD name and IP and I'm not able to change this despite all licenses detached and deleted... @PhoneBoy - can you give us a hint how to rename/reIP SMS in SMS or clish? 🙂 Cheers!

[Jerry]

sk112914 ?

[Maarten_Sjouw]

I know this is a dangerous one but did you see SK42071?
As the R80 database is different, but the other parts could work, completely delete all SIC, then change name and IP and then reissue SIC.

[Jerry]

I did all this and at the moment I have no gateways with my SMS just cannot rename it as there are obsolete non-SIC'able licenses. Tried cplic db_rm * but cannot remove them 😞 Stuck and my customer is push to rename SMS before BOARDING new Cluster (obviously logical!).
--
[Expert@SMS-XXX:0]# cplic db_rm aiGCrrPBr-Hpd2w73o2-kSqttheJ4-rrqjQuNDk
Removing license from database ...
--
Cannot deleted license. License is attached to another module.
Please detach the license first!

help !!!

[Jerry]

Done!

[Expert@SMS-XXX:0]# cplic db_rm -all
Removing license from database ...
cprlic rm <signature>

[Maarten_Sjouw]

cplic print -x
Does that show anything?

[Jerry]

yes Marteen, I'm all sorted. New Cluster boarded and just adopting policies from the import 🙂 Yay! All sorted with no SR's from CP TAC etc. Customer is already happy that their R80.30 SMS is all "Green" and working.

Thanks for your assistance mate!

Cheers

[Maarten_Sjouw]

Good to hear it all worked out in the end.