R77.30 to R80.10 Migrating logs

[Kennith_Tucker]

Upgrading R77.30 to R80.10, Management server only to start. Current Management server is a VM running GAIA R77.30, a new VM with R80.10 has been created. The plan is to use the Migrate command to export the database, swap IP addresses, then import the database on the new server. We have a large amount of log files and do not want to use the “-l” to include the logs. We are trying to minimize the amount of time to get the new server online and would like to move most of the logs over prior to the move.  

Can just the .log files be copied over, or do the other files (.log_stats, .logaccount_ptr, .loginitial_ptr, .logptr) need to be copied as well, or will the new server create these files?

[PhoneBoy]

Historically speaking, you could get away with just moving the .log files and everything else would regenerate.

However it's a little more complicated these days as aside from the pointer files, there are indexes used by SmartLog and SmartEvent.

Refer to the "Importing Log Files from SmartEvent Servers" section of the following document: Logging and Monitoring R80.10 (Part of Check Point Infinity) 

In short, you need to copy all the files (.log and related pointer files) and tell the indexer to index files older than 14 days from the date of initial server installation. 

[Kaspars_Zibarts]

It all depends on the amount of logs you have. We ended up giving up on idea trying to transfer all 30 days worth of logs (~6TB) - it just took so much CPU time to index those old files that MLM could not keep up indexing newly arriving logs. In nutshell it took 3 days to index one week of logs. Esx VM with 16 cores and 64GB RAM.

Something Checkpoint should improve going forward - been on my wishlist for long time, log handling during upgrade...

[Paul_Warnagiris]

@Dameon.  I just did an MDM upgrade where I did the PUV, fixed all the errors and then exported.  I created a brand new 80.10 box and imported the MDM config.  Everything went well and we copied over logs from 1/1/18 through migration.  However, I can't see any logs except for current logs.

Quick question on this.  SK111766 and your link above to "Importing Log Files from SE Servers" talks specifically about SmartEvent.  What if there is no SmartEvent running there? 

These files do not exist in my log_indexer_custom_settings.conf

:time_restriction_for_fetch_all (<existing_data>) 
:time_restriction_for_fetch_all_disp (<existing_data>)

My max_disk_space_usage is set to 0.

I followed SK111766 anyway in hopes it would work but it does not.  I'm running this on an MDM so I looked here /opt/CPrt-R80/log_indexer/log_indexer_custom_settings.conf and here /var/opt/CPmds-R80/customers/AWESOME.CUSTOMER_Management_Server/CPrt-R80/log_indexer/log_indexer_custom_settings.conf and no love.  The SK talks about Open the R80.x SmartEvent Server object properties - expand Logs - click on Storage page - configure the Disk Space Management.  But that doesn't exist.  Sure you can edit the disk space, but there is no where to say "keep logs X amount of days."  So my question is does this apply to SmartLog or just SmartEvent.  Regardless of the answer is there an easy way to import old logs from 77.30 and have SmartLog see them?

Here is my particular CMA .....custom_settings.conf file (if its relevant).  Any pointers/links/SKs you can send me would be greatly appreciated.

[Expert@CX.mds:0]# more /var/opt/CPmds-R80/customers/AWESOME.CUSTOMER_Management_Server/CPrt-R80/log_indexer/log_indexer_custom_settings.conf
(
:data ("/var/opt/CPmds-R80/customers/AWESOME.CUSTOMER_Management_Server/CPrt-R80/log_indexer/data")
:server_port ("a.b.c.d:18244")
:dns_resolving (true)
:dns_backresolving (true)
:connections (
:domain (
:management (
:name (a.b.c.d)
:uuid ()
:log_files (all)
:is_local (true)
:read_mode (CPMI)
)
:log_servers (
: (
:name (a.b.c.d)
:uuid ()
:log_files (all)
:folder ("/opt/CPmds-R80/customers/AWESOME.CUSTOMER_Management_Server/CPsuite-R80/fw1/log")
:is_local (true)
:read_mode (FILES)
)
)
)
)
:max_disk_space_usage (0)
)
[Expert@CX.mds:0]#

Thanks,
Paul

[PhoneBoy]

The indexes for SmartLog and SmartEvent are the same in R80.x.

Which means, even if SmartEvent isn't used, the same process is used to import/index the logs.

I suppose you can manually try adding the lines you mention to the .conf file and do an evrestart.

If that doesn't work, I recommend engaging with the TAC.