topic Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs in Firewall and Security Management

Migrating MDM/MLM from R80.30 to R81.10 with all logs

cem82 — Sun, 14 Nov 2021 02:56:49 GMT

We are looking to migrate our current R80.30 MDM +MLM to R81.10 on new servers. I have tested with migrate_server export -v R81.10 -x <filename> on the MDM as a starting point but in R81.10 SmartConsole we see what appears to only go back to midnight of the previous day from when the migrate_server was run. How can we extend that back to include all traffic/audit logs?

Thanks

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

Ido_Shoshana — Tue, 16 Nov 2021 12:23:02 GMT

Hi,

Important note: You won’t be able to preserve log-Indexes (the actual SME log-DB), if you’re upgrading to R81.x from R80.x, as the Solr I/S had been upgraded.

so this procedure should only be done on R80.x to R80.x or R81.x to R81.x (like R80.20 to R80.40 or R81 to R81.10).

If upgrading to R81, then you can enlarge the indexing backwards time limit to index older logs/events - more than the default 1 day back (assuming the actual log-files from its Log-Servers still exist).
See sk111766.

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

cem82 — Mon, 15 Nov 2021 19:29:46 GMT

If it's not possible for the log indexes, how about the raw log files after doing a logswitch and copy those over or is that not an option?

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

Ido_Shoshana — Tue, 16 Nov 2021 08:14:08 GMT

yes, this is an option since the log files are kept.

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

cem82 — Tue, 16 Nov 2021 08:20:38 GMT

That's great! So all we'd need to do is copy over the old raw log files to the new server? Are they in the same format and don't need any sort of script to be run to be able to be read by R81.10? If we do that could we also then also follow the SK you mentioned to index the additional days as well?

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

Ido_Shoshana — Tue, 16 Nov 2021 14:25:14 GMT

No need to copy. The log files should have already been imported as part of the migrate server as you have shown above (-x / -l).

Yes, same format - all good. Then you can follow the sk111766 to re-index back as many log files days of data as you wish.

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

SunilShivnani1 — Fri, 26 May 2023 16:46:07 GMT

For MLM migration on new hardware running same version (R81.10), if we are not worried about log files migration, migrate_server export -v R81.10 -x <filename> should be enough. Is that correct?

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

Maja_B — Tue, 05 Sep 2023 08:20:25 GMT

And what about Mgmt upgrade from R80.40 to R81.20? We tried exporting 30 days of logs, but it took a lot of time. We decided to archive the logs, delete them and do migrate export with 5 days of logs. Can we just copy the archived logs back to R81.20 Mgmt and index them?

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

emmap — Tue, 05 Sep 2023 08:41:26 GMT

Yes you can manually copy log files over (make sure you have all the pointer files as well) and (optionally) re-index them.

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

Maja_B — Tue, 05 Sep 2023 08:45:07 GMT

Thanks!

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

Amir_Senn — Tue, 05 Sep 2023 09:23:05 GMT

From R80.xx to R81.xx there's a different version of SOLR (newer version has newer SOLR that works better) so everything will need to be re-indexed so don't bother exporting/importing your indexes. So instead of using -x (export logs and indexes) you can use -l (export logs only) and that will be faster.

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

Daniel_Fidlin — Tue, 05 Sep 2023 11:02:54 GMT

Hi Maja_B,

As my colleagues said, the SOLR version was upgraded in Version R81 and above, I would recommend for an upgrade from R80.xx versions to use (migrate_server export -v R81.xx -l <file name>) command to export only the log files not including the indexes.

this will transfer all logs under $FWDIR/log to the new version, than you may increase the days to index backwards as much days as you desire, you can track the index process with the following command (watch -d 'cat $INDEXERDIR/data/FetchedFiles').

to increase the days to index, just follow this simple steps:
1) cp $INDEXERDIR/log_indexer_custom_settings.conf $INDEXERDIR/log_indexer_custom_settings.conf.ORIGINAL

2) vi $INDEXERDIR/log_indexer_custom_settings.conf

3) add under ":max_disk_space_usage (0)" the following ":days_to_index (<number of days>)" then save changes

4) run 'stopIndexer' & 'startIndexer'

Kind regards, Daniel Fidlin

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

Maja_B — Tue, 05 Sep 2023 14:55:02 GMT

Thanks. We already did it with -l flag.

Re: Migrating MDM/MLM from R80.30 to R81.10 with all logs

Maja_B — Tue, 05 Sep 2023 14:57:45 GMT

Thanks, Daniel!