topic Re: Reports data loss in General Topics

Reports data loss

Bernardes — Tue, 06 Dec 2022 23:19:20 GMT

Hello Mates!

I have a distributed environment (SMS + Gateway Cluster + Smart Event and Logs) and did an upgrade on SMS and Smart Event from 80.40 to 81.10. The SMS upgrade was by migrate process(create a new VM) and the Smart Event was by CPUSE.

After that, I can't see the reports about Application and URL Filtering or any other blade, look at this example bellow:

When I filter by "All Time" it just shows me this month which is the date I did the upgrade until now.

As I said, the Smart Event doesn't change, the upgrade was by CPUSE, but to the SMS was created a new VM and imported the migrate file.

Is there anything missing to bring from the old SMS to the new SMS so that I can see the old report information from other months?

Thank you!

Re: Reports data loss

Chris_Atkinson — Tue, 06 Dec 2022 23:31:39 GMT

Did you move and reindex the old log data?

R81

GNG-1259,

PMTR-52941 R81 includes new logs indexing mechanism, so when upgrading Management server/Log Server/Multi-Domain Server/Multi-Domain Log Server/SmartEvent from R80.x, old log indexes are not upgraded.

The indexing mechanism will re-index the last 24 hours automatically. To increase the period of offline indexing (how far in the past to re-index the logs), see sk111766.

Re: Reports data loss

PhoneBoy — Tue, 06 Dec 2022 23:52:36 GMT

When upgrading to an R81.x release from an R80.x release, you will need to manually reindex the logs after upgrading.
This is because we’ve changed the index format in R81.
See: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk111766&partition=Advanced&product=Logging

Re: Reports data loss

the_rock — Wed, 07 Dec 2022 01:29:26 GMT

You got 100% correct answers from Chris and Dameon, as thats exactly what you need to do. I been through this 3 times before and that was sk I had to follow.

Re: Reports data loss

Bernardes — Wed, 07 Dec 2022 12:56:08 GMT

Hello @Chris_Atkinson @PhoneBoy @the_rock ! First of all, thank you very much!

I did the process of index old logs and configured for 60 days both on "./log_indexer -days_to_index 60" and in the storage config.

After that, I can see logs just 14 days since yesterday to back.

I saw the secondary process of the sk164553, but I didn't understand very well.

Is this second process from sk164553 is mandatory or it already would works with just the process of the index (./log_indexer -days_to_index 60) ?

Thank you

Re: Reports data loss

PhoneBoy — Wed, 07 Dec 2022 13:48:46 GMT

The procedure in sk164553 is if you need to index a specific log file.
If you’re indexing the last 60 days of logs per your CLI command this shouldn’t be necessary.

Note that it can take some time to index the log files since it is done at a lower priority.
You will see the CPU utilization on the management server higher than normal as a result, but the indexing process backs off when the CPU is needed for other tasks.

Re: Reports data loss

the_rock — Wed, 07 Dec 2022 13:57:08 GMT

Keep in mind, it may take 24 hours for all to show up, I seen that before myself.

Re: Reports data loss

Bernardes — Thu, 08 Dec 2022 18:38:50 GMT

@PhoneBoy thank you for the explanation!

I can see a high-load CPU on Smart Event appliance indeed. And slowly I saw the old log being shown on the smart console.

Re: Reports data loss

Bernardes — Thu, 08 Dec 2022 18:42:46 GMT

@the_rock thank you for the advice!

It has taken more than 3 days and just now I can see de CPU in normal use.

Is there any way to monitor when the index process is running or is finished?