index log files after moving logs from one log server to another log server

[checkmate-Go]

We recently replaced our open server hardware that was running the dedicated log server.  I moved only the .log files (not other log files) from /var/log/opt/CPsuite-R81.20/fw1/log/2024-11-*.log to /var/log/opt/CPsuite-R81.20/fw1/log/ on another server ranging from 2024-11-1 to 2024-11-19.

I’m wondering how indexing works after this transfer. Does it happen automatically, or do I need to manually re-index the logs? I only need the last 14 days of indexed logs.

It seems like the article I found is relevant, but I wanted to confirm: is moving just the .log files sufficient, or should I have moved the other log files as well?

https://support.checkpoint.com/results/sk/sk111766

[PhoneBoy]

While I believe the log files alone are sufficient, the other files have to be rebuilt if they are not transferred.
It's better to move them all.

And yes, you will have to manually reindex the logs after moving files into the directory.

[checkmate-Go]

what does "rebuilt if they are not transferred mean"?

I just copied all the .log files and followed instruction from this sk artice:

https://support.checkpoint.com/results/sk/sk111766

I am not sure if it indexed logs or not. How can I verify that? How long does it generally take to re-index logs?

[AkosBakos]

Don't forget, indexing the logs takes a while.

And don't forget evstop, and evstart. Ususallly thats why we don't apply this after upgrades. Is not worth the time.

Akos

[PhoneBoy · Accepted solution]

There are "pointer" files that are necessary for working with the logs (thus why the instructions state to copy $FWDIR/log/*.log* instead of $FWDIR/log/*.log.
I was mistaken that they are rebuilt automatically, you'd have to use fw repairlog (from the CLI) to do that.

As for whether the logs will get imported/indexed automatically, I would assume this would not be the case if you simply copied the files over.
Starting the reindexing process (as described in the SK) ensures this will be done.
The time will depend on the amount of logs, overall management/log server load, etc.
You will notice some increased CPU during this time, which will "back off" when other management processes need to use the CPU.
This is normal, expected behavior.

[checkmate-Go · Accepted solution]

steps that I followed to resolve this issue:

Running sk111766 and then performing the below:

• • This is required because we are removing the indexes that already exists so we don't create duplicates.
• This also removes the FetchedFiles, which tells the server if files are already indexed.
• So we remove the FetchedFiles and the indexes then when we restart services it will index xxx days' worth of logs.
• If you do not run the commands listed and only run sk111766 then it will not index the log files from before sk111766 was ran.

1. # cpstop
2. b. # rm -r $RTDIR/log_indexes/other*
3. c. # rm -r $RTDIR/log_indexes/audit*
4. d. # rm -r $RTDIR/log_indexes/firewallandvpn*
5. e. # rm -r $RTDIR/log_indexes/smartevent*
6. f. # rm $INDEXERDIR/data/FetchedFiles
7. g. # rm -r $INDEXERDIR/data/CpmiLocalCopy
8. h. # cpstart

(I went through these steps but I am not sure if it fixed the issue. I was not seeing any indexed logs even after going through them)

The following steps actually started showing indexed logs in smart console.

Go to expert mode:   fw repairlog -u 2024-11-15_032113_2226.log          (you have to repair all the logs file that you want to repair)

After running fw repair log, I am seeing indexed logs. Thanks!

[CheckMate-R77]

Hello,

Is it possible, to import (copy) index files (audit, other, firewallandvp, smartevent) from backup and use them instead of waiting (for example a week) for reindexing? What are the requirements for this procedure (e.g. FetchedFiles modifications or anything like that)?

Regards

Mirek

[CheckMate-R77]

And what that output from doctor-log means (after manual copying archive indexes):

"other_2025-07-13T00-00-00 should be transient, should have changed to transient after 30 days"

Maintenance Configuration:
Maintenance type : daily
Keep logs for : 730
Delete indexes older than: 365 days

In  $INDEXERDIR/log_indexer_custom_settings.conf 

:days_to_index (120)

In SMS Logs->Storage Daily Logs Retention Configuration:

Keep indexed logs for no longer than 365 days

Keep log files for an extra 365 days

[PhoneBoy]

No idea if that's possible and recommend asking TAC.