Difference between log files and log indexes

[Ed_Eades]

I am trying to understand the difference between log files and log indexes in terms of retention.  We have a management server setup for Logging&Status, SmartEvent Viewer, and Correlation Unit.  Log indexing is enabled and set to alert when space is below 20% and delete old files below 22%.  Index files is set to delete older than 7 days. The  folder /opt/CPrt-R80/log_indexes  shows 7 days worth of index files.  However the folder /opt/CPsuite-R80/fw1/log shows log files going back several months.  We have cleared out log files from this folder before.  With indexing set to delete older than 7 days we can only go back 7 days on reports but the log files themselves go back past 7 days.  I am trying to understand the difference between the log files themselves and the indexing retention.

Thanks.

[_Val_]

Hi Ed,

Thanks for the explanation. Indeed, there are differences between managing logs and log indexes.

On this pic, you can see both log storage settings and indexing retention management

Logs are stored as the files under $FWDIR/logs, this is a part of MGMT Log Server functionality. There is no automatic built-in mechanism to remove old log files. The only option you have is to start removing older logs when disk space utilization reaches a certain threshold. I have highlighted this part with the blue rectangle. 

Log indexing is done by an indexing engine, and the indexes are stored to $RTDIR/log_indexes. You can set the maximum depth of indexing, which is important for Event Analysis performance and stability. The indexer has a built in retention option, and older indexes are routinely removed.

The main reason not to remove logs automatically is simple. You may want to keep your security logs to maintain ability of investigating past breaches and other security incidents.  In some cases compliance regulations require keeping up to 2 years of logs available. 

So to manage log retention I would advise you to run a cron task with a script that performs backup and removal of older logs. There are quite a few publicly available samples of such a script. On of example is here: Log Backup/Archive Script 

[Ed_Eades]

Thanks for the great information.  I will look into setting up a cron task for the log file removal. 

When running reports and searching logs from SmartDashboard is it only going to look at the index log files which in our case log index retention is 7 days?  If so, how would  you then search the log files past the 7 days?

Thanks again!

[Alejandro_Mont1]

I believe that is correct, you will be able to run reports for 7 days worth of logs as both indexes and log files must be present to be seen in R80x. In order to go back further you would need to re index those log files which is detailed in the R80 admin guide and sk111766. Once the setting is made you will observe higher CPU/RAM consumption until all those log files have been indexed (could take hours and even days depending on your resources/amount of logs).

[AlekseiShelepov]

You also have the option to open SmartView Tracker and check logs there:

C:\Program Files (x86)\CheckPoint\SmartConsole\R80.10\PROGRAM\CPlgv.exe

Index files allow you to see/search logs for a period, for which you have these index files (a week in your case). But you still can open separate log files and check some traffic there, but you would be able to see/search only in this specific file (for example, for a day if you set up a midnight log file switch).

So, index files give you ability to look into multiple log files at once, but you pay for that with additional hardware usage (CPU and free space on HDD).

[cezar_varlan1]

If by any chance the Logs Were removed because of disk space and there is no "Run the following script before deleting" configured, where could i check that the action happened?

I am in a situation where i can only find logs going back less than a week. Considering that Indexes are configured to be deleted at 30 days and old log files when disk space is below 5 GB. Disk space is currently at 20 GB and there are no signs it was ever below this.  

Looking over audit log files they also stop at the same date the last log file stops. Checking $FWDIR/log/ also shows the last log as of 17th of January 2019.

Is this action logged by default somewhere? Should i configure this script to run and write in a file that it has deleted logs?