I am trying to understand the difference between log files and log indexes in terms of retention. We have a management server setup for Logging&Status, SmartEvent Viewer, and Correlation Unit. Log indexing is enabled and set to alert when space is below 20% and delete old files below 22%. Index files is set to delete older than 7 days. The folder /opt/CPrt-R80/log_indexes shows 7 days worth of index files. However the folder /opt/CPsuite-R80/fw1/log shows log files going back several months. We have cleared out log files from this folder before. With indexing set to delete older than 7 days we can only go back 7 days on reports but the log files themselves go back past 7 days. I am trying to understand the difference between the log files themselves and the indexing retention.
Thanks.
Hi Ed,
Thanks for the explanation. Indeed, there are differences between managing logs and log indexes.
On this pic, you can see both log storage settings and indexing retention management
Logs are stored as the files under $FWDIR/logs, this is a part of MGMT Log Server functionality. There is no automatic built-in mechanism to remove old log files. The only option you have is to start removing older logs when disk space utilization reaches a certain threshold. I have highlighted this part with the blue rectangle.
Log indexing is done by an indexing engine, and the indexes are stored to $RTDIR/log_indexes. You can set the maximum depth of indexing, which is important for Event Analysis performance and stability. The indexer has a built in retention option, and older indexes are routinely removed.
The main reason not to remove logs automatically is simple. You may want to keep your security logs to maintain ability of investigating past breaches and other security incidents. In some cases compliance regulations require keeping up to 2 years of logs available.
So to manage log retention I would advise you to run a cron task with a script that performs backup and removal of older logs. There are quite a few publicly available samples of such a script. On of example is here: Log Backup/Archive Script