Is it Safe to Delete log_indexes Folder When Not Using SmartEvent?

[rozkie20]

Hi everyone,

We are using centrally manage with separate SMS and Gateway.

We’ve recently encountered an issue where our disk is full. Upon investigation, we noticed that both of the following directories are consuming significant space:

• /var/opt/CPsuite-R81.20/fw1/log/

• /var/opt/CPrt-R81.20/log_indexes/

After some research, we found that the log_indexes directory is created when SmartEvent and SmartEvent Correlation are enabled. However, in our current environment, we are not using SmartEvent, as we don’t have a license for it. Additionally, we already utilize a separate SIEM solution for dashboarding and analysis.

Is it safe to delete subfolders inside /var/opt/CPrt-R81.20/log_indexes/ such as firewallandvpn_* or audit_* to free up disk space?

If anyone has experience with this, appreciate your feedback.

[the_rock]

Yup, 100% safe. But, having said that, just in case, generate backup and/or snapshot beforehand.

Andy

[rozkie20]

Hi Andy,

We have disabled SmartEvent but we still see SMS generate log (new indexes on May although we disable SmartEvent for a long time). Can we stop the system generate log on directory log_index? 

[the_rock]

I would reboot it if you can.

Andy

[PhoneBoy]

Log indexes are used by SmartEvent, but they are also used without it.

[the_rock]

That is definitely true, though I recall TAC telling me twice before they are safe to remove to save space, but probably more prudent to check option for it in the log setting on the object itself in smart console.

Andy

[the_rock]

@rozkie20 

Just found email from TAC about it, shows it was folders starting with other_*

But, do NOT delete folders themselves, just the context. I would still double check via TAC case, but thats the note I found.

Andy

[Blason_R]

It is recommended to execute smartlogstop, followed by the deletion of the sub-folders $RTDIR/log_indexes, and subsequently perform smartlogstart. This procedure ensures that Solr processes are halted, allowing for the safe deletion of data. If you prefer not to generate log_indexes at all, you should click on the management dashboard icon and disable the log indexing option found under Log settings.

[the_rock]

Good to know!

[Martin_Raska]

Guys,

can some one explain differencies and why its used that way for those two directories?

/var/opt/CPsuite-R82/fw1

/opt/CPsuite-R82/fw1/

[Blason_R]

Those are symlinked to /var/log/opt - The actual directories are created under /opt however since logs are being created huge in size firewalls are formatted with two lvms one with lv_current which is OS and lv_log where log files are stored. So all logs are stored under /var/log/opt/

[Amir_Senn · Accepted solution]

@rozkie20 @the_rock @PhoneBoy @Martin_Raska @Blason_R 

Lets start with not deleting the files, this will not cause major issues and you can always re-index the logs but I will explain everything and show best practices.

Here's a crash course:

Log indexing is happening without any relation to SmartEvent. It's enabled by default on every  dedicated log server and primary management server unless it's functioning as gateway as well or on weak machines.

Indexing is not only much faster, it allows to query multiple log files and query multiple log servers with custom timeframes. Not working with indexes will force you to work with log files.

This is enabled here:

The storage issue is common when not defining log policy. By default we have only emergency cleanup which is triggered by default when storage reaches 5 GB it will start deleting oldest logs and indexes. This was determined quite a while ago and if you do an upgrade it will retain this policy even if the default policy will be changed for newer installations.

Storing logs long enough will fill any amount of storage, just depends how long it takes as it just keeping piling up.

Each type of log is indexed into a different index core: firewallandvpn (~85-98% of your logs will be found here), audit (anything in the audit logs), other, smartevent, other-smartlog, resources (extended logging on Application Control and URL Filtering) and files (extended logging on Content Awareness).

What SmartEvent does (amongst other features) is to extend the indexing on existing indexes in order to create informational views and reports using SmartView. You can check the amount of storage it actually takes using "du -s $RTDIR/log_indexes/smart* | sort -n" .

The best way to control this is by defining daily log retention (on every server functioning as log server/SmartEvent):

Audit logs (since they're the drop in the sea and could be useful) are exempted from this.

Deleting index files will drop the relevant logs from query so for guidance I will ask the following:

How far back do I need to search for logs (this is the desired amount of time to keep indexes, maybe give a little extra for safety).

How long do I need to keep my logs? Recommended value should be equal to <days> - <index retention set> + safety.

You finish it up by installing database on all management servers (SmartEvent included).

Installing database is the equivalent of installing policy on gateways, so any changes you make for such servers - follow with this action.

It's possible that if you disabled SmartEvent but never installed database that server still thinks it needs to do it. Relevant processes on SmartEvent are CPSEMD and CPSEAD. If you can spot them on "cpwd_admin list" - you're running a SmartEvent server.

If somehow this still happens, you can use the SK below and define extended log policy that can define different time to retain specific index cores.

sk117317 (How to configure log / log indexes maintenance policy for Global SmartEvent Server / Log Server and MDS / MLM)

[the_rock]

Wow, EXCELLENT explanation @Amir_Senn 

[Amir_Senn]

Got some info about other/other-smartlog, so index core in general:

types of indexes- most lost (~95%) are connection logs that will be indexed in the firewallandvpn index.  most of the other logs will be stored in other-smartlog index (for non SME machine) or other index (in SME machine)