cancel
Showing results for 
Search instead for 
Did you mean: 
Post a Question

Index Files option for R80.10

Jump to solution

Hi everyone,

We have migrated over from R77.30 to R80.10. We were deleting files based on date under the manager's options under Logs, Storage.

That option is no longer available in R80.10. Anyone has any suggestions on how to delete files based on date in SmartConsole?

1 Solution

Accepted Solutions

Re: Index Files option for R80.10

Jump to solution

The log_policy_extended.C can be modified to retain or delete files based on disk space or length of time.

The file needs to be placed in /var/opt/CPmds-R80/conf

Here is an example that deletes the indexs that are older than 45 days and keeps the free disk space at 20%.

(
        :stop_logging_on_free_disk_space (true)
        :min_free_disk_space (100)
        :stop_free_disk_space_metrics (mbytes)
        :reject_connections (false)
        :alert_on_disk_space (true)
        :alert_free_disk_space (3000)
        :alert_free_disk_space_metrics (mbytes)
        :alert_type (alert)
        :log_switch_on_file_size (false)
        :scheduled_switch (false)
        :forward_logs (false)
        :log_delete_on_below (true)
        :log_delete_below_metrics (percent)
        :log_delete_below_value (20)
        :log_delete_on_run_script (false)
        :dlp_blob_delete_on_run_script (false)
        :dlp_blob_delete_above_value_percentage (20)
        :dlp_blob_delete_on_above (true)
        :packets_capture_reserved_disk_metrics (mbytes)
        :packets_capture_reserved_disk_size_MB (500)
        :dlp_blob_fetch_bulk_size (200)
        :dlp_blob_fetch_interval (5)
        :dlp_blob_retry_interval (180)
        :accept_syslog_mds (false)
        :daily_maintenance_at_least_script (dailyAtLeastLogMaintenance.sh)
        :daily_maintenance_script (dailyLogMaintenance.sh)
        :emergency_script (emergencyLogMaintenance.sh)
        :maintenance_type (daily)
        :log_keep_days_value (3650)
        :index_delete_older_than (true)
        :index_delete_older_than_value (45)
)

 if this file is used it will invalide any options choose in the GUI.

0 Kudos
6 Replies
Danny
Pearl

Re: Index Files option for R80.10

Jump to solution

I recommend following Check Points Best Practice.

Alternatively you can create a new log file on scheduled times under 'Additional Logging' and have your own Script deleting older log files as a cronjob or via the 'Run the following script' option under 'Local Storage'.

0 Kudos

Re: Index Files option for R80.10

Jump to solution

This is a limitation of R80.10 that will be resolved in our next releases.

0 Kudos

Re: Index Files option for R80.10

Jump to solution

Hi Tomer

I'd like to re-start this thread.

You have stated here that its a limitation on R80.10 yet sk115872 states that this is a feature.

My situation is as follows:

MDS R80.10

I have a 2TB /var/log partition to which my domains / CMA's log. After a log file (switched at midnight) reaches a certain age it is moved to another partition (SAN) and a symlink is added to /var/log/mds_logs/<domain_name>/log directory to enabled older log files to be opened using the log viewer in SmartConsole.

I have found that although the log files themselves occupy only 276GB of disk spave the index files occupy 1.5TB:

<snip>
1.5T ./opt
276G ./mds_logs
<snip>
1.7T .

Having looked into how these files are being handled, documentation suggests that the index files are removed only when the log file is removed because they are linked. But presumably because of the symlink the indexes are never going to be removed. I don't want to lose the functionality of being able to keep aged logs easily available (for up to a year and I have the disk space to allow this) but it seems the only way I can reduce the indexing disk usage is to delete the index files manually / script. But then because the symlink still exist its my assumption that they will just be indexed again?

So what I really need is the ability to dictate that even if the log file exists, don't index files older than, for example, 30 days and actively delete index files that are over 30 days old. Which is kind of how it used to work pre-R80.x I believe?

I appreciate your time and response on this.

0 Kudos

Re: Index Files option for R80.10

Jump to solution

The log_policy_extended.C can be modified to retain or delete files based on disk space or length of time.

The file needs to be placed in /var/opt/CPmds-R80/conf

Here is an example that deletes the indexs that are older than 45 days and keeps the free disk space at 20%.

(
        :stop_logging_on_free_disk_space (true)
        :min_free_disk_space (100)
        :stop_free_disk_space_metrics (mbytes)
        :reject_connections (false)
        :alert_on_disk_space (true)
        :alert_free_disk_space (3000)
        :alert_free_disk_space_metrics (mbytes)
        :alert_type (alert)
        :log_switch_on_file_size (false)
        :scheduled_switch (false)
        :forward_logs (false)
        :log_delete_on_below (true)
        :log_delete_below_metrics (percent)
        :log_delete_below_value (20)
        :log_delete_on_run_script (false)
        :dlp_blob_delete_on_run_script (false)
        :dlp_blob_delete_above_value_percentage (20)
        :dlp_blob_delete_on_above (true)
        :packets_capture_reserved_disk_metrics (mbytes)
        :packets_capture_reserved_disk_size_MB (500)
        :dlp_blob_fetch_bulk_size (200)
        :dlp_blob_fetch_interval (5)
        :dlp_blob_retry_interval (180)
        :accept_syslog_mds (false)
        :daily_maintenance_at_least_script (dailyAtLeastLogMaintenance.sh)
        :daily_maintenance_script (dailyLogMaintenance.sh)
        :emergency_script (emergencyLogMaintenance.sh)
        :maintenance_type (daily)
        :log_keep_days_value (3650)
        :index_delete_older_than (true)
        :index_delete_older_than_value (45)
)

 if this file is used it will invalide any options choose in the GUI.

0 Kudos

Re: Index Files option for R80.10

Jump to solution

Worked a treat thanks.

0 Kudos

Re: Index Files option for R80.10

Jump to solution

Run into a problem with this now.

I have just noticed that since I implemented this the MDS has decided to start deleting logs based on "Cyclic Logging Mechanism":

Log file 2018-04-03_000000.adtlog has been deleted by the "Cyclic Logging" mechanism

My disk space usage is only 47% (thanks to the advice given on this thread) so I am at a loss as to the logic of why it has decided on a nightly basis to start deleting the oldest log on each of my CMA's i.e. it doesn't appear to be based on any age calculation nor relates to any config I have implemented.

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_splat-lv_current 1.9T 51G 1.8T 3% /
/dev/sda1 289M 78M 197M 29% /boot

tmpfs 126G 4.0K 126G 1% /dev/shm
/dev/mapper/vg_splat-lv_log 1.9T 855G 983G 47% /var/log
/dev/sdh1 6.5T 2.3T 3.9T 38% /san

This is my "extended" config that I assume overwrites any config in log_policy.C and log_policy_default.C as per SK's I have seen:

(
:stop_logging_on_free_disk_space (true)
:min_free_disk_space (100)
:stop_free_disk_space_metrics (mbytes)
:reject_connections (false)
:alert_on_disk_space (true)
:alert_free_disk_space (3000)
:alert_free_disk_space_metrics (mbytes)
:alert_type (alert)
:log_switch_on_file_size (false)
:scheduled_switch (false)
:forward_logs (false)
:log_delete_on_below (true)
:log_delete_below_metrics (mbytes)
:log_delete_below_value (5000)
:log_delete_on_run_script (false)
:dlp_blob_delete_on_run_script (false)
:dlp_blob_delete_above_value_percentage (20)
:dlp_blob_delete_on_above (true)
:packets_capture_reserved_disk_metrics (mbytes)
:packets_capture_reserved_disk_size_MB (500)
:dlp_blob_fetch_bulk_size (200)
:dlp_blob_fetch_interval (5)
:dlp_blob_retry_interval (180)
:daily_maintenance_at_least_script (dailyAtLeastLogMaintenance.sh)
:daily_maintenance_script (dailyLogMaintenance.sh)
:emergency_script (emergencyLogMaintenance.sh)
:maintenance_type (daily)
:log_keep_days_value (-1)
:index_delete_older_than (true)
:index_delete_older_than_value (45)
:maintenance_items (
: (
:type (firewallandvpn)
:delete_after (45)
)
: (
:type (audit)
:delete_after (45)
)
: (
:type (other)
:delete_after (45)
)
: (
:type (smartevent)
:delete_after (45)
)
)
)

# Maintenance Types (maintenance_type attribute)
# None - Unlimited until we have no space - then start deleting the last day of all indexes+logs
# Daily - Keep exact number of days according to configuration
# Daily at least - Try to keep number of days like the policy - When there is no disk space - maintenance routine tries to delete the indexes+logs so that it will work according to configuration
#
# Examples:
#
# 1. Keep Logs unlimited and delete indexes after 2 weeks
# (
# ...
# :maintenance_type (daily)
# :log_keep_days_value (-1)
# :maintenance_items (
# : (
# :type (firewallandvpn)
# :delete_after (14)
# )
# : (
# :type (audit)
# :delete_after (14)
# )
# : (
# :type (other)
# :delete_after (14)
# )
# : (
# :type (smartevent)
# :delete_after (14)
# )
# )
# )
#
# 2. Keep logs unlimited and delete indexes after 2 weeks if space is needed.
# (
# ...
# :maintenance_type (daily_at_least)
# :log_keep_days_value (-1)
# :maintenance_items (
# : (
# :type (firewallandvpn)
# :delete_after (14)
# )
# : (
# :type (audit)
# :delete_after (14)
# )
# : (
# :type (other)
# :delete_after (14)
# )
# : (
# :type (smartevent)
# :delete_after (14)
# )
# )
# )
#
# 3. SmartEvent - remove only firewall index and logs after 14 days - unlimited for all other
# (
# ...
# :maintenance_type (daily)
# :log_keep_days_value (14)
# :maintenance_items (
# : (
# :type (firewallandvpn)
# :delete_after (14)
# )
# )
# )
#

A bug? Or have I missed something? I seems to me I have followed the example configs to achieve what I want i.e. 45 days index, unlimited log retention.