Index Files option for R80.10

[Alan_Dressner1]

Hi everyone,

We have migrated over from R77.30 to R80.10. We were deleting files based on date under the manager's options under Logs, Storage.

That option is no longer available in R80.10. Anyone has any suggestions on how to delete files based on date in SmartConsole?

[Danny]

I recommend following Check Points Best Practice.

Alternatively you can create a new log file on scheduled times under 'Additional Logging' and have your own Script deleting older log files as a cronjob or via the 'Run the following script' option under 'Local Storage'.

[Tomer_Sole]

This is a limitation of R80.10 that will be resolved in our next releases.

[Jonathan_Pitt]

Hi Tomer

I'd like to re-start this thread.

You have stated here that its a limitation on R80.10 yet sk115872 states that this is a feature.

My situation is as follows:

MDS R80.10

I have a 2TB /var/log partition to which my domains / CMA's log. After a log file (switched at midnight) reaches a certain age it is moved to another partition (SAN) and a symlink is added to /var/log/mds_logs/<domain_name>/log directory to enabled older log files to be opened using the log viewer in SmartConsole.

I have found that although the log files themselves occupy only 276GB of disk spave the index files occupy 1.5TB:

<snip>
1.5T ./opt
276G ./mds_logs
<snip>
1.7T .

Having looked into how these files are being handled, documentation suggests that the index files are removed only when the log file is removed because they are linked. But presumably because of the symlink the indexes are never going to be removed. I don't want to lose the functionality of being able to keep aged logs easily available (for up to a year and I have the disk space to allow this) but it seems the only way I can reduce the indexing disk usage is to delete the index files manually / script. But then because the symlink still exist its my assumption that they will just be indexed again?

So what I really need is the ability to dictate that even if the log file exists, don't index files older than, for example, 30 days and actively delete index files that are over 30 days old. Which is kind of how it used to work pre-R80.x I believe?

I appreciate your time and response on this.

[Alan_Dressner1 · Accepted solution]

The log_policy_extended.C can be modified to retain or delete files based on disk space or length of time.

The file needs to be placed in /var/opt/CPmds-R80/conf

Here is an example that deletes the indexs that are older than 45 days and keeps the free disk space at 20%.

(
        :stop_logging_on_free_disk_space (true)
        :min_free_disk_space (100)
        :stop_free_disk_space_metrics (mbytes)
        :reject_connections (false)
        :alert_on_disk_space (true)
        :alert_free_disk_space (3000)
        :alert_free_disk_space_metrics (mbytes)
        :alert_type (alert)
        :log_switch_on_file_size (false)
        :scheduled_switch (false)
        :forward_logs (false)
        :log_delete_on_below (true)
        :log_delete_below_metrics (percent)
        :log_delete_below_value (20)
        :log_delete_on_run_script (false)
        :dlp_blob_delete_on_run_script (false)
        :dlp_blob_delete_above_value_percentage (20)
        :dlp_blob_delete_on_above (true)
        :packets_capture_reserved_disk_metrics (mbytes)
        :packets_capture_reserved_disk_size_MB (500)
        :dlp_blob_fetch_bulk_size (200)
        :dlp_blob_fetch_interval (5)
        :dlp_blob_retry_interval (180)
        :accept_syslog_mds (false)
        :daily_maintenance_at_least_script (dailyAtLeastLogMaintenance.sh)
        :daily_maintenance_script (dailyLogMaintenance.sh)
        :emergency_script (emergencyLogMaintenance.sh)
        :maintenance_type (daily)
        :log_keep_days_value (3650)
        :index_delete_older_than (true)
        :index_delete_older_than_value (45)
)

 if this file is used it will invalide any options choose in the GUI.

[Jonathan_Pitt]

Worked a treat thanks.

[Jonathan_Pitt]

Run into a problem with this now.

I have just noticed that since I implemented this the MDS has decided to start deleting logs based on "Cyclic Logging Mechanism":

Log file 2018-04-03_000000.adtlog has been deleted by the "Cyclic Logging" mechanism

My disk space usage is only 47% (thanks to the advice given on this thread) so I am at a loss as to the logic of why it has decided on a nightly basis to start deleting the oldest log on each of my CMA's i.e. it doesn't appear to be based on any age calculation nor relates to any config I have implemented.

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_splat-lv_current 1.9T 51G 1.8T 3% /
/dev/sda1 289M 78M 197M 29% /boot

tmpfs 126G 4.0K 126G 1% /dev/shm
/dev/mapper/vg_splat-lv_log 1.9T 855G 983G 47% /var/log
/dev/sdh1 6.5T 2.3T 3.9T 38% /san

This is my "extended" config that I assume overwrites any config in log_policy.C and log_policy_default.C as per SK's I have seen:

(
:stop_logging_on_free_disk_space (true)
:min_free_disk_space (100)
:stop_free_disk_space_metrics (mbytes)
:reject_connections (false)
:alert_on_disk_space (true)
:alert_free_disk_space (3000)
:alert_free_disk_space_metrics (mbytes)
:alert_type (alert)
:log_switch_on_file_size (false)
:scheduled_switch (false)
:forward_logs (false)
:log_delete_on_below (true)
:log_delete_below_metrics (mbytes)
:log_delete_below_value (5000)
:log_delete_on_run_script (false)
:dlp_blob_delete_on_run_script (false)
:dlp_blob_delete_above_value_percentage (20)
:dlp_blob_delete_on_above (true)
:packets_capture_reserved_disk_metrics (mbytes)
:packets_capture_reserved_disk_size_MB (500)
:dlp_blob_fetch_bulk_size (200)
:dlp_blob_fetch_interval (5)
:dlp_blob_retry_interval (180)
:daily_maintenance_at_least_script (dailyAtLeastLogMaintenance.sh)
:daily_maintenance_script (dailyLogMaintenance.sh)
:emergency_script (emergencyLogMaintenance.sh)
:maintenance_type (daily)
:log_keep_days_value (-1)
:index_delete_older_than (true)
:index_delete_older_than_value (45)
:maintenance_items (
: (
:type (firewallandvpn)
:delete_after (45)
)
: (
:type (audit)
:delete_after (45)
)
: (
:type (other)
:delete_after (45)
)
: (
:type (smartevent)
:delete_after (45)
)
)
)

# Maintenance Types (maintenance_type attribute)
# None - Unlimited until we have no space - then start deleting the last day of all indexes+logs
# Daily - Keep exact number of days according to configuration
# Daily at least - Try to keep number of days like the policy - When there is no disk space - maintenance routine tries to delete the indexes+logs so that it will work according to configuration
#
# Examples:
#
# 1. Keep Logs unlimited and delete indexes after 2 weeks
# (
# ...
# :maintenance_type (daily)
# :log_keep_days_value (-1)
# :maintenance_items (
# : (
# :type (firewallandvpn)
# :delete_after (14)
# )
# : (
# :type (audit)
# :delete_after (14)
# )
# : (
# :type (other)
# :delete_after (14)
# )
# : (
# :type (smartevent)
# :delete_after (14)
# )
# )
# )
#
# 2. Keep logs unlimited and delete indexes after 2 weeks if space is needed.
# (
# ...
# :maintenance_type (daily_at_least)
# :log_keep_days_value (-1)
# :maintenance_items (
# : (
# :type (firewallandvpn)
# :delete_after (14)
# )
# : (
# :type (audit)
# :delete_after (14)
# )
# : (
# :type (other)
# :delete_after (14)
# )
# : (
# :type (smartevent)
# :delete_after (14)
# )
# )
# )
#
# 3. SmartEvent - remove only firewall index and logs after 14 days - unlimited for all other
# (
# ...
# :maintenance_type (daily)
# :log_keep_days_value (14)
# :maintenance_items (
# : (
# :type (firewallandvpn)
# :delete_after (14)
# )
# )
# )
#

A bug? Or have I missed something? I seems to me I have followed the example configs to achieve what I want i.e. 45 days index, unlimited log retention.

[Martijn]

Hi All,

We have configured the log_policy_extended.C file and when we look at fwd.elg, we can see two types of information about the log policy.

The current showing the changes we made in the log_policy_extended.C. But the fwd.elg file is also showing a ' working set' which has other values for some items.

For example. We configured to keep the log files for 100 days and the log_keep_days_value is set to 100 which shows in the current policy file set in fwd.elg. But in the part of the ' 'working set' we can see the setting for log_keep_days_value is set to 3650!

What is the correct setting Check Point will use to clean up log files. We see old files are not removed, so it looks like the 3650 value is used.

Any ideas?

Regards, Martijn 

[Amir_Senn]

There could be a few reasons for that, this is probably illegal value of some sorts. If you changed the number of days to retain make sure the days to retain indexes is lower - :index_delete_older_than_value (<lower than 100>)

You can share the policy itself if you want me to take a look.

The link below is for the sk explaining usage of the extended policy: 

https://supportcenter.checkpoint.com/supportcenter/portal?action=portlets.SearchResultMainAction&eve...

[Martijn]

Amir,

Below the log_policy_extended.C file

[Expert@mds:0]# more log_policy_extended.C
(
:stop_logging_on_free_disk_space (true)
:min_free_disk_space (100)
:stop_free_disk_space_metrics (mbytes)
:reject_connections (false)
:alert_on_disk_space (true)
:alert_free_disk_space (3000)
:alert_free_disk_space_metrics (mbytes)
:alert_type (alert)
:log_switch_on_file_size (false)
:scheduled_switch (false)
:forward_logs (false)
:log_delete_on_below (true)
:log_delete_below_metrics (mbytes)
:log_delete_below_value (5000)
:log_delete_on_run_script (false)
:dlp_blob_delete_on_run_script (false)
:dlp_blob_delete_above_value_percentage (20)
:dlp_blob_delete_on_above (true)
:packets_capture_reserved_disk_metrics (mbytes)
:packets_capture_reserved_disk_size_MB (500)
:dlp_blob_fetch_bulk_size (200)
:dlp_blob_fetch_interval (5)
:dlp_blob_retry_interval (180)
:daily_maintenance_at_least_script (dailyAtLeastLogMaintenance.sh)
:daily_maintenance_script (dailyLogMaintenance.sh)
:emergency_script (emergencyLogMaintenance.sh)
:maintenance_type (daily)
:log_keep_days_value (100)
:index_delete_older_than (true)
:index_delete_older_than_value (14)
:maintenance_items (
: (
:type (firewallandvpn)
:delete_after (14)
)
: (
:type (audit)
:delete_after (95)
)
: (
:type (other)
:delete_after (14)
)
: (
:type (smartevent)
:delete_after (14)
)
: (
:type (other-smartlog)
:delete_after (14)
)
: (
:type (resources)
:delete_after (14)
)
: (
:type (files)
:delete_after (14)
)
)
)

And below the output from fwd.elg

Tue Sep 1 15:32:07 2020: FWD START
SetSignals: handle sighup for netflow
Warning : FwdIsIntegrityServer: fwobj_get_myown() failed
InitServers: Log asynch buffer size was initialized with size: 3145728
InitServers: Log buffer initialized with size: 64000
Warning : FwdIsIntegrityServer: fwobj_get_myown() failed
initSyslogServers: SysLog buffer initialized with size: 64000
initSyslogServers: failed to read gateway's settings
CPLogGetMyIp: fwobj_get_myown failed
FireWall-1 Daemon is running
syslog_run: syslogd was executed under pid 6180
load_host_log_policy: log_policy_extended.C is the current policy file

CHostLogPolicy: Attribute include_tcp_state_information not found
loaded set =
(
:stop_logging_on_free_disk_space (true)
:min_free_disk_space (100)
:stop_free_disk_space_metrics (mbytes)
:reject_connections (false)
:alert_on_disk_space (true)
:alert_free_disk_space (3000)
:alert_free_disk_space_metrics (mbytes)
:alert_type (alert)
:log_switch_on_file_size (false)
:scheduled_switch (false)
:forward_logs (false)
:log_delete_on_below (true)
:log_delete_below_metrics (mbytes)
:log_delete_below_value (5000)
:log_delete_on_run_script (false)
:dlp_blob_delete_on_run_script (false)
:dlp_blob_delete_above_value_percentage (20)
:dlp_blob_delete_on_above (true)
:packets_capture_reserved_disk_metrics (mbytes)
:packets_capture_reserved_disk_size_MB (500)
:dlp_blob_fetch_bulk_size (200)
:dlp_blob_fetch_interval (5)
:dlp_blob_retry_interval (180)
:daily_maintenance_at_least_script (dailyAtLeastLogMaintenance.sh)
:daily_maintenance_script (dailyLogMaintenance.sh)
:emergency_script (emergencyLogMaintenance.sh)
:maintenance_type (daily)
:log_keep_days_value (100)
:index_delete_older_than (true)
:index_delete_older_than_value (14)
:maintenance_items (
: (
:type (firewallandvpn)
:delete_after (14)
)
: (
:type (audit)
:delete_after (95)
)
: (
:type (other)
:delete_after (14)
)
: (
:type (smartevent)
:delete_after (14)
)
: (
:type (other-smartlog)
:delete_after (14)
)
: (
:type (resources)
:delete_after (14)
)
: (
:type (files)
:delete_after (14)
)
)
)

working set =
(
:stop_logging_on_free_disk_space (true)
:min_free_disk_space (100)
:stop_free_disk_space_metrics (mbytes)
:reject_connections (false)
:alert_on_disk_space (true)
:alert_free_disk_space (3000)
:alert_free_disk_space_metrics (mbytes)
:alert_type (alert)
:log_switch_on_file_size (false)
:scheduled_switch (false)
:forward_logs (false)
:log_delete_on_below (true)
:log_delete_below_metrics (mbytes)
:log_delete_below_value (5000)
:log_delete_on_run_script (false)
:dlp_blob_delete_on_run_script (false)
:dlp_blob_delete_above_value_percentage (20)
:dlp_blob_delete_on_above (true)
:packets_capture_reserved_disk_metrics (mbytes)
:packets_capture_reserved_disk_size_MB (500)
:dlp_blob_fetch_bulk_size (200)
:dlp_blob_fetch_interval (5)
:dlp_blob_retry_interval (180)
:daily_maintenance_at_least_script (dailyAtLeastLogMaintenance.sh)
:daily_maintenance_script (dailyLogMaintenance.sh)
:emergency_script (emergencyLogMaintenance.sh)
:maintenance_type (daily)
:log_keep_days_value (3650)
:index_delete_older_than_value (14)
:index_delete_older_than (true)
:maintenance_items (
: (
:type (firewallandvpn)
:delete_after (14)
)
: (
:type (audit)
:delete_after (95)
)
: (
:type (other)
:delete_after (14)
)
: (
:type (smartevent)
:delete_after (14)
)
: (
:type (other-smartlog)
:delete_after (14)
)
: (
:type (resources)
:delete_after (14)
)
: (
:type (files)
:delete_after (14)
)
)
)

InitBlobFetcher Warning: Failed to get own object
start_diskspace_check_schedule_event started successfully
start_daily_log_switch_schedule_event started successfully
start_log_switch_schedule_event started successfully
start_log_forward_schedule_event: no log_forward target.
Log forward scheduling not started
start_cyclic_logging_event started successfully
build_opsec_entity: receiving the virtual IP address for MDS

fw_init_lea_server: LEA server initialized
Unable to open '/dev/fw0': No such file or directory
Set operation failed: failed to get parameter enable_netflow
Unable to open '/dev/fw0': No such file or directory
Set operation failed: failed to get parameter enable_netflow
CPLogGetMyIp: fwobj_get_myown failed

There are two values of log_keep_days_value in the output of fwd.elg. The 100 we have configured, but also 3650 which is shown in the working set.

Which value is used?

Regards, Martijn

[Amir_Senn]

Working set will be used. I used the same definitions as you and it changed it properly.

Which version are you using?

[Martijn]

Amir,

We have two MDM servers at this customer (two different networks) and one is at R80.30 take 111 and the other is at R80.30 take 215. Both have the same issue.

We checked the index files and it looks they are deleted. The oldest one is from August 19 which is about 14 days.

Can you explain why I still can see old log entries in SmartLog (May 2nd 2020) even when old index files are deleted?

Regards, Martijn

[Martijn]

Amir,

I found sk123532 telling the following:

" log_keep_on_days and log_keep_days_value are no longer supported."

Is this relevant here?

Regards, Martijn

[Amir_Senn · Accepted solution]

Yes. Fixed in R80.40. MDS level doesn't have traffic logs so the logs file are very small, but domain logs will still be deleted according to the information there.