How to store logs? Focus on time range instead of disk space.

[Oleg_Khomutinin]

Dear Check Point community.

I have requirements to store all logs no more than 3 month on log server, after this period all logs should automatically deleted. 

My infrastructure is distributed level and we are use 3 different servers (based on VMware solution):

Firewall 80.10

Management 80.10

Logging server (500 Gb, 16384 MB, 8 CPUc). 80.10

Logging server is responsible to store on their site all logs, but when I do configuration I can see that solution which provided by CP is focus on disk space configuration (see the pic). Maybe somebody already impenitent the same solution, can you please give me advice how is better to implement this solution? Thank you.

[Kaspars_Zibarts]

I'm guessing it's a simple management server, not MDS.

Easiest in past at least always has been running script, since log files are in $FWDIR/log directory, you could do

find $FWDIR/log//20*log* -mtime +90 |  xargs rm -f

for example that would filter all files that start with 20 (as in 2018-***) and containing *log* basically filtering all relevant log files, audit log files and their pointers that are older than 90 days

for example

./2018-03-05_000000.adtlog
./2018-03-05_000000.adtlogaccount_ptr
./2018-03-05_000000.adtloginitial_ptr
./2018-03-05_000000.adtlogptr
./2018-03-05_000000.log
./2018-03-05_000000.logaccount_ptr
./2018-03-05_000000.loginitial_ptr
./2018-03-05_000000.logptr
./2018-03-06_000000.adtlogaccount_ptr
./2018-03-06_000000.logaccount_ptr

Just add usual wrappers and your script is done

#!/bin/bash
. /opt/CPshared/5.0/tmp/.CPprofile.sh
find $FWDIR/log/20*log* -mtime +90 |  xargs rm -f‍‍‍‍‍‍

save it as a file somewhere then add scheduled daily job via Gaia webui or CLI, for example

add cron job log_cleanup command "/home/admin/log-cleanup.sh" recurrence daily time 04:00

[AlekseiShelepov]

Although this script is easy, it would be much better to have settings for logs timeframe embedded into SmartConsole. I don't see any reason not to have it in there. With cron you need to look for that in a different place, somebody might forget about that during migration, etc.

There was a setting "Do not delete log files from the last ... days" in R77.30. And add one more setting "Start deleting log files older than ... days" and it would be pretty good.

Previously I did something like this:

I suspect there might be a setting for that in GuiDBedit for R80.10...

[PhoneBoy]

It's not as if we completely forgot about it

Some perfunctory poking around suggests there are some vestiges of those settings still there in objects_5_0.C

Whether they are honored or not in R80+ is a different matter entirely.

[AlekseiShelepov]

Yes, not completely

Maybe we can suggest to add "Do not delete log files from the last ... days" setting back to the SmartConsole in the upcoming R80.20? Of course, there might be some new case when it would make things worse in R80.X versions compared to R77.30, but I don't see it for now.

And as another suggestion, can we ask to add a setting to "Start deleting log files older than ... days", not based on free space, but on a time frame?

Hard drives becoming bigger and bigger these days, and we can store more and more logs to have full history over a year. But there are also some cases when we don't want to store "as many logs as we can till we fill up all free space". For example, when we have one CMA that needs to store logs only for 1 month, but five other CMAs need to store logs for 6 months. I would not like to store logs for these CMAs for 2-3 years (especially for the first one) untill they fill 80% of a Smart-1 appliance with 12TB (as I remember). More logs usually means more time required for upgrade, migrate, etc.

There is a possibility to use scripts, of course, and do whatever admin wants with logs. But wouldn't it be better to simply configure it in "single pane of glass" SmartConsole? (I thought it's Palo Alto's marketing term  )

[Dror_Aharony]

Hi All,

Regarding the Log Retention by Days (time-range) option you're requesting:

We're currently working on it, for MDS servers as well with an ability to configure a different setting for each domain.

Hopefully it'll be available in the next upcoming version or two.

Stay tuned...

[Kaspars_Zibarts]

Fully agree - best would be to use standard apps/options, but failed to find one in R80.10. Thought it was strange

[Oleg_Khomutinin]

Hello everybody, maybe somebody mention if that possible to change configuration for deleting old logs which based on time range via GuiDBedit.exe?

[AlekseiShelepov]

I found two parameters in GuiDBedit for keeping logs for ... days:

Network Objects → network_objects → host_ckp/vs_cluster_netobj:

log_keep_days_value

log_keep_on_days

It should return back the setting that was possible in R77.30. But I didn't test it, that's up to you  With it you can do a similar to R77.30 thing:

So, start deleting logs pretty early, but save last 90 days of them. And then just keep looking ar free space on HDD. Because if 90 days of logs will fill up all space, they will not be automatically deleted.

[PhoneBoy]

If anyone is brave enough to try editing these options and test it, please report

Agree these options should be in the UI, especially if they work.

[Vladimir]

So long as we are talking about improvements to the log handling, I may suggest some changes to the log forwarding as well:

Have an option (checkbox) of retaining logs locally according to the improved (i.e. non existent/retired parameters in Log/Storage) shown above.

Information archiving is an important subject and Check Point should put a bit more effort in its implementation.

Another thing I would suggest is implementation of the automated/scheduled log transfer to the external SFTP/SCP servers configurable either through SmartConsole or WebUI with the ability to import the authentication keys for the target server.

[Oleg_Khomutinin]

Hello everybody I'm focusing on Aleksei Shelepov solution (спасибо Алесей), at the moment just need to clean up all configuration which mention Licences, Blades, Configuration log transfer such as: FW->MGM->Log server (here more detailed info:Best Practices - Configuration of logging from Security Gateway to Security Management Server / Log ... ). After testing I will to come back this topic and let you know how is going.