Traffic logs only go back about 25-days, best way to improve this?

[Parabol]

Hi all, I wonder if you could advise, we currently have Active and Standby management servers, as well as a SmartEvent server.

Logs are currently sent to the ACTIVE only, and as such its /var/log partition has become 95% full. This is reducing our historical log window, currently we can only check logs back as far as about 25-days. We wish to increase this logging window as much as possible without increasing the storage if possible.

ACTIVE;

Filesystem                       Size  Used Avail Use% Mounted on

/dev/mapper/vg_splat-lv_current   80G   51G   30G  63% /

/dev/sda1                        282M   47M  221M  18% /boot

tmpfs                             32G   35M   32G   1% /dev/shm

/dev/mapper/vg_splat-lv_log      1.9T  1.8T  104G  95% /var/log

cgroup                            32G  4.0K   32G   1% /sys/fs/cgroup

The STANDBY server has no logs sent to it, its used purely as a backup, and has 1.8TB available.

SMARTEVENT server has 1.5TB available.

I feel like there could be something we can do with these to have a more optimal logging solution?

I do see the option to distribute logs between log servers, I am not sure how this would work with viewing the logs in smartconsole, presumably you would you need to log into both management servers separately to view the logs stored on them, losing that single centralized log view in smartconsole?

One final point, 1.8TB for roughly 25 days of traffic logs - does this sound like a lot to you guys? Another option I guess is to try and reduce unnecessary noise, doing things like disabling accounting, and logging on unnecessary rules. 

I appreciate any tips or advise on this logging topic you could offer! 

Thanks

[Gaurav_Pandya]

Hi Parabol,

Please go through sk98126 - Best Practices - Configuration of logging. 

Additionally, I would suggest verifying log settings from admin guide,

https://sc1.checkpoint.com/documents/R81/WebAdminGuides/EN/CP_R81_LoggingAndMonitoring_AdminGuide/To...

[Martijn]

Hi,

You can also change the logging from 'per Connection' to 'per Session'. This should reduce the amount of logging.
Keep in mind, older versions will not show NAT information is 'per Session' is enabled. This is solved in newer versions. What version are you running?

But it would not hurt to review your rules and check which are usefull to log and which are not.

How long do you keep the index files? Or is Retention not enabled? 

Regards,
Martijn

[Parabol]

Hi Martjin, thanks for your help, this is our current config:

 So even though we have 45 and 180 days configured, we only can see back about 25 days in the log viewer on smartconsole.

Interesting point with the 'per session' logging, everything is on R82, is there any disadvantage to enabling this logging, any less visibility? 

 

[Martijn]

Hi,

To my knowledge there is not a real disadvantage with 'per Session' logging, but I would advice enabling this on some rules and check to see if your are missing important data.

Default indexing is 14 days. Is 45 days a 'must have' or a 'nice to have'? Do administrators check logs from 40 days ago?
Index files can take up a lot of storage.

Have you considered using Log Exporter?

Martijn

[Parabol]

To be honest it is rare admins check logs more than 2 weeks back. 

But we recently had a security incident where we needed to obtain logs from 30+ days ago, but of course our logs only extended to 25-days which caused some issues.

I am not sure if there is some way to export the logs into a sort of "cold storage", where we could import them back in for rare scenarios like this, and view in smartconsole? Maybe something like this would be more effective and storage efficient. I am not sure how easy it would be to export and import them back into the management server/smartconsole

[Martijn]

Hi,

You can do the following.

Set the Index Retention to 14 days. You can check the logs via SmartConsole (SmartLog) for investigation and troubleshooting.
This should free up some disk space.

If you need to investigate older logs, just use good old SmartView Tracker:

C:\Program Files (x86)\CheckPoint\SmartConsole\R82\PROGRAM\CPlgv.exe

The raw log files are still on the system and can be viewed with SmartView Tracker. You can only open one log file at once and if multiple filer are generated per day, it may take some time to find the correct one with the data you need.  So it is not ideal, but should work in the rare moments you need them to investigate a security incident.

You can always copy log-files back into the $FWDIR/log directory and re-index those files. May take some time if the log files are big.

We can give you tips and tricks to free up disk space, but in the end the log retention policy of your company dictates what kind of hardware (disk space) is needed.

Good luck.
Martijn 

[the_rock]

All makes sense Martijn.

[Parabol]

Thanks MartIjn, I tried with the SmartView tracker, when going "Open log file" it does appear to only list as far as 25th Dec. Do you know if there would be a way to search further back? Thanks again

 

[Martijn]

No, because you mentioned you can only see logs for about 25 days. Today is January 15th so 25 days ago would be around December 25th. All raw log files before December 25th are removed because if disk space is below 5GB old log files are deleted.

Set you index to a lower setting and maybe this helps free up disk space for additional days of raw log files.

Martijn

[Parabol]

Ahh so changing my indexing from 45 to 14 days for example will give us 14 days of "live logging" I suppose you could say, accessed in the SmartConsole. So we lose capacity here.. BUT, in doing so, it gives more room for the RAW log files (which smart tracker can read) which will give us a longer historical window this way.

I guess the SmartConsole "live" logging is bigger, more taxing on storage which would make sense. And we really don't need 45 days of this kind of logging.

I think I am understanding now, thanks

[the_rock]

Hey @Parabol 

Just to make sure Im not misunderstanding something, are you saying when you search for logs, you can only find ones 25 days back, nothing further? If so, thats definitely not normal. That setting you gave looks right to me. 

This is setting TAC gave to one of my customers, which is what I set in the lab as well.

 

[Parabol]

Hi @the_rock that is correct, in SmartConsole, Logs & Events the furthest back we can see is roughly 25-days. 

[the_rock]

Just wondering...do you have any idea when this happened originally? Or you just noticed it recently? I saw your response about searches going back...you can check what you see if $FWDIR/log directory.

From my lab (just a portion)

[Expert@CP-MANAGEMENT:0]# cd $FWDIR/log
[Expert@CP-MANAGEMENT:0]# ls -lh
total 3.6G
-rw-rw-r-- 1 admin config 53K Mar 25 2025 2025-03-25_000000.adtlog
-rw-rw-r-- 1 admin config 80 Mar 24 2025 2025-03-25_000000.adtlogaccount_ptr
-rw-rw-r-- 1 admin config 692 Mar 24 2025 2025-03-25_000000.adtloginitial_ptr
-rw-rw-r-- 1 admin config 1.4K Mar 24 2025 2025-03-25_000000.adtlogptr
-rw-rw---- 1 admin root 136K Mar 26 2025 2025-03-26_000000.adtlog
-rw-rw---- 1 admin root 80 Mar 25 2025 2025-03-26_000000.adtlogaccount_ptr
-rw-rw---- 1 admin root 1012 Mar 25 2025 2025-03-26_000000.adtloginitial_ptr
-rw-rw---- 1 admin root 2.4K Mar 25 2025 2025-03-26_000000.adtlogptr
-rw-rw---- 1 admin root 89K Mar 27 2025 2025-03-27_000000.adtlog
-rw-rw---- 1 admin root 80 Mar 26 2025 2025-03-27_000000.adtlogaccount_ptr
-rw-rw---- 1 admin root 884 Mar 26 2025 2025-03-27_000000.adtloginitial_ptr
-rw-rw---- 1 admin root 2.0K Mar 26 2025 2025-03-27_000000.adtlogptr
-rw-rw---- 1 admin root 33K Mar 28 2025 2025-03-28_000000.adtlog
-rw-rw---- 1 admin root 80 Mar 27 2025 2025-03-28_000000.adtlogaccount_ptr
-rw-rw---- 1 admin root 444 Mar 27 2025 2025-03-28_000000.adtloginitial_ptr
-rw-rw---- 1 admin root 856 Mar 27 2025 2025-03-28_000000.adtlogptr
-rw-rw---- 1 admin root 28K Mar 29 2025 2025-03-29_000000.adtlog
-rw-rw---- 1 admin root 80 Mar 28 2025 2025-03-29_000000.adtlogaccount_ptr
-rw-rw---- 1 admin root 408 Mar 28 2025 2025-03-29_000000.adtloginitial_ptr
-rw-rw---- 1 admin root 784 Mar 28 2025 2025-03-29_000000.adtlogptr
-rw-rw---- 1 admin root 25K Mar 30 2025 2025-03-30_000000.adtlog
-rw-rw---- 1 admin root 80 Mar 29 2025 2025-03-30_000000.adtlogaccount_ptr
-rw-rw---- 1 admin root 340 Mar 29 2025 2025-03-30_000000.adtloginitial_ptr
-rw-rw---- 1 admin root 648 Mar 29 2025 2025-03-30_000000.adtlogptr
-rw-rw---- 1 admin root 37K Mar 31 2025 2025-03-31_000000.adtlog

[Parabol]

To be honest it has always been around 30-ish days of logs, I believe as we've upgraded it's chipped away at this. The latest to R82, when I noticed it reduced to about 25-days. I can only assume the upgrade takes up more storage than the previous version.

Interesting there is files from July here still.

[Expert@sms]# ls -lh | less
total 439G
-rw-rw---- 1 admin root 561K Jul 1 2025 2021-12-01_000000.adtlog
-rw-rw---- 1 admin root 80 Jul 1 2025 2021-12-01_000000.adtlogaccount_ptr
-rw-rw---- 1 admin root 1.1K Jul 1 2025 2021-12-01_000000.adtloginitial_ptr
-rw-rw---- 1 admin root 2.8K Jul 1 2025 2021-12-01_000000.adtlogptr
-rw-rw---- 1 admin root 214K Jul 1 2025 2021-12-02_000000.adtlog
-rw-rw---- 1 admin root 80 Jul 1 2025 2021-12-02_000000.adtlogaccount_ptr
-rw-rw---- 1 admin root 576 Jul 1 2025 2021-12-02_000000.adtloginitial_ptr
-rw-rw---- 1 admin root 1.7K Jul 1 2025 2021-12-02_000000.adtlogptr
-rw-rw---- 1 admin root 395K Jul 1 2025 2021-12-03_000000.adtlog
-rw-rw---- 1 admin root 80 Jul 1 2025 2021-12-03_000000.adtlogaccount_ptr
-rw-rw---- 1 admin root 856 Jul 1 2025 2021-12-03_000000.adtloginitial_ptr
-rw-rw---- 1 admin root 2.3K Jul 1 2025 2021-12-03_000000.adtlogptr
-rw-rw---- 1 admin root 204K Jul 1 2025 2021-12-04_000000.adtlog

[the_rock]

Mind post output of df -h?

[Parabol]

Sure:

[Expert@sms]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_splat-lv_current 80G 51G 30G 63% /
/dev/sda1 282M 47M 221M 18% /boot
tmpfs 32G 41M 32G 1% /dev/shm
/dev/mapper/vg_splat-lv_log 1.9T 1.8T 130G 94% /var/log
cgroup 32G 4.0K 32G 1% /sys/fs/cgroup

[the_rock]

So though /var/log is 94% full, your screenshot shows when its below 5%, start deleting old log files. Honestly, I would set it to what I mentioned in my post, to 14 days and 3650.

[AkosBakos]

Hi,

This is always a sensitive topic. I suggest you the followings:

• reduce the amount of logs if possible
  • turn off logging where is not neccessarry (eg.: NTP and DNS, syslog)
• Divide the log sources between the two SMS servers
  • If you have clusters, security groups, set for eg.: every second one to log to secondary SMS
  • to determine the receiving log amount on the Primary SMS: #cpstat mg -f log_server
  • Try to balance the logs 50%-50% between the primary and secondary SMS

Are they physical SMS servers? If not, and they are virtual, you can add more disk.

Akos 

[the_rock]

Definitely all super valid points brother.

[Parabol]

Thanks @AkosBakos for sure we have a lot of unnecessary logging we will look into these things. I see our catch-all deny rules at the bottom of policies generate a good portion of the logs, especially on our perimeter. Yes virtual servers so more disk space could be possible.

[AkosBakos]

Hi,

A more professional solution is the Dynamic Log Distribution

Steps:

In R81 and lower versions, each Log Server received a copy of every log. If one Log Server was disconnected, the Security Gateway / Cluster connected to the backup Log Server and sent it a copy of every log.

Starting in R81.10, with Dynamic Log Distribution, you can configure the Security Gateway / Cluster to distribute its logs between the active Log Servers.

If all the primary Log Servers are disconnected, logs are distributed between backup Log Servers.

If no Log Servers are connected, the Security Gateway / Cluster writes the logs locally.

Use Case - Log distribution reduces:

• The high utilization of resources on the Log Servers.

• The log traffic on each network that connects the Security Gateway / Cluster to its Log Server.

Configuring log distribution between multiple Log Servers

1. Connect with SmartConsole to the Management Server that manages this Security Gateway / Cluster.

2. From the left navigation panel, click Gateways & Servers.

3. Open the Security Gateway / Cluster object.

4. From the left tree, click Logs > Log Distribution.

5. In the section Log Distribution, select Distribute logs between log servers for improved performance (applies to primary and backup log servers).

6. In the section Log Servers > Primary log server, select the applicable primary Log Server object(s).

7. In the section Log Servers > Backup log server, select the applicable backup Log Server object(s).

8. Click OK.

9. Publish the SmartConsole session.

10. Install the Access Control policy on the Security Gateway / Cluster object.

Akos

[Parabol]

Thanks @AkosBakos I did look at this, and we do have the standby sms, so technically could distribute the logs 50/50 which I suppose could double our capacity. Especially with the standby sms sitting there with 1.8TB of storage not be used.

[AkosBakos]

That's all folks 🙂