Logging statistics | MDM R80.20 - question regarding the actual rates

[Maik]

Hello guys,

I have a question regarding the measurement of incoming logs, let's say some kind of "rate" of logs per second or minute. Related to that I have found two SKs:

- sk88681 [How to calculate/count the total amount of FireWall Logs per second that arrive to Securi...

- sk120341 [How to monitor the Log Receive Rate on Management Server / Log Server R80 and above]

The intersting thing is that both SKs are valid for R80.20 and also apply for MDM environmnets, such as the current case where I want to use this. However; here is the example output of one of my domains:

sk120341 output:

[Expert@MDMSERVER:0]# cpstat mg -f log_server

Log Receive Rate: 50
Log Receive Rate Peak: 4722
Log Receive Rate Last 10 Minutes: 57
Log Receive Rate Last Hour: 59

Log Server Connected Gateways
---------------------------------------------------------------------------------------------
|Name |State |Last Login Time |Log Receive Rate|
---------------------------------------------------------------------------------------------
|Local Clients |Connected|N/A | 0|
|VSX-GW-A|Connected|Wed Aug 14 10:57:19 2019 | 0|
|VSX-GW-B|Connected|Fri Aug 9 11:28:53 2019 | 50|
---------------------------------------------------------------------------------------------

sk88681 output:

[Expert@MDMSERVER:0]# ls -l fw.logptr ; sleep 180 ; ls -l fw.logptr
-rw-r--r-- 1 admin root 18895720 Sep 19 12:18 fw.logptr
-rw-r--r-- 1 admin root 18972856 Sep 19 12:21 fw.logptr

(18972856 - 18895720) / (4* 180) = about107

I have run these commands in the related management domain (switched via mdsenv) - however the results are kinda different. The output of sk88681 seems to be more correct as lots of connections pass through the gateway. But how does the first output lists only about 50 logs per second and the second one more than 100? I guess the "log receive rate" of the cpstat mg command also references to logs per second. Where is the difference between both? And why does the cpstat command also tell that "Log Receive Rate Last 10 Minutes" is just 57. That's definitely not the case or do I mix something up in this case?

Also a different question regarding log file movement. Is it save to move e.g. the following files from /CPsuite-R80.20/fw1/log to a different system in order to save them as a backup? Or do I have some negative impact on the management server if I'm going to remove these files?

2018-11-10_000000.adtlog
2018-11-10_000000.adtlogaccount_ptr
2018-11-10_000000.adtloginitial_ptr
2018-11-10_000000.adtlogptr
2018-11-10_000000.log

I have the requirement to move all log entries which are older than 3 months to a different system - or even delete them. As far as I know this option is not possible via the SmartConsole itself (only via a script that checks the timestamp of files in the log dir and removes them if they are older than three months). Or are there other ways in order to achieve the described goal? Currently thinking about a cronjob that runs each night, which looks for log files older than x days and removes them. But still - I am not sure if this method is "clean".

Thanks and best regards,

Maik

[Maik]

*push* as this thread already disappeared from the front page

[PhoneBoy]

cpstat mg -f log_server gives you actual numbers, where the "Log Receive Rate" is a longer-term average.
The method in sk88681 is used to estimate these values when only the log files are available.
I would expect these numbers to be somewhat different.

If you want to remove older log files, I believe it is safe to remove the stated files provided they are archived appropriately.

[Dror_Aharony · Accepted solution]

Adding some more info:

cpstat mg -f log_server is the most accurate method. as PhoneBoy said, just a matter of different time averages.

your result from sk88681 is actually double the actual rate, as the pointer file was changed from 4 to to 8 bytes on R80.x.

(I'll make sure to have the sk fixed for R80.x).

so simply divide by 8 instead of 4:

(18972856 - 18895720) / (8 * 180) = about 53-54 logs/sec (around the same as the 57 logs/sec).

Always best to copy/move all .log*/.adtlog* files to include all 4 files for backup.

Example of 09-22 (Sep 22nd log-files):

for GW Security logs: *09-22*.log*

for Audit logs: *09-22*.adtlog*

[Maik]

Ah the increase of 8 bytes per log explains a lot - thanks for the clarification and general information!

Will make sure to back up the logs as you mentioned. One more question regarding this...

If I need to load the logs back into the logging engine, in order to inspect them via the SmartConsole or SmartView - is it enough to move them back into the directory where they originated and reinstall the event policy in order to update the index? Or do I just to move them into the directory and it automatically recognizes the new files? If this is explained in any SK or document, I'd also be happy if you could link it - will be able to dig into details on my own then. 🙂

[Dror_Aharony]

Yea. Simply moving/copying back the files to original location ($FWDIR/log/) - that's it.

Assuming the indexes of the LS/SML weren't deleted, then viewing them in SmartConsole should work immediately.

for an SME, its indexes are independent, so Smartview will continue to function without the logs.

(it doesn't requires the log-files, once they were read & indexed).

2. for LS/SML, if some-time has passed & the matching indexes of the LS/SML (of that day) were already deleted by the disk-space maintenance mechanism, then it becomes much more complicated to re-index them, but in-general one would assume they are no longer needed.

It's possible, but it requires a total re-indexing  of the log DB, best done after setting the chosen no. of days back to reach the desired log-file. See sk111766:

https://supportcenter.checkpoint.com/supportcenter/portal?action=portlets.SearchResultMainAction&eve...