useful command for log size investigation

[George_Liu]

The follow is a useful command to get and collect firewall log size and record count.

command:

CPLogInvestigator -a -m -p

======  Sample output  ======

[Expert@XXXXXXXX:0]# CPLogInvestigator -a -m -p

Thank you for using log investigator tool.

==============================================================
Start reading log file: /opt/CPsuite-R77/fw1/log/fw.log

................
Reading log file is DONE.

Total scanned 3050306 logs out of 3050306 logs in file
Scanned logs dates are from 23-07-2015 23:58:49 to 24-07-2015 16:30:19

========================================
Product log statistics (Per Day):
     - Anti Malware : 16430
     - Application Control : 1748816
     - Connectra : 129
     - Security Gateway/Management : 49
     - SmartDefense : 190
     - URL Filtering : 294281
     - VPN-1 & FireWall-1 : 2370107

Total logs per day:

    Date     |   GB   |   Count   
  2015-02-25 | 0.0333 | 255434
  2015-02-26 | 0.0456 | 344836
  2015-02-27 | 0.0365 | 279161
  2015-02-28 | 0.0362 | 277044
  2015-03-01 | 0.0378 | 288268
  2015-03-02 | 0.0501 | 381685
  2015-03-03 | 0.0480 | 366158
  2015-03-04 | 0.0512 | 390534
  2015-03-05 | 0.0462 | 349615
  2015-03-06 | 0.0471 | 353194
  2015-03-07 | 0.0426 | 318594
  2015-03-08 | 0.0415 | 310887
  2015-03-09 | 0.0461 | 343107
  2015-03-10 | 0.0463 | 347655
  2015-03-11 | 0.0453 | 338776
  2015-03-12 | 0.1459 | 1138706
  2015-03-13 | 0.0640 | 477258
  2015-03-14 | 0.0482 | 359125
  2015-03-15 | 0.0420 | 313691
  2015-03-16 | 0.0477 | 357323
  2015-03-17 | 0.0538 | 402038
  2015-03-18 | 0.0540 | 404283
  2015-03-19 | 0.0625 | 470422
  2015-03-20 | 0.0638 | 446467
  2015-03-21 | 0.0704 | 475022
  2015-03-22 | 0.0707 | 479010
  2015-03-23 | 0.0839 | 573192
  2015-03-24 | 0.0752 | 514870
  2015-03-25 | 0.0573 | 387271
  2015-03-26 | 0.0462 | 305485
  2015-03-27 | 0.0480 | 319043
  2015-03-28 | 0.0424 | 278708
  2015-03-29 | 0.0422 | 276846
  2015-03-30 | 0.0507 | 338030
  2015-03-31 | 0.0638 | 433103
  2015-04-01 | 0.0975 | 676181
  2015-04-02 | 0.0665 | 461179
  2015-04-03 | 0.0426 | 292025
  2015-04-04 | 0.0426 | 288689
  2015-04-05 | 0.0428 | 286940
  2015-04-06 | 0.0471 | 320728
  2015-04-07 | 0.1320 | 930117
  2015-04-08 | 0.1001 | 704473
  2015-04-09 | 0.0449 | 302196
  2015-04-10 | 0.0464 | 313163
  2015-04-11 | 0.0389 | 257880
  2015-04-12 | 0.0396 | 263557
  2015-04-13 | 0.0158 | 105195
  2015-04-14 | 0.0000 | 9
  2015-04-15 | 0.0000 | 1
  2015-04-16 | 0.0000 | 9
  2015-04-17 | 0.0000 | 1
  2015-04-18 | 0.0000 | 9
  2015-04-19 | 0.0000 | 1
  2015-04-20 | 0.0000 | 9
  2015-04-21 | 0.0000 | 1
  2015-04-22 | 0.0000 | 9
  2015-04-23 | 0.0000 | 1
  2015-04-24 | 0.0000 | 9
  2015-04-25 | 0.0000 | 1
  2015-04-26 | 0.0000 | 9
  2015-04-27 | 0.0000 | 1
  2015-04-28 | 0.0000 | 9
  2015-04-29 | 0.0000 | 1
  2015-04-30 | 0.0000 | 9
  2015-05-01 | 0.0000 | 3
  2015-05-02 | 0.0000 | 11
  2015-05-03 | 0.0000 | 3
  2015-05-04 | 0.0000 | 11
  2015-05-05 | 0.0000 | 1
  2015-05-06 | 0.0000 | 9
  2015-05-07 | 0.0000 | 1
  2015-05-08 | 0.0037 | 27941
  2015-05-09 | 0.0000 | 3
  2015-05-10 | 0.0000 | 11
  2015-05-11 | 0.0000 | 3
  2015-05-12 | 0.0000 | 11
  2015-05-13 | 0.0000 | 3
  2015-05-14 | 0.0000 | 11
  2015-05-15 | 0.0000 | 3
  2015-05-16 | 0.0000 | 11
  2015-05-17 | 0.0000 | 3
  2015-05-18 | 0.0000 | 11
  2015-05-19 | 0.0000 | 3
  2015-05-20 | 0.0000 | 11
  2015-05-21 | 0.0000 | 3
  2015-05-22 | 0.0000 | 11
  2015-05-23 | 0.0000 | 3
  2015-05-24 | 0.0000 | 11
  2015-05-25 | 0.0140 | 110974
  2015-05-26 | 0.0641 | 490665
  2015-05-27 | 0.0684 | 513296
  2015-05-28 | 0.0672 | 498948
  2015-05-29 | 0.0738 | 547163
  2015-05-30 | 0.0726 | 541831
  2015-05-31 | 0.0729 | 548021
  2015-06-01 | 0.0789 | 591918
  2015-06-02 | 0.0814 | 610398
  2015-06-03 | 0.0842 | 619991
  2015-06-04 | 0.0767 | 561824
  2015-06-05 | 0.0773 | 572785
  2015-06-06 | 0.0458 | 341212
  2015-06-07 | 0.0460 | 342448
  2015-06-08 | 0.0611 | 454891
  2015-06-09 | 0.0811 | 597523
  2015-06-10 | 0.0818 | 602761
  2015-06-11 | 0.1115 | 814637
  2015-06-12 | 0.0846 | 623603
  2015-06-13 | 0.0702 | 515941
  2015-06-14 | 0.0726 | 533099
  2015-06-15 | 0.1029 | 754834
  2015-06-16 | 0.1189 | 871273
  2015-06-17 | 0.1613 | 1175605
  2015-06-18 | 0.1564 | 1169985
  2015-06-19 | 0.1667 | 1287849
  2015-06-20 | 0.1358 | 1026636
  2015-06-21 | 0.1369 | 1029263
  2015-06-22 | 0.1440 | 1084049
  2015-06-23 | 0.1528 | 1155860
  2015-06-24 | 0.1670 | 1262396
  2015-06-25 | 0.1601 | 1204950
  2015-06-26 | 0.1679 | 1255164
  2015-06-27 | 0.1806 | 1340304
  2015-06-28 | 0.1844 | 1373050
  2015-06-29 | 0.2088 | 1557819
  2015-06-30 | 0.2225 | 1655548
  2015-07-01 | 0.2122 | 1574930
  2015-07-02 | 0.2024 | 1500243
  2015-07-03 | 0.2025 | 1588413
  2015-07-04 | 0.2115 | 1791576
  2015-07-05 | 0.2044 | 1698598
  2015-07-06 | 0.1996 | 1576800
  2015-07-07 | 0.2643 | 2030466
  2015-07-08 | 0.1788 | 1418933
  2015-07-09 | 0.1776 | 1420445
  2015-07-10 | 0.2768 | 2204455
  2015-07-11 | 0.1779 | 1326958
  2015-07-12 | 0.2167 | 1632107
  2015-07-13 | 0.2245 | 1679169
  2015-07-14 | 0.1632 | 1216088
  2015-07-15 | 0.1348 | 1073933
  2015-07-16 | 0.1220 | 936694
  2015-07-17 | 0.1198 | 903149
  2015-07-18 | 0.1044 | 803158
  2015-07-19 | 0.1083 | 849078
  2015-07-20 | 0.1342 | 1075080
  2015-07-21 | 0.1324 | 984871
  2015-07-22 | 0.1746 | 1338419
  2015-07-23 | 0.2254 | 1616671
  fw.log | 0.4212 | 3050306

==============================================================
Logs per minute table can be found at logPerMinute.txt

==============================================================

[Neville_Kuo]

這招用過幾次,基本上中小客戶沒什麼問題,但是在Log量大的客戶,比方說大學或教網,一天的Log可以到10~30G的,通常就不能帶這麼多參數了,否則會失敗。

[Don_Paterson]

Nice. Thanks.

Here is one from a small lab (R80.10):

[Expert@A-SMS:0]# CPLogInvestigator -a -m -p

Thank you for using log investigator tool.

==============================================================
Start reading log file: /opt/CPsuite-R80/fw1/log/fw.log

Start reading log file: /opt/CPsuite-R80/fw1/log/fw.log from log 0

..
Reading log file is DONE.

Total scanned 17888 logs out of 17888 logs in file
Scanned logs dates are from 17-04-2019 11:22:39 to 17-04-2019 15:00:38

========================================
Product log statistics (Per Day):
Days of counting: 0.151377
Product name: Anti Malware Amount of logs: 508 Average: 3355
Product name: Application Control Amount of logs: 224 Average: 1479
Product name: Compliance Blade Amount of logs: 1 Average: 6
Product name: Content Awareness Amount of logs: 28 Average: 184
Product name: Eventia Analyzer Client Amount of logs: 1 Average: 6
Product name: Identity Awareness Amount of logs: 7 Average: 46
Product name: N/A Amount of logs: 350 Average: 2312
Product name: New Anti Virus Amount of logs: 27 Average: 178
Product name: Security Gateway/Management Amount of logs: 10 Average: 66
Product name: SmartConsole Amount of logs: 7 Average: 46
Product name: URL Filtering Amount of logs: 21 Average: 138
Product name: VPN-1 & FireWall-1 Amount of logs: 16719 Average: 110445

Total logs per day:

Date | GB | Count
2018-02-19 | 0.0006 | 17568
2018-02-20 | 0.0006 | 4750
2018-02-21 | 0.0294 | 338432
2018-03-23 | 0.0036 | 39726
2018-05-30 | 0.0008 | 12594
2018-06-01 | 0.0005 | 8224
2018-07-03 | 0.0009 | 15486
2018-11-14 | 0.0001 | 1588
2019-04-15 | 0.0001 | 1698
2019-04-16 | 0.0025 | 40772
2019-04-17 | 0.0041 | 58396
fw.log | 0.0029 | 35776

==============================================================
Logs per minute table can be found at logPerMinute.txt

==============================================================
[Expert@A-SMS:0]#

[genisis__]

Hi Don - Long time!

Do you know if the stats include indexed logs or is this just raw log files?

[Don_Paterson]

Hello! 🙂

I believe it is only the active log file (fw.log)
Not sure how the index could be scanned. I understand that it summarized logs so I am not sure if it is possible.

There is a SOLR command line option so maybe that would allow it.
That's beyond my knowledge at this point.

Regards,

Don

[genisis__]

Thanks Don!

[Don_Paterson]

@PhoneBoy 

Hi Dameon,
Is there another log analyzer tool that captures more than just the active log file?
(more than  CPLogInvestigator -a -m -p)

Regards,

Don

[PhoneBoy]

I think -m might be causing it to only get the active log file.
See: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut... 

[ebenezer]

Product name: N/A Amount of logs: 350 Average: 2312 what is the meaning of N/A on the logs. which blade is related to N/A

[Don_Paterson]

I am not sure.

I can't find anything on it.

Maybe is it Control (Type) logs. Search "Control" in the LOGS & MONITOR Logs tab.

Since those are not Security logs they are not listed in the Log Description Fields, but it is in some CLI guides.

https://support.checkpoint.com/results/sk/sk144192

https://sc1.checkpoint.com/documents/R80.30/WebAdminGuides/EN/CP_R80.30_CLI_ReferenceGuide/html_fram...

[genisis__]

Don,

Just a not, I believe in R80.x and later this is not available, and you would need to run doctor-log located in $RTDIR/scripts

[Don_Paterson]

It's included in R81.10 by default.

Expert mode: just type in CPLogInvestigator and press enter.

Doctor log is another option. 

[genisis__]

Tried running that, but did not work (Its an MDS setup),  doctor-log attempts to run this as well, but could not find it.

My main objective is to determine the daily amount of logs and more challenging I export TP data via logexporter, so would like to determine daily amount I'm exporting.

The information drive is related to migrating a DMS to Smart-1 Cloud.