CPlogInvestigator - Your Key to Success in Sizing Log Server

[PhoneBoy]

The tool collects fundamental information to properly size log servers.

SmartEvent Sizing Tool (sk87263) 

[Nader_Assi__Old]

Hi Dameon,

Is there any similar tool available in R80.X?

Thx

[PhoneBoy]

I think the tool is still relevant, what's different is the sizing guidelines since R80.x assumes indexed logs (whereas that's not the case in R77.x).

[Nader_Assi__Old]

I tried to run it on our R80.10 CP Mgmt server and it won't start. Instead, it will crash and generate a core dump file!

[PhoneBoy]

It's entirely possible this may be not supported on R80, but it definitely shouldn't crash.

I would open a TAC case.

[RickLin]

I run this command on my customer without problem.

Security Management :R80.10 Jumbo Hotfix T103.

[Expert@xxxx:0]# CPLogInvestigator -a -m -p    

Thank you for using log investigator tool.

==============================================================

Start reading log file: /opt/CPsuite-R80/fw1/log/fw.log

Start reading log file: /opt/CPsuite-R80/fw1/log/fw.log from log 0

...........................

Reading log file is DONE.

Total scanned 5109144 logs out of 5109143 logs in file

Scanned logs dates are from 11-05-2018 14:20:44 to 11-05-2018 15:17:25

========================================

Product log statistics (Per Day): 

Days of counting:       0.0393634

Product name:   Anti Malware                Amount of logs:     228             Average:        5792

Product name:   Application Control         Amount of logs:     271888          Average:        6907122

Product name:   Connectra                   Amount of logs:     19              Average:        482

Product name:   Identity Awareness          Amount of logs:     588739          Average:        14956497

Product name:   N/A                         Amount of logs:     176225          Average:        4476871

Product name:   Security Gateway/Management Amount of logs:     4               Average:        101

Product name:   IPS                         Amount of logs:     135306          Average:        3437353

Product name:   System Monitor              Amount of logs:     22              Average:        558

Product name:   URL Filtering               Amount of logs:     156815          Average:        3983774

Product name:   VPN-1 & FireWall-1          Amount of logs:     3781797         Average:        96073878

Total logs per day: 

    Date     |   GB   |   Count    

  2018-02-10 | 7.7170 | 74234656

  2018-02-11 | 7.3400 | 70904796

  2018-02-12 | 14.6907 | 130706556

  2018-02-13 | 14.8150 | 131616524

  2018-02-14 | 14.4768 | 129228068

  2018-02-15 | 7.9124 | 77324608

  2018-02-16 | 8.0363 | 78587766

  2018-02-17 | 7.8498 | 76812992

  2018-02-18 | 7.8168 | 76744764

  2018-02-19 | 7.9787 | 77995078

  2018-02-20 | 8.1322 | 80146642

  2018-02-21 | 15.6809 | 138732466

  2018-02-22 | 14.5247 | 125943000

  2018-02-23 | 14.5114 | 126277356

  2018-02-24 | 7.7846 | 73974552

  2018-02-25 | 7.5231 | 71016134

  2018-02-26 | 14.4977 | 125213358

  2018-02-27 | 14.1192 | 122165608

  2018-02-28 | 8.1322 | 76311620

  2018-03-01 | 13.8683 | 119817466

  2018-03-02 | 13.7451 | 118279922

  2018-03-03 | 8.1156 | 76477180

  2018-03-04 | 7.3485 | 69412182

  2018-03-05 | 13.7205 | 118144206

  2018-03-06 | 13.5062 | 115787030

  2018-03-07 | 14.5205 | 125430466

  2018-03-08 | 14.7425 | 127474960

  2018-03-09 | 14.3615 | 124039788

  2018-03-10 | 7.4037 | 68910076

  2018-03-11 | 7.3932 | 69320686

  2018-03-12 | 14.8301 | 129172552

  2018-03-13 | 15.5869 | 136542722

  2018-03-14 | 16.3094 | 143276452

  2018-03-15 | 15.9171 | 139370046

  2018-03-16 | 14.8460 | 129605342

  2018-03-17 | 9.2400 | 87200472

  2018-03-18 | 9.2907 | 87865558

  2018-03-19 | 15.9531 | 138923332

  2018-03-20 | 16.7656 | 147823484

  2018-03-21 | 18.8047 | 172101126

  2018-03-22 | 38.2976 | 413416262

  2018-03-23 | 21.2566 | 203725852

  2018-03-24 | 14.7055 | 149744078

  2018-03-25 | 14.5656 | 150905666

  2018-03-26 | 20.5275 | 190987540

  2018-03-27 | 21.6443 | 196768808

  2018-03-28 | 22.3363 | 202348622

  2018-03-29 | 20.6306 | 184499626

  2018-03-30 | 19.9007 | 176509060

  2018-03-31 | 22.1647 | 201614582

  2018-04-01 | 18.9973 | 184064152

  2018-04-02 | 21.4189 | 190776348

  2018-04-03 | 22.1479 | 198801306

  2018-04-04 | 13.5555 | 129287084

  2018-04-05 | 13.1483 | 125652316

  2018-04-06 | 13.2263 | 126319352

  2018-04-07 | 15.2102 | 146501160

  2018-04-08 | 12.5828 | 120863124

  2018-04-09 | 23.2057 | 209771948

  2018-04-10 | 23.7384 | 215535572

  2018-04-11 | 22.7938 | 206349772

  2018-04-12 | 24.2364 | 219188870

  2018-04-13 | 22.3351 | 203192348

  2018-04-14 | 16.2754 | 157014054

  2018-04-15 | 11.6282 | 112205302

  2018-04-16 | 22.7395 | 207002610

  2018-04-17 | 20.6640 | 185395180

  2018-04-18 | 22.9000 | 207497694

  2018-04-19 | 25.5837 | 233602240

  2018-04-20 | 25.5605 | 232053214

  2018-04-21 | 14.6392 | 140548466

  2018-04-22 | 15.6229 | 150490844

  2018-04-23 | 22.3775 | 200796624

  2018-04-24 | 23.2701 | 209462844

  2018-04-25 | 21.3462 | 191103532

  2018-04-26 | 23.1698 | 209881228

  2018-04-27 | 20.8206 | 189064160

  2018-04-28 | 13.3968 | 128741202

  2018-04-29 | 13.4174 | 129346706

  2018-04-30 | 21.8181 | 198407414

  2018-05-01 | 17.7662 | 171171352

  2018-05-02 | 25.2506 | 230229768

  2018-05-03 | 26.0021 | 237142788

  2018-05-04 | 24.8001 | 225913828

  2018-05-05 | 17.3057 | 166749712

  2018-05-06 | 18.9016 | 182878552

  2018-05-07 | 25.8999 | 234935224

  2018-05-08 | 28.2340 | 258688492

  2018-05-09 | 28.5828 | 262709190

  2018-05-10 | 25.3027 | 231469136

  2018-05-11 | 15.9980 | 145745988

  fw.log | 1.1948 | 10218086

==============================================================

Logs per minute table can be found at logPerMinute.txt

==============================================================

[Expert@xxxx:0]#

[ED]

Generated a core dump file on my mgmt too. R80.10. Have you found a solution for this, Nader?

[Sebastien_Rho]

Have you looked for the proper version of the CPLogInvestigator in :

./opt/CPsuite-R80.10/fw1/bin/CPLogInvestigator

mine did not create a core dump.

Regards.

[BigHec]

Hi,

May I ask is there any tools or command that is able to count and list out the total logs that have been forwarded to my SIEM daily?

Thank you

[Hugo_vd_Kooij]

I suggest you open a new thread and add additional information about your setup. As the question you have is actually not related to this thread based on your short description.

[Don_Paterson]

Hi PhoneBoy,

Is there an SK or known info on the log size for Threat Prevention?

https://support.checkpoint.com/results/sk/sk181782 says:

"The average log size is ~150 bytes in size. Use this information to perform the following calculation
(Sustained logs per second) x (Seconds per working day) x (Log entry size) ÷ 1024 ÷ 1024 ÷ 1024 = Log storage in GB per day"

Also in the SK:

"Security Checkup 

When capturing data during a security checkup, it is important to note the logs generated do not include firewall logs. Threat Prevention, Application Control, and URL Filtering are estimated to be 2-10% of the total logs for a customer. Therefore, divide the Sustained logs per second by 0.06 (6%) to obtain the total sustained logs per second, including firewall logs.
Note: 6% is the median for 2-10%."

Regards,

Don

[PhoneBoy]

I don't believe we make a distinction between traffic logs and Threat Prevention logs in terms of log sizing.