Looking at CPLogInvestigator. Daily stats do not match total logs per day. What gives?

[Dale_Lobb]

I am investigating the sheer number of logs we are generating on our management system.  According to my SIEM admins, the number of logs being ingested has more than doubled in the last year.

  In any case, I'm looking at CPLogInvestigator.  But what I am seeing does not make sense:  The log stats per day do not match the number of logs for the last day, not even close.    Here is a snippet:

# CPLogInvestigator -a -m -p

Thank you for using log investigator tool.

==============================================================
Start reading log file: /opt/CPsuite-R80.40/fw1/log/fw.log

Start reading log file: /opt/CPsuite-R80.40/fw1/log/fw.log from log 0

..................................................
Reading log file is DONE.

Total scanned 9618460 logs out of 9618459 logs in file
Scanned logs dates are from 09-06-2022 14:26:26 to 10-06-2022 18:33:25

========================================
Product log statistics (Per Day):
Days of counting: 1.17152
Product name: Anti Malware Amount of logs:                                312 Average: 266
Product name: Application Control Amount of logs:                112390 Average: 95935
Product name: Connectra Amount of logs:                                        7 Average: 5
Product name: ESOD Amount of logs:                                           118 Average: 100
Product name: Firewall Amount of logs:                                            6 Average: 5
Product name: HTTPS Inspection Amount of logs:                 1291956 Average: 1102806
Product name: Identity Awareness Amount of logs:                   65266 Average: 55710
Product name: MTA Amount of logs:                                           1640 Average:  1399
Product name: N/A Amount of logs:                                        601007 Average: 513016
Product name: New Anti Virus Amount of logs:                                  9 Average: 7
Product name: Security Gateway/Management Amount of logs:       58 Average: 49
Product name: IPS Amount of logs:                                              5763 Average: 4919
Product name: System Monitor Amount of logs:                               11 Average: 9
Product name: Threat Emulation Amount of logs:                      51085 Average: 43605
Product name: Threat Extraction Amount of logs:                            11 Average: 9
Product name: URL Filtering Amount of logs:                           117484 Average: 100283
Product name: VPN-1 & FireWall-1 Amount of logs:               7371478 Average: 6292254

Total logs per day:

Date | GB | Count
<snip>
2022-06-08 | 33.9062 | 331916400
2022-06-09 | 27.6229 | 266948294
fw.log          | 1.9735   |    19235418

  If you notice, the total number of logs in the section "Product log statistics (Per Day):", which is a ~28 hour period, is 9,618,601.  But the total number of logs on 6/8/2022 is over 331 million and on 6/9/2022, today is 266 million.

  What am I to make of this?

[Timothy_Hall]

Perhaps the former number (9,618,601) counts the amount of consolidated session logs, while 266 million is the total number of raw unconsolidated logs sent from the gateway to the log server?  A single consolidated session log consists of many, many raw logs sent by the gateway for individual product blades and updates for Accounting and such that are rolled up at the log server level.

[PhoneBoy]

Log Exporter sends a log every 10 minutes for log entries that have accounting data as well as an entry when the session ends.
This can mean a single log entry in SmartView can generate many, many logs to your SIEM.

[prakash7]

Product name: N/A Amount of logs:                                        601007 Average: 513016

What is N/A logs?

[PhoneBoy]

N/A means it's not related to a specific product.
I assume this is just a log count since the last "marker" (what this appears to be).

[prakash7]

Hi phoneboy,

N/A gives higher logs. It's normal or abnormal