I am investigating the sheer number of logs we are generating on our management system. According to my SIEM admins, the number of logs being ingested has more than doubled in the last year.
In any case, I'm looking at CPLogInvestigator. But what I am seeing does not make sense: The log stats per day do not match the number of logs for the last day, not even close. Here is a snippet:
# CPLogInvestigator -a -m -p
Thank you for using log investigator tool.
==============================================================
Start reading log file: /opt/CPsuite-R80.40/fw1/log/fw.log
Start reading log file: /opt/CPsuite-R80.40/fw1/log/fw.log from log 0
..................................................
Reading log file is DONE.
Total scanned 9618460 logs out of 9618459 logs in file
Scanned logs dates are from 09-06-2022 14:26:26 to 10-06-2022 18:33:25
========================================
Product log statistics (Per Day):
Days of counting: 1.17152
Product name: Anti Malware Amount of logs: 312 Average: 266
Product name: Application Control Amount of logs: 112390 Average: 95935
Product name: Connectra Amount of logs: 7 Average: 5
Product name: ESOD Amount of logs: 118 Average: 100
Product name: Firewall Amount of logs: 6 Average: 5
Product name: HTTPS Inspection Amount of logs: 1291956 Average: 1102806
Product name: Identity Awareness Amount of logs: 65266 Average: 55710
Product name: MTA Amount of logs: 1640 Average: 1399
Product name: N/A Amount of logs: 601007 Average: 513016
Product name: New Anti Virus Amount of logs: 9 Average: 7
Product name: Security Gateway/Management Amount of logs: 58 Average: 49
Product name: IPS Amount of logs: 5763 Average: 4919
Product name: System Monitor Amount of logs: 11 Average: 9
Product name: Threat Emulation Amount of logs: 51085 Average: 43605
Product name: Threat Extraction Amount of logs: 11 Average: 9
Product name: URL Filtering Amount of logs: 117484 Average: 100283
Product name: VPN-1 & FireWall-1 Amount of logs: 7371478 Average: 6292254
Total logs per day:
Date | GB | Count
<snip>
2022-06-08 | 33.9062 | 331916400
2022-06-09 | 27.6229 | 266948294
fw.log | 1.9735 | 19235418
If you notice, the total number of logs in the section "Product log statistics (Per Day):", which is a ~28 hour period, is 9,618,601. But the total number of logs on 6/8/2022 is over 331 million and on 6/9/2022, today is 266 million.
What am I to make of this?
