Log rate calculator for MDS / MLM

[Kaspars_Zibarts]

There are two great SKs out there, so this is just re-inventing the wheel

sk120341 - How to monitor the Log Receive Rate on Management Server

sk88681 - How to calculate/count the total amount of FireWall Logs per second that arrive to Security Management Server

I simply bastardised the script from the second article to make it faster (it runs parallel instead of sequential through all CMAs) and output is more easy to read. But all the credits go to CP!

Here's your script, a you can see you can supply time that you want to run it for, by default 5secs

#!/bin/sh
# Print log rate data on all Domains

# execute the script that defines Check Point environment variables
source /opt/CPshared/5.0/tmp/.CPprofile.sh
total=0

if [ "x$1" = "x" ]; then
   SLEEP_TIME=5
else
   SLEEP_TIME=$1
fi

echo
echo "Log rate per second, measuring for ${SLEEP_TIME}s"
echo "----------------------------------"
ls -l /var/log/mds_logs/*/log/fw.logptr > BEFORE
sleep $SLEEP_TIME
ls -l /var/log/mds_logs/*/log/fw.logptr > AFTER

for DOMAIN in $(ls -1 $MDSDIR/customers)
do
   SIZE_BEFORE=$(cat BEFORE | grep "/$DOMAIN/" | awk '{print $5}')
   SIZE_AFTER=$(cat AFTER | grep "/$DOMAIN/" | awk '{print $5}')
   rate=`echo "scale=0 ; ($SIZE_AFTER - $SIZE_BEFORE) / ( 4 * $SLEEP_TIME )" | bc`
   let total=total+rate
   echo -e -n "${DOMAIN}:\t"
   if [ ${#DOMAIN} -le 15 ]; then echo -e -n "\t"; fi
   echo "$rate"
done

echo "----------------------------------"
echo -e "Total rate:\t\t$total"
echo

[PhoneBoy]

Going to move this to the https://community.checkpoint.com/community/management/visibility-monitoring?sr=search&searchId=e7e66...‌ space

Thanks for sharing!

[Martin_Valenta]

Log rate per second, measuring for 5s
----------------------------------
dcwclm01: 272
dcwclm02: 220
dcwclm03: 1788
dcwclm04: 2120
dcwclm05: 364
dcwclm06: 652
dcwclm07: 3860
dcwclm08: 4
dcwclm09: 2328
dcwclm10: 2144
dcwclm11: 2280
dcwclm12: 884
----------------------------------
Total rate: 16916

Running on physical server with 28 CPU cores and 14 TB storage of with Raid 6..

Is there any recommendation how much logs should CLM receive?

[Kaspars_Zibarts]

I haven't seen exact recommendations - it really depends on how fast is your IO to disks I guess. As long as you don't have massive backlog (https://community.checkpoint.com/thread/6269-how-to-quickly-check-log-indexing-backlog ) and rest of the resources (CPU/RAM/disks) are under control you should be good.

[Martin_Valenta]

@Kaspars_Zibarts  I've modified script to get a little bit different output format in order to graph those data, but having issues that sometimes it's throwing negative values, check attached image. What could be the cause ? I suspect this:

rate=`echo "scale=0 ; ($SIZE_AFTER - $SIZE_BEFORE) / ( 4 * $SLEEP_TIME )" | bc`

[Kaspars_Zibarts]

I would say when log is rotated then before value would end up higher than after. So you would need to add a check if log file has changed whilst waiting for calculation 🙂 and then ignore results. Or simply ignore negative results as they are caused (most likely) by log file rotation