topic Re: SmartLog only indexing the current fw.log in AI Network Firewall

SmartLog only indexing the current fw.log

Ryan_Ryan — Thu, 23 Aug 2018 01:42:56 GMT

Hi there,

Our smartlog for some reason is only showing logs for the current fw.log file (meaning I only have half a day of logs in smartlog). The time of my smartlog index ties up exactly with the start of the current fw.log file. When fw.log rolls over I lose all my previous logs and it starts the index again.

Smartview tracker is showing 40 days worth of log files, I have over 1tb of free hdd space. tried disabling smart log, pushing policy then re-enabling it again but the same results. It was working previously, not sure what changed.

This doesn't seem to be a common issue either, any ideas?

Re: SmartLog only indexing the current fw.log

Kaspars_Zibarts — Fri, 21 Jun 2019 09:10:26 GMT

You may check this one below - it has references to two good SKs you may want to read regarding log indexing. Your indexer is possibly stuck at a specific log file and you may want to skip it manually.

https://community.checkpoint.com/message/11199-how-to-quickly-check-log-indexing-backlog

Re: SmartLog only indexing the current fw.log

Ryan_Ryan — Thu, 23 Aug 2018 05:00:55 GMT

thanks for the reply, I think that command is suited only for MDM as it didn't work well on my box. I also went through those two SK's to no avail.

if I check $SMARTLOGDIR/data/FetchedFiles it is showing 122 log files, I also followed another sk73361, the fetchedfiles accumulated back up to what it previously was, but still cant see the logs in smartlog!

Re: SmartLog only indexing the current fw.log

Kaspars_Zibarts — Thu, 23 Aug 2018 05:18:04 GMT

to take away MDS part, run it like this

cat $INDEXERDIR/data/FetchedFiles | while read line; do if [ `echo $line | awk '{print $7}'` -eq 0 ] && [ `echo $line | grep -c " fw."` -eq 0 ] && [ `echo $line | grep -c "serialization"` -eq 0 ]; then echo $line; fi done

can you share output of cat $INDEXERDIR/data/FetchedFiles?

Re: SmartLog only indexing the current fw.log

Ryan_Ryan — Thu, 23 Aug 2018 05:35:56 GMT

On my system the variable $INDEXERDIR doesnt exist. However removing that and using the absolute path (/var/log/opt/CPSmartLog-R77/data/FetchedFiles) the output of the command gives the exact same output as doing just the cat of the file (same number of lines - 122)

The output of cat is: (trimmed to just the top 5 lines)

22 serialization::archive 9 0 0 0 0 123 1 0 1 3 1 0
0 9 127.0.0.1 25 2018-08-08_105409_120.log 1533649377 0 4294967295 0 0 0 0 3
1 9 127.0.0.1 21 2018-07-08_010000.log 1530948689 0 4294967295 0 0 3
2 9 127.0.0.1 25 2018-08-14_232927_134.log 1534215276 0 4294967295 0 0 3
3 9 127.0.0.1 24 2018-07-28_015714_97.log 1532656050 0 4294967295 0 0 3
4 9 127.0.0.1 24 2018-07-21_010436_83.log 1532050379 0 4294967295 0 0 3
5 9 127.0.0.1 25 2018-08-21_151813_150.log 1534804867 0 4294967295 0 0 3
6 9 127.0.0.1 24 2018-07-24_111548_88.log 1532350801 0 4294967295 0 0 3
7 9 127.0.0.1 21 2018-07-12_010000.log 1531298007 0 4294967295 0 0 3
8 9 127.0.0.1 21 2018-07-22_010000.log 1532163046 0 4294967295 0 0 3

<trim>

81 9 127.0.0.1 6 fw.log 1534992458 0 4294967295 1 0 2 0 0 5023588 3

<trim>

thanks!

Re: SmartLog only indexing the current fw.log

Kaspars_Zibarts — Thu, 23 Aug 2018 05:54:26 GMT

Oh, you're on R77! Will have to dig notes out on that subject. I thought it was R80

Re: SmartLog only indexing the current fw.log

Kaspars_Zibarts — Thu, 23 Aug 2018 09:46:05 GMT

Have you looked at these SKs:

SmartLog does not index logs that existed prior to SmartLog installation

SmartLog cannot index existing old logs

Also have you checked logs in $SMARTLOGDIR/log/smartlog_server.elg

Re: SmartLog only indexing the current fw.log

Ryan_Ryan — Thu, 23 Aug 2018 23:40:42 GMT

Looks like a miracle has happened, I now have 13 days of logs in smartlog, I think running sk73361 must have fixed it and it just took a very long time to reindex the files. I might even get more logs if I check again later.

thanks for your help!