FW logs missing -- all other lost are available

[Paul_Warnagiris]

I have an odd one.  Over the weekend I had a customer running 80.30 JHF236 stop logging all FW events.  Logging is working as expected.  GW log files are not incrementing, the date and time is good, SmartLog shows recent App/URLF/TE logs.  I have rebooted each GW in the cluster as well as the log server.  Still no logs.  When I say I see not FW logs that is not exactly true.  Any FW log with and "alert" type shows up.  But regular accepts/drops for sessions or connections are not visible.  If I go back to Tracker (CPlgv.exe) I can see the FW logs.  Any thoughts or ideas?

Tracker:

 SmartLog:

 

[the_rock]

Hm, could be log indexing issue, sounds like, but not 100% sure. Do you have that enabled?

[Paul_Warnagiris]

Yes, its been a working installation for years.  All was working fine until Saturday.  Boxes not pegged or exhausted and they have been rebooted within the past 45 days.  I guess what I didn't explain before is my environment is distributed.  Separate SMS/Log/SE.  The only thing I did not reboot was the SMS.  After rebooting it, it was resolved.  Still, no signs of problems before reboot.  Odd for sure.

[the_rock]

I agree with you brother, it is odd, for sure. I will tell you, normally what I follow to fix any logging issue is below:

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

OR

Change $FWDIR/conf/masters file on gateway(s) to reflect management object IP rather than name and then apply below sk:

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

There is "old school" way of fixing logging too, but I shall not mention it here, as probably no one uses it any more anyway : )

Cheers,

Andy

[Paul_Warnagiris]

Yup.  I do installs/upgrades/troubleshooting for a living.  I'm very familiar with both of those SKs.  I am used to seeing logs work or not work, not some logs work and some not (from the same log source).  I just never figured it would be the SMS since it doesn't do the indexing, but what do I know?  You learn something new every day.  Thanks for the input.

Paul

[Maarten_Sjouw]

When you open the logfile itself, is the info there?

[Paul_Warnagiris]

Yes, there were logs in there.  I actually opened it through tracker though, I forgot about that piece in SmartLog.  I assume if its visible in tracker it would be visible there.  Or is that an indexed 2G file?

[Maarten_Sjouw]

It is the same, just not the indexed piece.
I have had many issues with logging in R80.30 and R80.40 and had to do a evstop and mdsstart to get it to resolve, but in your case it sounds like there is an issue with the indexer itself or there is a an issue in Solr, however rebooting the SMS and log server should resolve that. 
Do keep in mind that your not directly connecting to the log server but to the SMS which is forwarding your request to the log server. so you should also do the evstop/cpstart on the SMS.

To restart Solr only:

cd /opt/CPrt-R81/scripts/
./stopSolr.sh;./startSolr.sh

[G_W_Albrecht]

The SMS in fact does do the indexing - you enable it on SMS Tab Logs...

[Ruan_Kotze]

I just have to take a guess on the old school methods:

- Replace log server with dummy object with same IP as "proper" log server.  Push policy. Swap out with proper log server and push policy.

<or>

- Nuke from orbit (aka delete FetchedFiles)

[the_rock]

Yup, pretty much on both 🙂

[Timothy_Hall]

Definitely sounds like a log indexing issue; my experience is that TAC will normally need to figure out what is happening.  If you'd like to avoid a full reboot in the future for resolution, run these:

• stopIndexer
• startIndexer

If the problem still persists try these:

• evstop
• evstart