topic Re: FW logs missing -- all other lost are available in Firewall and Security Management

FW logs missing -- all other lost are available

Paul_Warnagiris — Tue, 28 Sep 2021 19:29:20 GMT

I have an odd one. Over the weekend I had a customer running 80.30 JHF236 stop logging all FW events. Logging is working as expected. GW log files are not incrementing, the date and time is good, SmartLog shows recent App/URLF/TE logs. I have rebooted each GW in the cluster as well as the log server. Still no logs. When I say I see not FW logs that is not exactly true. Any FW log with and "alert" type shows up. But regular accepts/drops for sessions or connections are not visible. If I go back to Tracker (CPlgv.exe) I can see the FW logs. Any thoughts or ideas?

Tracker:

SmartLog:

Re: FW logs missing -- all other lost are available

the_rock — Tue, 28 Sep 2021 21:13:21 GMT

Hm, could be log indexing issue, sounds like, but not 100% sure. Do you have that enabled?

Re: FW logs missing -- all other lost are available

Paul_Warnagiris — Tue, 28 Sep 2021 21:17:25 GMT

Yes, its been a working installation for years. All was working fine until Saturday. Boxes not pegged or exhausted and they have been rebooted within the past 45 days. I guess what I didn't explain before is my environment is distributed. Separate SMS/Log/SE. The only thing I did not reboot was the SMS. After rebooting it, it was resolved. Still, no signs of problems before reboot. Odd for sure.

Re: FW logs missing -- all other lost are available

the_rock — Tue, 28 Sep 2021 21:49:10 GMT

I agree with you brother, it is odd, for sure. I will tell you, normally what I follow to fix any logging issue is below:

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk38848

Change $FWDIR/conf/masters file on gateway(s) to reflect management object IP rather than name and then apply below sk:

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk102712&srcFavorites=favorites

There is "old school" way of fixing logging too, but I shall not mention it here, as probably no one uses it any more anyway : )

Cheers,

Andy

Re: FW logs missing -- all other lost are available

Paul_Warnagiris — Tue, 28 Sep 2021 21:56:50 GMT

Yup. I do installs/upgrades/troubleshooting for a living. I'm very familiar with both of those SKs. I am used to seeing logs work or not work, not some logs work and some not (from the same log source). I just never figured it would be the SMS since it doesn't do the indexing, but what do I know? You learn something new every day. Thanks for the input.

Paul

Re: FW logs missing -- all other lost are available

Maarten_Sjouw — Wed, 29 Sep 2021 07:06:54 GMT

When you open the logfile itself, is the info there?

Re: FW logs missing -- all other lost are available

Ruan_Kotze — Wed, 29 Sep 2021 09:23:48 GMT

I just have to take a guess on the old school methods:

- Replace log server with dummy object with same IP as "proper" log server. Push policy. Swap out with proper log server and push policy.

<or>

- Nuke from orbit (aka delete FetchedFiles)

Re: FW logs missing -- all other lost are available

G_W_Albrecht — Wed, 29 Sep 2021 09:40:31 GMT

The SMS in fact does do the indexing - you enable it on SMS Tab Logs...

Re: FW logs missing -- all other lost are available

Timothy_Hall — Wed, 29 Sep 2021 12:02:37 GMT

Definitely sounds like a log indexing issue; my experience is that TAC will normally need to figure out what is happening. If you'd like to avoid a full reboot in the future for resolution, run these:

stopIndexer
startIndexer

If the problem still persists try these:

evstop
evstart

Re: FW logs missing -- all other lost are available

Paul_Warnagiris — Wed, 29 Sep 2021 12:12:35 GMT

Yes, there were logs in there. I actually opened it through tracker though, I forgot about that piece in SmartLog. I assume if its visible in tracker it would be visible there. Or is that an indexed 2G file?

Re: FW logs missing -- all other lost are available

the_rock — Wed, 29 Sep 2021 13:08:59 GMT

Yup, pretty much on both 🙂

Re: FW logs missing -- all other lost are available

Maarten_Sjouw — Thu, 30 Sep 2021 07:41:39 GMT

It is the same, just not the indexed piece.
I have had many issues with logging in R80.30 and R80.40 and had to do a evstop and mdsstart to get it to resolve, but in your case it sounds like there is an issue with the indexer itself or there is a an issue in Solr, however rebooting the SMS and log server should resolve that.
Do keep in mind that your not directly connecting to the log server but to the SMS which is forwarding your request to the log server. so you should also do the evstop/cpstart on the SMS.

To restart Solr only:

cd /opt/CPrt-R81/scripts/
./stopSolr.sh;./startSolr.sh