Firewalls stop logging to Management Server (R80.20)

[Paul_Mainhardt1]

We are currently experiencing issues with logging from our firewalls to the management server. It logs correctly for awhile then all off a sudden stops logging. We are running 5600 appliances for our gateways and our management server is an open server.

We are running R80.20 T87 for both our firewalls and SMS.

I suspect its something related to high cpu for fw_full as i notice it reaches 80- 90% CPU but fw_worker_0 - 2 have low CPU usage.

I do have identity awareness, App Control, URL Filtering, IPS, Threat Emulation and Anti-bot and Antivirus turned on for the gateways. I am not sure if one of these blades are causing us issues.

[Dror_Aharony]

Your Mgmt is at its max Log receiving capacity when the fw_full is consistently close to a 100% (80-90% is close).

When you have log traffic peaks that cause it to reach ~100%, it may stop receiving logs & the GW will locally logs on itself for a few secs or more, depending on load.

Unless the log-rate is actually low, cause you said the fw_worker's CPU is low, then it requires a TAC investigation.

Can you specific Mgmt HW details & Log-rate? 

  run on Mgmt: cpstat mg -f log_server

If possible, I would advise to up the Mgmt's HW specs (CPU mostly).

[Paul_Mainhardt1]

Sorry my mistake - I wasn't clear enough in my original post.

Its the gateway thats reaching 90%+ CPU for fw_full and the same behavior also happens on the standby member. I suspect that this is causing the gateways not to send any logs to the SMS server.

The SMS has low CPU and RAM utilization

cpstat mg -f log_server - shows log receive rate of 0 from both gateways.

[PhoneBoy]

You should probably get the TAC involved.

[Timothy_Hall]

Some good tips in these SKs:

sk40090: Troubleshooting Check Point logging issues when Security Management Server / Log Server is ...

sk38848: Practical troubleshooting steps for logging issues

[Timothy_Hall]

In addition to what @Dror_Aharony said, usually contention for the hard drive on your SMS/Log Server is the culprit here.  A high Waiting for I/O (wio) percentage displayed by the top command on the SMS is a good indication of hard drive contention.  Do you have SmartEvent enabled on your SMS?  That will typically exacerbate hard drive performance issues. 

Also take a look at the swap numbers on your SMS in the output of the free -m command since if the system is paging/swapping due to low free RAM that will make the wio percentage much higher than it normally would be.

[Gary_Lipets1]

Another possibility is lots of "Alerts"

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[Dror_Aharony]

Best to get the TAC involved as Phoneboy suggested, but also try Timothy's suggestions, most especially run on GW:

cpstat fw -f log_connection

df -h

GW's HW details, please.

Did everything work well before, how long ago did it start?

Do you remember any changes made from around the time it started?