I was working with an organization recently who wanted to log OSPF transition events on their Check Point gateways to syslog so that they could alert on them using the same method as their routers and switches. After turning on Remote System Logging in Gaia Web UI, their syslog server was getting inundated with Check Point messages. The one gateway that was enabled was accounting for well over 30% of their total logs, and this was the first of 30+ gateways in a device pool of several hundred devices. Needless to say, they didn't want all the noise in syslog but what to do?
I'll save you all the pain of most of the trial and error and just get to the good stuff. The end result is that we successfully are now logging only events that transition from FULL -> something else, or return back to FULL status and this is exactly what the customer needed.
- To begin, we know that OSPF events are logged to /var/log/routed.log and we will need to extract them from there.
# tail -n 0 -F /var/log/routed.log
- And, we know that we only want transition events where there has been a neighbor change to/from FULL state
# tail -n 0 -F /var/log/routed.log | grep -E 'FULL ->|-> FULL'
But now how to get it to syslog?
You may or may not be familiar with the command logger, but simply put: logger makes entries in the system log. Read all about it here.
So now we put it all together and it works, right?
# tail -n 0 -F /var/log/routed.log | grep -E 'FULL ->|-> FULL' | logger
Not quite. It's moving messages to syslog, but they are showing up as info messages. Configuring the Remote System Logging to send only info messages still sends an inordinate amount of chatter. But there is hope, in the form of the switch -p / --priority.
- So now we add the classification to make it an emergency message and configure Remote System Logging to send only emergency messages.
# tail -n 0 -F /var/log/routed.log | grep -E 'FULL ->|-> FULL' | logger -p syslog.emerg
Almost there. This is simple enough to run at the command line and does what we want it to, but we need it automated. We don't want to have to start this manually, however, so we need to add this as a schedule job.
Weird behavior, and I didn't troubleshoot this at all, but for some reason the command above does not work as a scheduled job. I think it's because of the multi-pipe, and as such devised a two-step approach to accomplish the same thing.
# tail -n 0 -F /var/log/routed.log | grep --line-buffered -E 'FULL ->|-> FULL' >> /var/log/OSPF_script.log
*Note the addition of the --line-buffered switch on the grep command, and this is because we are now appending to a file.
# tail -n 0 -F /var/log/OSPF_script.log | logger -p syslog.emerg
Run both jobs on startup and reboot, then voila! Now OSPF FULL transition events are being sent to the syslog server of choice. Many thanks to Jian Wu, @Sundeep Mudgal, and @Thurston Zhu for working through the tail and grep commands with me until we got the right combo :-)
This may not be the most elegant way to do this, so comments are always appreciated. Ultimately, the solution works and the customer is happy so I figured I would share with the world since we couldn't find anything similar online and asking around yielded some suggestions but nothing concrete.