How many log files a day is usual

[Trevor_Bruss]

I realize this is a very subjective question as it all depends on your environment, but I've noticed a dramatic rise in log files over the years as we've migrated from an R77 environment to R81.10.

We log basically two datacenters in our single log server with one of them getting the majority of the Internet traffic inbound/outbound. We have the server set to rotate the logs every 2 GB. We probably are generating one 2 GB file every 20-30 minutes. Yesterday alone we generated 52 2GB log files. And so far on our 4 TB log server we've generated 2.3T of fw logs since Jan 13.

Just to add context, I opened a support case back in October 2016. At that time we were struggling because we were generating 24GB of logs a day. And at that time that was a significant increase from what we had seen in the past. Mind you, this is the same datacenter so the number of users has stayed constant over the years for the most part. We were able to get to about half of that by not logging telnet drops before the final drop rule.

Since then its risen to 40-50 GB a day (without the final drop rule log) and with the drop rule logging it's almost double that. I would say 66% of the logs are inbound Internet drops from our major datacenter. I used to be able to get a better count by using the older SmartView Tracker application and opening up a single file for analysis, but the log files being generated now don't play so well with that tool anymore. I just don't understand why several years ago we saw this dramatic increase and it has only gotten worse. Each 2 GB file is made up of approximately 8 million entries (based on SmartView tracker). So that's almost 260 bytes of data per entry and roughly 320,000 entries a minute.

Is this just normal? I know there isn't an exact size on log entries produced, but I just feel like we're drowning in data.

[the_rock]

Hey Trevor,

Personally, thats not normal, in my opinion. Here is what I would suggest...this is just my guess, as I dont know anything about your environment, apart from what you mentioned here. Can you tell which rules have highest  hit number? Can you disable logging for those rules and see the results after policy push?

Andy

[Trevor_Bruss]

As mentioned previously, a good majority of logs are generated just on our cleanup rule. Yesterday, I changed the cleanup rules so that traffic from the Internet to our internal networks would be dropped but not logged, and our final cleanup rule now is only traffic from internal going to the Internet being dropped and logged. I've kind of been running it in this mode before. Today (half way through the day) we've now only generated 23GB of logs. So, approximately 46GB of logs will be generated today, which I still consider a lot, but far better than 85-100GB a day.

Will not logging Internet inbound drops affect anything? I'm not sure. From a troubleshooting standpoint, I'm mostly concerned with traffic going out. From a security standpoint, I don't know if not logging this traffic affects any of our reports. That is, would SmartEvent not be able to detect things like ping sweeps from outside, etc. or do we even care if the traffic is getting dropped. That is, should I be more concerned about what is getting in from the Internet that is unwanted than worrying about what's beating at our door but can't get in?

[CE_SE]

Not logging inbound drops won't impact your environment, but interested in what your internet bandwidth usage looks like starting on Jan 13th to see if there is any correlation between firewall traffic and increased bandwidth utilization on your internet circuits. I would personally start with tracking down whom are the largest talkers in/out of the environment to understand the traffic flow/patterns. From that point I think you can follow other contributors ideas as it relates to NULL routes, Geo Policy, etc.

[CE_SE]

You can also review the following to limit traffic. 

Advanced Track options will increase the log file size.

Detailed Log and Extended Log are only available if one or more of these Blades are enabled on the Layer: Applications & URL Filtering, Content Awareness, or Mobile Access.

• Detailed Log - Equivalent to the Log option, but also shows the application that matched the connections, even if the rule does not specify an application. Best Practice - Use for a cleanup rule (Any/Internet/Accept) of an Applications and URL Filtering Policy Layer that was upgraded from an R77 Application Control Rule Base.
• Extended Log - Equivalent to the Detailed option, but also shows a full list of URLs and files in the connection or the session. The URLs and files show in the lower pane of the Logs view.

Log Generation you have the ability to limit size by doing per session instead of the default per connection. 

• per Connection - Select this to show a different log for each connection in the session. This is the default for rules in a Layer with only Firewall enabled. These are basic firewall logs.
• per Session - Select this to generate one log for all the connections in the same session. This is the default for rules in a Layer with Applications and URL Filtering or Content Awareness enabled. These are basic Application Control logs.

Also, since you have a date I would review all changes associated with the firewall policy to see if any type of rules/configuration item changed that has caused this increase in log file size generation. 

[Trevor_Bruss]

We do have some of our outbound web rules that are set for Extended Logs. We wanted to be able to capture URLs that users are going to as we get the occasional request from managers, etc. as to what sites their users are browsing. I'm not certain how to provide that data as we're not running an internal proxy for them to use. (Perhaps something we should look into). Most rules however are set to Log only. 

We also are coming from migrating from older versions of Checkpoint over several years and our Access Control policy contains multiple layers that include App and URL filtering. Initially, the security layer was just firewall but we've been trying to move some of what we did in the application layer into the security layer now that we're on more current versions of Checkpoint (R81.10).

I've looked at changing some rules to Session logs only, but I'm not a fan of them because I lose out on seeing any NAT information in them which at times is crucial for troubleshooting. That leaves us with doing per Connection logs on many of those rules.

[Chris_Atkinson]

Do you employ any upstream filtering at your edge routers e.g.

* Bogon filters

* Null routing unused / dormant public facing subnets 

* Geo blocking

[Trevor_Bruss]

I've reached out to the networking team, but my guess is no on all accounts. I had to look into what bogon filters were. Just based on our logs when searching for Address Spoofing as well as some non-private address ranges, I really don't see any high number of logs generated, but I didn't go through the full bogon list either.

As for geo blocking, we're just doing that today on the Checkpoint gateways.  In some cases, we don't even log the drop. So I don't know if we're gaining a lot from a logging perspective by filtering sooner.