Logging with limited log samples?

[Mille]

Hi Mate,

Do you know any way to enable limited logging on an access rule, say 5% rule hits are logged and the rest is not logged?

Why?
A typical access rule will have log enabled and all matches for that rule will be logged.
Some types of trafic (like DNS, NTP, SNMP and NETBIOS) will generate a lot of hits and you may opt to disable log for this 'noise'.
However that will leave you 'blind' both in terms of the direct trafic and in terms of statical reporting and other data processing.

BR Mille

[Amir_Senn]

There is no partial logging.

Best suggestions I can offer are:

a. You can create build in filter that will filter unwanted logs from results

b. Identify all the features of the noisy logs and instead of the current rule, replace action with layer. Under this layer, create the first rules with a combo of source, destination and service and don't log those. If you keep it well defined I don't think you should have issues.

[Mille]

Amir Senn, Thank you for the suggestions. I was hoping for some kind of partial logging.

/Mille

[Amir_Senn]

You can also change the tracking options on layer suggestion to be session instead of connection. Should lower number of all logs that match the rule.

[Tomer_Noy · Accepted solution]

Actually, there is a feature that is specifically meant for this use-case and allows you to reduce the noise of highly repeating logs.

It's not called "partial logging", but "Session Logs".

If you open the Track settings on your rule with logging, you will see the default logging configuration:

These default settings tell the gateway to create a new log for each connection attempt. In the case of DNS, indeed that means many logs that look the same as machines create a DNS request (and connection) for every DNS resolving.

You can of course switch that rule to Track=None, but that indeed leaves you blind to all this traffic and requests.

Instead, you can modify the Track settings to look like the below:

Deactivate "per Connection" logs and activate "per Session" logs.

Session logs are an aggregation of all connection logs that have the same significant parameters (source, destination, port, action, user, ...). Once a new connection is opened a session log will be sent. Subsequent connections (that share the same significant parameters) will not send another log, instead every 10 minutes a log update will be sent on the session to state how many connections were seen in that time period. That way, if you double click a session log, you can still see how many connection, without having a dedicated log row for each connection.

This reduces the noise and also reduces the load on your log server. We've seen customers reduce their total logging by 30%-60% by utilizing Session logs on noisy rules.

[Mille]

Thanks for the solution. It work for me.

In my case I also had to make the same adjustment in both the Network access rule and in the Application rule as the policy does not use Layes.

[Tomer_Noy]

Great to hear 😀

I'm curious, by how much did that reduce your logging on those rules?

For example, from 100K logs per day to 10K?

[Mille]

Work in progress. An early result is 30K DNS connection logs is reduced to 6K session logs for 24/hours. It's about 80% reduction.