Aggregate log updates before export Log Exporter option

[imamuzic]

What does "Aggregate log updates before export" option exactly achieve?

Best Regards,

Igor

[G_W_Albrecht]

See https://sc1.checkpoint.com/documents/R81/WebAdminGuides/EN/CP_R81_LoggingAndMonitoring_AdminGuide/To... for an explanation:

• 2. Optional: Select Aggregate log updates before export to export all logs with the full data.

     By default, update logs contain the data that was changed compared to the last log for the same event 

[imamuzic]

Thanks, but already saw this and I'm actually looking for some clarification, for example, does it means that the Log Exporter will accumulate all the logs belonging to this particular session and then forward a single syslog message containing information from all connection events that form this particular session or something like this?

[Amir_Senn]

Session logs are getting updates until session is ended/enough time passes for a new log session to replace it (I think 3 hours is the default limit AFAIK).

A user is surfing the same site/application. With time, more and more information is relevant (number of connections, files/URLs used, accounting data etc.). Holding this will send you final data instead of data at the point of the creation of the log or latest update of the log.

[imamuzic]

Hi Amir, 

What are you explaining are actually differences between session and connection logging in Smart Log, but does that explanation applies to the "Aggregate log updates before export" Log Exporter option?

[PhoneBoy · Accepted solution]

Yes, it does.
Without this option, logs are sent when the connection is open and every 10 minutes thereafter until the session is over when one final log is sent.
With this option, only a single log entry is sent AFTER the session is over. 

[imamuzic]

Thank you for the answer... This is what I was looking for.

[Tomer_Noy · Accepted solution]

The actual behavior is different, so the above answer is partially inaccurate...

There is a bit of a mix-up between connection & session logs, and the Log Exporter aggregation setting.

1) Connection Logs - These are firewall access logs that are sent for every single connection.
2) Session Logs - These are "higher level" logs that combine multiple connection logs to a single session as long as they have identical matching attributes (source, destination, port, action, ...).

Both of these log types may receive updates.
A connection log is created immediately when a connection is first seen. Some connections are long lived and the gateway may gather more information about them over time. For example, identifying the user that originated the connection or how much traffic has passed so far (if Accounting is activated). This additional information is sent as partial log updates that only contains the fields that changed.
A session log is created immediately when the first connection of its type is created. Following that, the gateway will accumulate information from all connections that match this session over a default of 10 minutes, and will issue an update log for the session. This update log will usually include how many connections matched and possibly accounting information.

The log exporter has two options for exporting:
1) Raw (not unified) mode - in this case, it exports the fields that exist on the log. If this is an update log, you will have an exported log that has just a handful of fields.
2) Aggregated (semi-unified) - in this case, it still exports every arriving log, but before doing so, it will fetch the previous data it has for that log and will unify it into a single complete log (up to that point).

Both options will export every log, but the second is more useful since every log shows a complete picture of the connection / session. Also, in both options you need to be aware that you will have multiple logs in your SIEM for the same connection.

Unfortunately, we don't have an option today for a "full unified" mode where we only send the last log. This is mainly because the log server doesn't have a final indication that no more future updates will arrive. We are looking into adding this in our roadmap.