topic Re: Estimate data load for logging to splunk in Firewall and Security Management

Estimate data load for logging to splunk

hutinop — Wed, 01 Apr 2020 11:43:24 GMT

Hi,

We are installing and configuring NGFW for multiple sites and due to the current splunk configuration, we need to send the log from CheckPoint to a syslog server prior to the splunk environment.

We therefore need to estimate the logging data flowbefore the installation (all solutions to estimate the log size based on CheckPoint interface are then not applicable).

Is there a simple way to estimate the size of the logging flow? Based on the equipment (for example CP5800), number of users (for example 10) and the traffic going through the firewall (for example 10G/sec)?

Thanks for the help!

Re: Estimate data load for logging to splunk

PhoneBoy — Fri, 03 Apr 2020 00:02:03 GMT

Logging is a function of three things: Traffic, blades enabled, and your precise policy configuration, all of which determine what is actually logged.
A "size" of appliance doesn't really tell you how much logs will be generated.
Are you using Log Exporter here or what's the precise configuration?

Re: Estimate data load for logging to splunk

hutinop — Fri, 03 Apr 2020 09:04:07 GMT

For now we are assuming that all the blades of NGFW will be active (therefore not the sandblast ones).

We are using the checkpoint Log Exporter to send the log to the splunk environment via a syslog server (we need the syslog server to ensure the load balancing over the 4 splunk indexers).

As for traffic, is it a more or less linear function? i.e. 10G/s will generate 10x more log than 1G/s?

Thanks for you help @PhoneBoy !

Re: Estimate data load for logging to splunk

PhoneBoy — Fri, 03 Apr 2020 21:10:14 GMT

No, because a single, 10GB stream will generate less logs than, say, 1,000 users surfing the web will, even if the aggregate bandwidth they use is less than 10GB.
The number of concurrent connections, the exact rules they match and the level of logging for those rules (None versus Log versus Detailed versus Extended) is what will determine the log volume.

While there is also non-user traffic, it's almost guaranteed that user traffic will generate the most logs.
You could probably simulate typical user traffic in the lab for one user and have it accepted on the expected rule they'd hit (e.g. with Detailed or Extended Logs) for whatever period of time you're interested in.
Based on the volume of logs that simulation generates, multiply by the expected number of users and...you have an estimate over that period.

Re: Estimate data load for logging to splunk

hutinop — Mon, 06 Apr 2020 07:36:19 GMT

thank you very much@PhoneBoy - this is valuable information.

Running a test to get the log size for one user presupposes that you already have the CheckPoint infrastructure, at least in a test environment. Assuming we do not, is there any chance that there is a method / estimate for let's say all blades enabled, detailed or extended log policy, 1 user surfing for 1GB traffic?

I understand it is difficult to estimate but we are just looking at ballpark figures.

Thanks again for your help!

Re: Estimate data load for logging to splunk

PhoneBoy — Mon, 06 Apr 2020 14:57:16 GMT

Detailed versus Extended Logs can make a huge difference in log volume.
In any case, I personally don't have a way to test at this volume.
I can see if we have anything based on QA testing, but your best bet would be to engage your local office with this requirement.

Re: Estimate data load for logging to splunk

PhoneBoy — Mon, 06 Apr 2020 20:53:56 GMT

What R&D told me was that each log record is roughly 850 bytes to 1000 bytes when exported with Log Exporter, with the average being 950.

Since I don't have a way to test this, what I can give you is the stats from a family of four who making use of our Internet at home.
During the last 24 hours or so, my family of four generated about 30GB of traffic through my gateway, which generated roughly 80,000 logs...with most of the blades enabled and most (not all) things logged.

Using these estimates--and they are just that--I would be exporting 72.5mb a day in traffic via Log Exporter.
However, this is based on the traffic patterns of my family over the course of one day.
Lots of things will impact the real numbers, as I said.

Re: Estimate data load for logging to splunk

hutinop — Tue, 07 Apr 2020 08:05:16 GMT

Many thanks!
We will try to set up a test as you suggested!