- CheckMates
- :
- Products
- :
- Quantum
- :
- Management
- :
- Estimate data load for logging to splunk
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Are you a member of CheckMates?
×- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Estimate data load for logging to splunk
Hi,
We are installing and configuring NGFW for multiple sites and due to the current splunk configuration, we need to send the log from CheckPoint to a syslog server prior to the splunk environment.
We therefore need to estimate the logging data flowbefore the installation (all solutions to estimate the log size based on CheckPoint interface are then not applicable).
Is there a simple way to estimate the size of the logging flow? Based on the equipment (for example CP5800), number of users (for example 10) and the traffic going through the firewall (for example 10G/sec)?
Thanks for the help!
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Since I don't have a way to test this, what I can give you is the stats from a family of four who making use of our Internet at home.
During the last 24 hours or so, my family of four generated about 30GB of traffic through my gateway, which generated roughly 80,000 logs...with most of the blades enabled and most (not all) things logged.
Using these estimates--and they are just that--I would be exporting 72.5mb a day in traffic via Log Exporter.
However, this is based on the traffic patterns of my family over the course of one day.
Lots of things will impact the real numbers, as I said.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
A "size" of appliance doesn't really tell you how much logs will be generated.
Are you using Log Exporter here or what's the precise configuration?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
For now we are assuming that all the blades of NGFW will be active (therefore not the sandblast ones).
We are using the checkpoint Log Exporter to send the log to the splunk environment via a syslog server (we need the syslog server to ensure the load balancing over the 4 splunk indexers).
As for traffic, is it a more or less linear function? i.e. 10G/s will generate 10x more log than 1G/s?
Thanks for you help @PhoneBoy !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The number of concurrent connections, the exact rules they match and the level of logging for those rules (None versus Log versus Detailed versus Extended) is what will determine the log volume.
While there is also non-user traffic, it's almost guaranteed that user traffic will generate the most logs.
You could probably simulate typical user traffic in the lab for one user and have it accepted on the expected rule they'd hit (e.g. with Detailed or Extended Logs) for whatever period of time you're interested in.
Based on the volume of logs that simulation generates, multiply by the expected number of users and...you have an estimate over that period.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
thank you very much@PhoneBoy - this is valuable information.
Running a test to get the log size for one user presupposes that you already have the CheckPoint infrastructure, at least in a test environment. Assuming we do not, is there any chance that there is a method / estimate for let's say all blades enabled, detailed or extended log policy, 1 user surfing for 1GB traffic?
I understand it is difficult to estimate but we are just looking at ballpark figures.
Thanks again for your help!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In any case, I personally don't have a way to test at this volume.
I can see if we have anything based on QA testing, but your best bet would be to engage your local office with this requirement.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Since I don't have a way to test this, what I can give you is the stats from a family of four who making use of our Internet at home.
During the last 24 hours or so, my family of four generated about 30GB of traffic through my gateway, which generated roughly 80,000 logs...with most of the blades enabled and most (not all) things logged.
Using these estimates--and they are just that--I would be exporting 72.5mb a day in traffic via Log Exporter.
However, this is based on the traffic patterns of my family over the course of one day.
Lots of things will impact the real numbers, as I said.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Many thanks!
We will try to set up a test as you suggested!
