Next best alternative for the SmartEvent - what is the best Open Source option?

[Sergej_Gurenko]

Hello,

Can you please advise some easy-to-deploy and maintain soliton capable of digesting a LARGE number of events, a log-all-all-the-time style solution for at least 1-year retention? The scale is an enterprise with only a few sites and circa 10K users.

The solution should be able to generate some not-so-fancy reports now and again. For example, all users visiting NNN.YYY site last year, How many times MMM.ZZZ was hit every month last year etc. For the reports, we are concerned about quality rather than speed. Instant retrieval is nice but not a must. There is no need for forensic timestamping or threat hunting - just a plain reporting.

The reason for asking for an alternative - at the moment the customer is using the largest dedicated smart-event appliances but reporting quality is not always great. Long-period reports are not working great.

I checked some posts and can see people referring to ELK integrations. ELK looks fairly straightforward to install (if you know what you are doing - example link) However I'm worried about maintenance and support. I have seen Splunk management routines, and it is required to patch OS, Spluk app, and then Apps and resolve all issues with conf files. I never used ELK but worried about maintenance already.

I can see @masher and @Justin_Beavers were discussing ELK in the past. And curious if there is any more information in addition to this:

• Forum post titled Log Exporter and Elasticsearch  
• Partner brief April 2022 Check Point and Elastic | SOLUTION BRIEF 

Regards,

Serg

[the_rock]

Man, thats a tough one, so many out there. I know siem by Elastic is AMAZING. That can can give you such a fantastic visibility and you can literally do any sort of log query/report.

[Sergej_Gurenko]

When sizing Elastic, one need to know the compressing ratio. Do you know what is Elastic compression ratio for the firewall logs?

I'm concerned about some users are telling that logs are growing 30-100% rather then shrinking after the ingestion...

Forum posts:

• https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440
• https://www.reddit.com/r/elasticsearch/comments/kj310s/compression_ratio_between_raw_data_and_indexe...

[the_rock]

Hey Sergej,

Im not 100% positive, but will check with a colleague and update you a bit later.

Cheers mate.

[the_rock]

This is what my colleague told me. Apologies, Im not SIEM person myself, so hopefully it makes sense, but if not, let me know and I will be happy to clarify @Sergej_Gurenko 

****************************

Raw logs stored in Elastic, that's true growing about 30-100% usually, sometime even reaches to about 200%, and that is the primary only, the size will be doubled if replica is enabled and set to 1. Elastic uses LZ4 as default compression algorithm, it can be changed to DEFLATE to get higher compression ratio. The reason is Elastic tokenizes logs to keywords and stored them as inverted index or forward index which makes ES can search things almost real time (default 1 sec delay) but consumed more storage, if only need ES to store raw logs without tokenization and analyzing, I think it will be definitely small than raw logs.

[PhoneBoy]

What version/JHF?
It's worth noticing there's been some significant improvements in R81.10 with regards to speed of SmartEvent.
This assumes the appliances have been fresh installed from ISO to leverage XFS.
R81.20 might even be better.

As for other alternatives, there are a number of different SIEM solutions out there (some Open Source), all requiring various degrees of work to integrate with a Check Point installation.
I'll let others chime in on what they're using and how well it works.

[Sergej_Gurenko]

Thank you for the tips. We are on R81 with old EXT3. We got some real performance improvements when we moved from R77 to R8x but long-term reporting is still an issue. Reimaging from ISO and re-injecting and re-indexing will take some time and effort. We are working with Diamond Support on SmartEvent review.

But still interested in tried and tested/validated alternatives.

[PhoneBoy]

Just so you can see there is a noticeable performance benefit in reimaging with XFS: https://community.checkpoint.com/t5/Security-Gateways/Real-life-comparison-of-XFS-and-EXT3-file-syst...

The installer in R81.20 will likely improve this further by ensuring partitions are aligned to 1mb boundaries.
For related discussion, see: https://community.checkpoint.com/t5/Management/Gaia-partition-misalignment/m-p/160677#M32878 

[Tomer_Noy]

The newer filesystem (XFS) and R81.10 can bring significant performance benefits. If you're already upgrading and performance is critical, it's worth going for R81.20 clean install for the alignment improvement.

It looks like you are using a dedicated SmartEvent, which is important at high scales and long retention.

With regards to appliance, it's worth verifying that you are using the latest gen models (6000-XL) which have SSDs and utilize a good RAID configuration. When creating reports over long periods, disk speed becomes important as not all indexes will fit in memory.

Last, it might help to understand your average log rate to get an indication of how much data is being scanned.

[the_rock]

Hey @Tomer_Noy ,

Just curious, what is absolutely BEST appliance CP recommends to customers to use as dedicated smart event?

Andy

[Bob_Zimmerman]

Best appliance, I can't answer, but no appliance can hold a candle to a VM on a host with decent NVMe storage. Even with 1:2 lane oversubscription, PCIe 4 NVMe drives can manage ~3.2 GB/s each. PCIe 5 drives (which are available now) can do ~6.5 GB/s with just two lanes. Assuming perfect throughput scaling, it would take six SATA SSDs with no redundancy to keep up with a two-lane PCIe 4 NVMe drive or twelve SATA SSDs to keep up with a single two-lane PCIe 5 NVMe drive. Again, that's with zero redundancy, so if you lose a drive, you lose the set. And there are systems which can take 24 PCIe 5 NVMe drives without oversubscription. Even with the virtualization hit to I/O performance, that advantage is too big to overcome.

I've never found Check Point's branded boxes a good value, but the firewall boxes are at least halfway tolerable. The management servers are a decade out of date.

[the_rock]

Thanks @Bob_Zimmerman 

[Sergej_Gurenko]

We checked the SE servers and it looks like even at R81 they started using XFS at least for the log partition which is what we care about for indexes and raw logs:

Filesystem                      Type    Mounted on

/dev/mapper/vg_splat-lv_current xfs     /

/dev/sda1                       ext3   /boot

tmpfs                           tmpfs  /dev/shm

/dev/mapper/vg_splat-lv_log     xfs    /var/log   <=used for about 75%

cgroup                          tmpfs  /sys/fs/cgroup

[PhoneBoy]

Actually, XFS support was added on any system that uses the Linux 3.10 kernel.
In maintrain, this was added to R80.40, but it also exists on some R80.20/R80.30 ISOs for specific appliances.

[G_W_Albrecht]

Why not feed the logs into a database ? You then can create your own reports and do simple stats as mentioned.