Log Exporter Filtering

[Dan_Zada]

Hello all,

I'm happy to inform you that we added a new feature to the log exporter - the ability to filter logs.

Starting today, you will be able to configure which logs will exported, based on fields and values, including complex statements.

More information, including basic and advanced filtering instructions, can be found in SK122323.

If you have any question or comment, let me know.

Thanks!

Dan.

[Dror_Aharony]

in the next public R80.30-JumboHF (hopefully a few weeks).

[Maarten_Sjouw]

Ok thanks.
Another question I got was is log exporter following the rfc5424 format?

[Garrett_DirSec]

@Maarten_Sjouw a quick google search shows RFC5424 is simply syslog protocol (I didn't know this off top of head).

https://tools.ietf.org/html/rfc5424

answer:  yes.   check the "formats" section below.

reference:

Log Exporter - Check Point Log Export

http://supportcontent.checkpoint.com/solutions?id=sk122323

Background:

Log Exporter is a multi-threaded daemon service, running on a log server. Each log that is written on the log server is read by the log exporter daemon, transformed into the desired format and mapping, and then sent to the end target.

Log Exporter supports:

• SIEM applications: Splunk\Arcsight\RSA\LogRhythm\QRadar\McAfee\rsyslog\ng-syslog and any other SIEM application that can run a syslog agent.
• Protocols: syslog over TCP or UDP.
• Formats: Syslog, CEF, LEEF, Generic.
• Security: Mutual authentication TLS.The ability to export logs/audit or both.
• Filter out (don't export) firewall connections logs.

[genisis__]

I've now tried this, and also contacted TAC.  LogExporter cannot be used on a dedicated SmartEvent appliance. Logexporter currently does not support indexed logs, however TAC told me that this is in the pipeline.

[Garrett_DirSec]

interesting to hear this (and thanks for posting update). 

It's my understanding the Log Exporter is a service that works directly on Log Server (whever that may reside).

I do understand the requirement for SmartEvent users to push alerts into 3rd party system (SIEM, SOC, etc) and it does make sense that CP would leverage something like Log Exporter for this (and replace the current duct tape and bailing wire methods). 

[genisis__]

I'm still working with TAC on this, but in my case the objective is to send log data for specific gateways and for TP only which maybe managed by different domain servers to a central SIEM collector for the organisation.

My logic was, rather then burdening the Provider with multiple streams to the collector, send logs from the SmartEvent appliance for the specific gateways (single stream) and logs are already consolidated for TP events. In this way only filtering to do now and in the future would be to maintain the list of GWs in the filter.

Clearly not happening...well not yet.

[Tommy_Forrest]

Howdy, couple of things.

First, I think the SK article needs to be updated to reflect 80.40 changes.  Was trying to find the xml file to update our log exporter configuration and the path no longer exists.

cd $EXPORTERDIR/targets
bash: cd: /opt/CPrt-R80.40/log_exporter/targets: No such file or directory

Also, 80.40 upgrade wiped out our exporter rules.  Not cool.

Finally, what is the current best practice for including CloudGuard instances in a Origin filter?

Do I just statically add the current hosts to the exporter rule?  Or is there some way to include the dynamic nature of virtual machine scale sets and their resultant names into the exporter configuration?

[Dror_Aharony]

Did you run (post upgrade): cp_log_export reconf

it updates the log-exporter to current version, then you'll see it.
You mean the log-exporter sk122323 isn't updated, what exactly are you referring to?
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

That's a good question regarding the dynamic/CG GWs origin filtering.
I'll try to check that (Shay Hibah, FYI).

[Jeff_Gao]

Dear team
There is a Error as blow when i configure only send "TP" log to syslog server,this is why?
# cp_log_export set name TP_Test filter-blade-in "TP"
Error: Argument [filter-blade-in] is undefined for command: [set]

[Dror_Aharony]

which version is this? Did you try other filters?

I'd guess it's not supported on your version. 

see log-exporter sk122323 for supported versions:

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[pmiprojet_pmipr]

R80.30 log server (with smartevent ) +

       HOTFIX_R80_30_JHF_T111_LOGRHYTHM_MAIN  Take: 4

I'm having problems filtering log_export while the pipe between CP and Splunk is working.
I find it very easy to throw away a feature without customers being able to easily apply it, especially when the feature has been expected for months.
The documentation is confusing and we always have to spend a monster amount of time with expert support to get through it.
at this point basic questions remain:
- following the advanced configuration given as an example, the FilterConfiguration.xml file is fed by the commands but still no filters on the exported logs and I receive the access type logs from the firewall.

[Shay_Hibah]

Hi @pmiprojet_pmipr 

I am sorry to hear that it took long time to configure Log Exporter and filtering feature.

I would like to help you and guide you so you will be able to use this feature ASAP.

Could you please send me email and we will take it from there? my email address is shayhi@checkpoint.com

Regards,

Shay

[Shay_Hibah]

Hi Himanshu,

Can you please send me cpinfo output from the specific server to my mail and we will take it from there?

shayhi@checkpoint.com

Thanks

[AndyJ]

Hi,

Are there any plans for this to be configured with a GUI interface? As mentioned by others this process is not as straight forward as it is made out to be. For something that should be a "configure once" scenario it takes a lot of wasted time to configure. Having a GUI front end where you could select what fields from what blades you wanted to export and what SIEM you were using which would then ensure that the correct formatting and configurations were applied would be a lot more customer friendly.  

Filtering does not work properly. I had a call open with Checkpoint for a month trying to get this configured and it still isn't. We just decided to give up in frustration because the Checkpoint resource didn't know what to do either. I spent days more on my own trying to work through it. When you have a lot of other work to do it's frustrating to have to spend valuable time trying to work out something like this that should be straight forward.

We had to configure export all and then blacklist because some fields weren't being passed through with the "export none" and whitelist approach. This means long winded xml files need to be created to filter out all unwanted fields.

If i try to use the whitelist approach the rule fields (rule_uid, rule_name,etc) don't come through. If i try to use the blacklist approach the layer_uuid, layer_name, etc.. and other fields don't filter out. 

It would be great to just be able to tick some boxes and have the logs files flow through to the SIEM in the correct format. This would also save your staff valuable time trying to troubleshoot customer issues.

Thanks

[Shay_Hibah]

Hi @AndyJ

Thanks for reaching us.

Regarding your issues:

1. UI for Log Exporter is part of our plan. I am not sure exactly what features will be included in the first version but we will try our best to support as most as possible features of Log Exporter. For unsupported features by UI, it is still possible to configure them manually.

2. Regarding your issue with the filtering, I do sorry to hear you spent so much time in it and I would like to help.

Can we you please send me an email (shayhi@checkpoint.com) an we will take it together from them?

I will be also glad to hear a feedback so we can improve it.

Thanks,

Shay

[genisis__]

There is a general theme here.

I've gone through the same pains as well;  the documentation is not very clear, all of which has been fed back to TAC.

In an MDS setup I've noted that UDP traffic that should be using a CMA's IP actually uses the lead IP of the MDS to send traffic (By design I'm told...yeah right!)

If you specify TCP then the correct source IP is used (by design!).

The below ensures that Threat Prevent Logs are sent to a syslog server for two specific GWs.  The unified parameter is meantioned once in the SK as a note, but this ensures an entire event is recorded as one log entry.

cp_log_export add name TestSYSLOG domain-server CUSTOMERCMA target-server 1.1.1.1 target-port 514 protocol tcp format syslog
cp_log_export set name TestSYSLOG domain-server CUSTOMERCMA read-mode semi-unified
cp_log_export set name TestSYSLOG domain-server CUSTOMERCMA filter-blade-in "TP" filter-origin-in "<GW1 IP>,<GW2 IP>"
cp_log_export restart name TestSYSLOG domain-server CUSTOMERCMA

Also note that information record from R77.x gateways may be slighly different so you will need to check the logs sent to do any further tuning.

[Shay_Hibah]

Hi @genisis__
Regarding the different behavior of TCP and UDP in Multi Domain environment - it is something we are familiar with and we do need to handle it. This behavior caused by an infrastructure change that affected Log Exporter as well other features.

I want to help you but I still do not understand how - can you please elaborate what the problem is? Maybe I can suggest solution for your specific issue.

Thanks,
Shay

[Pathy]

Hi,

I set up cp_log_export to send FW logs to splunk but no data is being sent

Expert@fw-logsrv:0]# tail -f log_indexer.elg
[log_indexer 86745 3915397952]@fw-logsrv[28 Apr 16:16:44] Sent current: 0 average: 0 total: 0

[log_indexer 86745 3915397952]@fw-logsrv[28 Apr 16:16:49] Files read rate [adtlog] : Current=0 Avg=0 MinAvg=0 Total=5 buffers (0/0/0/0)

[log_indexer 86745 3915397952]@fw-logsrv[28 Apr 16:16:49] Sent current: 0 average: 0 total: 0

[log_indexer 86745 3915397952]@fw-logsrv[28 Apr 16:16:54] Files read rate [adtlog] : Current=0 Avg=0 MinAvg=0 Total=5 buffers (0/0/0/0)

[log_indexer 86745 3915397952]@fw-logsrv[28 Apr 16:16:54] Sent current: 0 average: 0 total: 0

[log_indexer 86745 3915397952]@fw-logsrv[28 Apr 16:16:59] Files read rate [adtlog] : Current=0 Avg=0 MinAvg=0 Total=5 buffers (0/0/0/0)

[Expert@fw-logsrv:0]# cp_log_export show

name: splunk_log
enabled: true
target-server: 172.31.20.7
target-port: 514
protocol: tcp
format: splunk
read-mode: semi-unified
encrypted: true

[Expert@fw-logsrv:0]# cp_log_export status

name: splunk_log
status: Running (41112)
last log read at: 28 Apr 17:56:15
debug file: /opt/CPrt-R80.30/log_exporter/targets/splunk_log/log/log_indexer.elg

[Expert@fw-logsrv:0]#

I also saw this message below in log_indexer.elg 

log_indexer 86745 4127701376]@fw-logsrv[28 Apr 15:38:51] FormatConfiguration::init: formatHeaderFile is not configured for type: splunk, applying default : conf/SplunkFormatDefinition.xml

Any idea?

Thanks 

Pat

[AndyJ]

Hi Shay,

Thanks for you response. Ironically i did manage to get it working a coupe of hours after i left the post. I ended up deleting it entirely and starting from scratch. I also found a post from Yonatan that specifically mentioned about configuring the layer_uuid filtering.

I look forward to seeing the GUI interface for the solution

Cheers,

Andy

[Shay_Hibah]

Hey Pat,

This issue needs a further investigation.
Would you like to take it offline with me and I'll try my best to help?

Shay

[Pathy]

Hi Shay,

Yes sure, how do we proceed?

Pat

[Shay_Hibah]

Please send me a mail to: shayhi@checkpoint.com.
I will be happy if you can attach to this mail log_indexer.log* files located under your exporter dir $EXPORTERDIR/targets/<exporter_name>/log/

[ZP]

Hi Guys,

I'm trying to configure filter in my syslog profile. I only want to send the logs from/to sprcific subnet. I have configure as below but it doesn't send the syslog.

<filters>
<filterGroup operator="or">
<field name="src" operator="or">
<value operation="eq">192.168.1.0/24</value>
<value operation="eq">192.168.2.0/24</value>
<value operation="eq">192.168.3.0/24</value>
</field>
<field name="dst" operator="or">
<value operation="eq">192.168.1.0/24</value>
<value operation="eq">192.168.2.0/24</value>
<value operation="eq">192.168.3.0/24</value>
</field>
</filterGroup>
</filters>

I have restarted but it doesnt works.

Please advise is there anything I missed out ?

[mjovovic]

Hello, Did You make this work?

[Antonis_Hassiot]

Did you manage to get the IP filtering working?

Tried myself but I get no logs at all. 

name: splunk
enabled: true
target-server: 10.xx.x.53
target-port: 12010
protocol: tcp
format: splunk
read-mode: semi-unified
export-link: false
export-attachment-link: false
export-attachment-ids: Found

<filters>
<filterGroup operator="or">
<field name="src" operator="or">
<value operation="eq">10.x.x.9/32</value>
</field>
</filterGroup>
</filters>

I have tried several different things, but I get no logs at all. 

[Shay_Hibah]

Hey @Antonis_Hassiot 

Can you please explain what are you trying to do so I will be able to assist?

[Antonis_Hassiot]

As we need to export our logs to a splunk server for review, we need to filter the logs by relevant source or destination IP subnets. i.e. only server related IP subnet source/destinations, not user ones. So I need to create such export filter, but using field name 'src' or 'src_ip' doesn't seem to work in the simple example above. Exporter doesn't send any logs to splunk when I apply the filter shown. 

[Shay_Hibah]

Hey @Antonis_Hassiot 

Subnets unfortunately are not supported yet in filter configuration file.

It is possible to specify white-list or black-list but not a grouped one as a subnet.

[Antonis_Hassiot]

I have tried the following statement: <value operation="eq">10.x.x.9</value>, i.e. written as single IP in my filter XML, but no logs at all get through when I apply. 

Can you provide a working example of a filter XML file on how I would allow specific IP sources or destinations through?

 Also in the documentation, I thought I read subnets are possible. Are you certain they are not?