Log Exporter guide

Yonatan_Philip · ‎2018-03-22

Hello All,

We have recently released the Log Exporter solution.
A few posts have already gone up and the full documentation can be found at sk122323.

However, I've received a few questions both on and offline and decided to create a sort of log exporter guide.

But before I begin I’d like to point out that I’m not a Checkpoint spokesperson, nor is this an official checkpoint thread.

I was part of the Log Exporter team and am creating this post as a public service.

I’ll try to only focus on the current release, and please remember anything I might say regarding future releases is not binding or guaranteed.
Partly because I’m not the one who makes those decisions, and partly because priorities will shift based on customer feedback, resource limitations and a dozen other factors. The current plans and the current roadmap is likely to drastically change over time.

And just for the fun of it, I’ll mostly use the question-answer format in this post (simply because I like it and it’s convenient).

Log Exporter – what is it?

Performance

Filters

Filters: Example 1

Filters: Example 2

Gosh darn it, I forgot something! (I'll edit and fill this in later)

Feature request

Yonatan_Philip · ‎2018-08-14

Hello Su,

From your log (Current=13 Avg=139 MinAvg=10 Total=331020 ) as well as the status command it appears that logs are being exported.

If you want to actually see this you can use tcpdump command: 'tcpdump port 514 -A -s0' (if you are using port 514 for anything else, you can add other qualifiers to narrow down the output).

This will show you the actual data being exported in a readable format. For example:

[Expert@ypsa:0]# tcpdump port 514 -A -s0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:25:25.094828 IP ypsa.47206 > XX.XX.XX.XX.syslog: SYSLOG local0.info, length: 1044
E..0..@.@..xd P...f......<134>1 2018-08-14T14:25:23Z ypsa CheckPoint 17857 - [action:"Accept"; ifdir:"inbound"; ifname:"eth0"; [deleted the payload] product:"VPN-1 & FireWall-1"; proto:"6"; s_port:"35700"; service:"22"; service_id:"ssh"; src:"XX.XX.XX.XX"; ]
1 packets captured
2 packets received by filter
0 packets dropped by kernel
[Expert@ypsa:0]#

(I deleted most of the payload since it just takes up space and not really relevant for this example - I just wanted to show that you can see and read the actual logs as they are being exported)

Since it looks like your logs are actually being exported, I would focus on the other end and try to see if it's being received and parsed correctly.

Use tcpdump or Wireshark on the other end. If it's not there, it's a connectivity issue, and if it's there it's probably a parsing issue.

HTH

Yonatan

Bogdan_Kirylyuk · ‎2018-08-16

Hey Yonatan,

Glad I found this post

Please let me know where it stands for supporting JSON format output.

My X-Pack charged ELK is waiting to search thru this data and I am currently working on parsing it myself which is not an easy task so far.

I believe JSON format will be friendly to work with in python scripts and rest API as well

Yonatan_Philip · ‎2018-08-16

HI Bogdan,

I don't think that official json output will be added anytime soon - there are currently many other items ahead of it in the queue.

However, earlier in this thread, I showed how you can edit the settings to generate json like output.

https://community.checkpoint.com/message/25414-re-log-exporter-guide?commentID=25414&et=watches.emai...

It does have the disadvantage of potentially having duplicate keys, but a fast google search returned some answers on how to deal with that:

Elasticsearch, Kibana and duplicate keys in JSON -

Please let me know if this worked.

Yonatan

Yonatan_Philip · ‎2018-09-04

Product (Blade) names.

As you might have noticed the names that are exported are not the same as the ones which appear in the GUI.

This is because each blade has a display name which is shown in the GUI and an actual value which appears in the raw log. Those names are often the same but not always.

For technical reasons, it's very difficult to change the actual value that the gateway sends but easy to change the display name. So while the actual value and the display name usually start the same, the values might drift over time.

This has caused a bit of confusion among some customers.

So as a sort of public service here is the current mapping of raw log values to display name.

This list also includes some legacy names. Frankly, there were a few names on this that I've never heard of and didn't know existed

Field Name	Anti Malware	Core	Identity Awareness	SmartView MonitorMonitor
Display Name	Anti-Bot	Core	Identity Awareness	SmartView Monitor
Field Name	Anti-Malware	DefensePro	Identity Logging	Syslog
Display Name	Anti-Malware	DDoS Protector	Identity Logging	Syslog
Field Name	Anti-Exploit	DLP	Management Blade	System Monitor
Display Name	Anti-Exploit	DLP	Management Blade	System Monitor
Field Name	Anti-Ransomware	Content Awareness	MEPP	Threat Emulation
Display Name	Anti-Ransomware	Content Awareness	Media Encryption & Port Protection	Threat Emulation
Field Name	WIFI Network	Edge AV	Connectra	Threat Extraction
Display Name	WIFI Network	Edge AV	Mobile Access	Threat Extraction
Field Name	Mobile App	Compliance	Policy Server	Anti Virus
Display Name	Mobile App	Endpoint Compliance	Policy Server	Traditional Anti-Virus
Field Name	Network Security	Integrity	Web Filtering	UAG
Display Name	Network Security	EndpointEndpoint Security	Legacy URL Filtering	UA Server
Field Name	OS Exploit	Everest	CVPN	WebAccess
Display Name	OS Exploit	FireWall-1 GX	CVPN	UA WebAccess
Field Name	Device	Firewall	FG	URL Filtering
Display Name	Device	Firewall	QoS	URL Filtering
Field Name	Text Message	VPN-1 & FireWall-1	rtm	VPN-1 Edge
Display Name	Text Message	Security Gateway/Management	Real Time Monitor	UTM-1 Edge
Field Name	iOS Profiles	Forensics	SecureClient	VPN
Display Name	iOS Profiles	Forensics	SecureClient	VPN
Field Name	Cellular Network	FDE	Server	VPN-1
Display Name	Cellular Network	Full Disk Encryption	Server	VPN
Field Name	Anti Spam	Capsule Docs	SmartConsole	VPN-1 Embedded Connector
Display Name	Anti-Spam and Email Security	Capsule Docs	SmartConsole	VPN Embedded Connector
Field Name	New Anti Virus	HTTPS Inspection	Eventia Analyzer Client	WebCheck
Display Name	Anti-Virus	HTTPS Inspection	SmartEvent Client	WebCheck
Field Name	Application Control	SmartDefense	SmartEvent	Zero Phishing
Display Name	Application Control	IPS Software Blade	Eventia Analyzer	Zero Phishing
Field Name	Compliance Blade	IPS-1	SmartView	MTA
Display Name	Compliance Blade	IPS-1 Sensor	SmartView	MTA

HTH

Yonatan

PhoneBoy · ‎2018-09-04

It's scary that I recognize almost all these names.

The only one I didn't know was Everest.

_Val_ · ‎2018-09-04

I think that was the code name for Connectra once

Vladimir · ‎2018-09-04

Yonatan, the top row has become a "header" row in the table you have posted.

Had a double-take reading offset entries under it for about 10 lines before it clicked

Yonatan_Philip · ‎2018-09-05

Partially fixed. I removed the bold, but it won't let me remove the bottom 'header' border for some reason.

PhoneBoy · ‎2018-09-05

You probably need to edit the raw HTML (which you can do).

Yonatan_Philip · ‎2018-09-05

Thanks - That seems to have done the trick.

I hate going into the HTML source code. I get flashbacks from trying to manually fix SKs.

The horror!

Matthew_Stovall · ‎2018-09-05

Hi Yonatan,

Thank you for the detailed write-up, I found it very helpful!

Is there a way to rate limit the amount logs that are exported? We ran into an issue where our log exporter process was overloading our syslog server with requests.

Thanks!
Matt

Yonatan_Philip · ‎2018-09-26

Hi Matthew,

At this time there is no way to limit the number of logs sent.

This will probably be possible in the future once we implement advanced filter capabilities.

Improving the filters is an item on our roadmap but I don't know when this will be implemented.

Alexandr_Kharch · ‎2018-09-26

Hi!

Nice writeup and nice tool!

I have small question

I need export only SmartEvent events width EN***** id in message body

Can i do it with LogExporter? Can u help me with configuration?

Yonatan_Philip · ‎2018-09-27

Hi Alexandr,

This can be done to a limited degree, probably not good enough for your use case.

This will most likely be possible once we implement advanced filtering options - unfortunately, I don't know exactly when that will actually happen (it's on the roadmap but we are currently working on another feature ahead of filters).

Alexandr_Kharch · ‎2018-09-27

Maybe i can use log exporter tool in custom script for sending messages?

Yonatan_Philip · ‎2018-09-27

Hi,

I'm not sure I understand the question. I suspect the answer is no. The Log Exporter uses the indexing infrastructure (that Checkpoint log servers use). It reads *.fw log files, but instead of 'indexing' them it sends the logs to the interface send queue.

I don't see how you can insert scripts into this chain, nor can this run on a server without the indexing infrastructure (e.g. log servers)

Maarten_Sjouw · ‎2018-10-30

Jonathan,

Something that is not completely clear to me, when we want to use TLS and we have a official cert at the other end, do I still need to created local files to allow this to be used? Or can I just tell log exporter to use TLS?

Regards, Maarten

Yonatan_Philip · ‎2018-10-31

The Log Exporter uses mutual authentication - both sides need to authenticate each other.

When we were looking at the TLS implementation of other vendors we noticed that some of them use single sided authentication, but after considering the issue we decided to err on the side of more security and implemented mutual authentication.

Hope that answers your question. it's always a bit difficult for me to address TLS related questions as that's not my area of expertise, and while I was involved in the TLS discussions during the implementations, I was mostly on the sidelines of those discussions and left it to the relevant experts to do the heavy lifting.

HTH

Yonatan

Jeff_Gao · ‎2018-11-10

Log_export can export all software blade log to external syslog server ?

Yonatan_Philip · ‎2018-11-11

Yes.

The Log Exporter can export everything in the fw.log file regardless of the content.

It basically treats everything in the log payload as an alphanumeric string.

All the adaptations, mappings, filters, etc. are all based on string/text manipulation regardless of the content ("blade").

HTH

Yonatan

Anton_Smirnoff · ‎2018-11-12

Hi Yonatan, I just configured log exporter so send logs via syslog to SIEM server, however when log sent I don't see protocol field in the log, when sent vi LEA saw for example - protocol=UDP etc. but now I get only proto=6, proto=17 etc. How do I convert it to protocol name? Is there any proto(number) to protocol(actual protocol name) mapping exist?

_Val_ · ‎2018-11-12

List of IP protocol numbers - Wikipedia

Timothy_Hall · ‎2018-11-12

Also the /etc/protocols file on Gaia/Linux.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

"Max Capture: Know Your Packets" Video Series
now available at http://www.maxpowerfirewalls.com

Anton_Smirnoff · ‎2018-11-12

Thanks!

Yonatan_Philip · ‎2018-11-13

For those who missed it, Dan has officially announced the new Log Exporter update with better Splunk integration.

*New* Splunk App for Check Point Logs

_Val_ · ‎2018-11-30

Will it work on Standalone machine?

Maarten_Sjouw · ‎2018-11-30

I have it running on a standalone logserver.

Regards, Maarten

Chris_W · ‎2018-11-30

Is it possible to run the LogExporter only on a Log Correlation Unit?

Yonatan_Philip · ‎2018-12-03

You can run the Log Exporter on any server where you can enable the Logging blade (Management, Log server, SmartEvent, etc.).

The Log Exporter uses the Indexing infrastructure so that infrastructure has to be installed on the server. (it doesn't have to actually be active, just needs to be installed. So even if you aren't actively using the server as an Indexer, as long as you have the option to enable the blade that's good enough).

HTH

Yonatan

Luca_Martinis · ‎2019-02-13

Hi, I'd need to add a string at the beginning of the exported logs, is it possible?

Not working on RSA NetWitness / Security Analytics:

Working fine on RSA NetWitness / Security Analytics:

the addition of the string "<1>" at the beginning of the exported log is needed in order to have the exported log correctly ingested and parsed in the RSA SIEM.

Many thanks

kind regards

Luca

Documentation

Integrations

Logging

Log Exporter guide