Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Yonatan_Philip
Employee Alumnus
Employee Alumnus

Log Exporter guide

Hello All,

We have recently released the Log Exporter solution.
A few posts have already gone up and the full documentation can be found at sk122323.

However, I've received a few questions both on and offline and decided to create a sort of log exporter guide.

But before I begin I’d like to point out that I’m not a Checkpoint spokesperson, nor is this an official checkpoint thread.

I was part of the Log Exporter team and am creating this post as a public service.

I’ll try to only focus on the current release, and please remember anything I might say regarding future releases is not binding or guaranteed.
Partly because I’m not the one who makes those decisions, and partly because priorities will shift based on customer feedback, resource limitations and a dozen other factors. The current plans and the current roadmap is likely to drastically change over time.

And just for the fun of it, I’ll mostly use the question-answer format in this post (simply because I like it and it’s convenient).

 

Log Exporter – what is it?

Performance

Filters

Filters: Example 1

Filters: Example 2

Gosh darn it, I forgot something! (I'll edit and fill this in later)

Feature request

146 Replies
Yonatan_Philip
Employee Alumnus
Employee Alumnus

Hello Su,

From your log (Current=13 Avg=139 MinAvg=10 Total=331020 ) as well as the status command it appears that logs are being exported.

If you want to actually see this you can use tcpdump command: 'tcpdump port 514 -A -s0' (if you are using port 514 for anything else, you can add other qualifiers to narrow down the output).

This will show you the actual data being exported in a readable format. For example:

[Expert@ypsa:0]# tcpdump port 514 -A -s0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:25:25.094828 IP ypsa.47206 > XX.XX.XX.XX.syslog: SYSLOG local0.info, length: 1044
E..0..@.@..xd P...f......<134>1 2018-08-14T14:25:23Z ypsa CheckPoint 17857 - [action:"Accept"; ifdir:"inbound"; ifname:"eth0";  [deleted the payload] product:"VPN-1 & FireWall-1"; proto:"6"; s_port:"35700"; service:"22"; service_id:"ssh"; src:"XX.XX.XX.XX"; ]

1 packets captured
2 packets received by filter
0 packets dropped by kernel
[Expert@ypsa:0]#

(I deleted most of the payload since it just takes up space and not really relevant for this example - I just wanted to show that you can see and read the actual logs as they are being exported)

Since it looks like your logs are actually being exported, I would focus on the other end and try to see if it's being received and parsed correctly.

Use tcpdump or Wireshark on the other end. If it's not there, it's a connectivity issue, and if it's there it's probably a parsing issue.

HTH 

 Yonatan 

0 Kudos
Bogdan_Kirylyuk
Employee Alumnus
Employee Alumnus

Hey Yonatan,

Glad I found this post

Please let me know where it stands for supporting JSON format output.

My X-Pack charged ELK is waiting to search thru this data and I am currently working on parsing it myself which is not an easy task so far.

I believe JSON format will be friendly to work with in python scripts and rest API as well

0 Kudos
Yonatan_Philip
Employee Alumnus
Employee Alumnus

HI Bogdan,

I don't think that official json output will be added anytime soon - there are currently many other items ahead of it in the queue.

However, earlier in this thread, I showed how you can edit the settings to generate json like output.

https://community.checkpoint.com/message/25414-re-log-exporter-guide?commentID=25414&et=watches.emai... 

It does have the disadvantage of potentially having duplicate keys, but a fast google search returned some answers on how to deal with that: 

Elasticsearch, Kibana and duplicate keys in JSON -  

Please let me know if this worked.

Yonatan 

0 Kudos
Yonatan_Philip
Employee Alumnus
Employee Alumnus

      Product (Blade) names.

As you might have noticed the names that are exported are not the same as the ones which appear in the GUI. 

This is because each blade has a display name which is shown in the GUI and an actual value which appears in the raw log. Those names are often the same but not always.

For technical reasons, it's very difficult to change the actual value that the gateway sends but easy to change the display name. So while the actual value and the display name usually start the same, the values might drift over time.

This has caused a bit of confusion among some customers.

So as a sort of public service here is the current mapping of raw log values to display name.

This list also includes some legacy names. Frankly, there were a few names on this that I've never heard of and didn't know existed Smiley Happy

Field Name

Anti Malware

Core

Identity Awareness

SmartView MonitorMonitor

Display Name

Anti-Bot

Core

Identity Awareness

SmartView Monitor

Field Name

Anti-Malware

DefensePro

Identity Logging

Syslog

Display Name

Anti-Malware

DDoS Protector

Identity Logging

Syslog

Field Name

Anti-Exploit

DLP

Management Blade

System Monitor

Display Name

Anti-Exploit

DLP

Management Blade

System Monitor

Field Name

Anti-Ransomware

Content Awareness

MEPP

Threat Emulation

Display Name

Anti-Ransomware

Content Awareness

Media Encryption &amp; Port Protection

Threat Emulation

Field Name

WIFI Network

Edge AV

Connectra

Threat Extraction

Display Name

WIFI Network

Edge AV

Mobile Access

Threat Extraction

Field Name

Mobile App

Compliance

Policy Server

Anti Virus

Display Name

Mobile App

Endpoint Compliance

Policy Server

Traditional Anti-Virus

Field Name

Network Security

Integrity

Web Filtering

UAG

Display Name

Network Security

EndpointEndpoint Security

Legacy URL Filtering

UA Server

Field Name

OS Exploit

Everest

CVPN

WebAccess

Display Name

OS Exploit

FireWall-1 GX

CVPN

UA WebAccess

Field Name

Device

Firewall

FG

URL Filtering

Display Name

Device

Firewall

QoS

URL Filtering

Field Name

Text Message

VPN-1 & FireWall-1

rtm

VPN-1 Edge

Display Name

Text Message

Security Gateway/Management

Real Time Monitor

UTM-1 Edge

Field Name

iOS Profiles

Forensics

SecureClient

VPN

Display Name

iOS Profiles

Forensics

SecureClient

VPN

Field Name

Cellular Network

FDE

Server

VPN-1

Display Name

Cellular Network

Full Disk Encryption

Server

VPN

Field Name

Anti Spam

Capsule Docs

SmartConsole

VPN-1 Embedded Connector

Display Name

Anti-Spam and Email Security

Capsule Docs

SmartConsole

VPN Embedded Connector

Field Name

New Anti Virus

HTTPS Inspection

Eventia Analyzer Client

WebCheck

Display Name

Anti-Virus

HTTPS Inspection

SmartEvent Client

WebCheck

Field Name

Application Control

SmartDefense

SmartEvent

Zero Phishing

Display Name

Application Control

IPS Software Blade

Eventia Analyzer

Zero Phishing

Field Name

Compliance Blade

IPS-1

SmartView

MTA

Display Name

Compliance Blade

IPS-1 Sensor

SmartView

MTA

HTH 

 Yonatan 

0 Kudos
PhoneBoy
Admin
Admin

It's scary that I recognize almost all these names. Smiley Happy

The only one I didn't know was Everest.

0 Kudos
_Val_
Admin
Admin

I think that was the code name for Connectra once

0 Kudos
Vladimir
Champion
Champion

Yonatan, the top row has become a "header" row in the table you have posted.

Had a double-take reading offset entries under it for about 10 lines before it clicked Smiley Happy

0 Kudos
Yonatan_Philip
Employee Alumnus
Employee Alumnus

Partially fixed. I removed the bold, but it won't let me remove the bottom 'header' border for some reason.

PhoneBoy
Admin
Admin

You probably need to edit the raw HTML (which you can do).

0 Kudos
Yonatan_Philip
Employee Alumnus
Employee Alumnus

Thanks - That seems to have done the trick.

I hate going into the HTML source code. I  get flashbacks from trying to manually fix SKs. 

The horror!

Smiley Happy

Matthew_Stovall
Explorer

Hi Yonatan,

Thank you for the detailed write-up, I found it very helpful!

Is there a way to rate limit the amount logs that are exported? We ran into an issue where our log exporter process was overloading our syslog server with requests.

Thanks!
Matt

0 Kudos
Yonatan_Philip
Employee Alumnus
Employee Alumnus

Hi Matthew,

At this time there is no way to limit the number of logs sent.

This will probably be possible in the future once we implement advanced filter capabilities.

Improving the filters is an item on our roadmap but I don't know when this will be implemented.

0 Kudos
Alexandr_Kharch
Explorer

Hi!

Nice writeup and nice tool!

I have small question

I need export only SmartEvent events width EN***** id in message body

Can i do it with LogExporter? Can u help me with configuration?

0 Kudos
Yonatan_Philip
Employee Alumnus
Employee Alumnus

Hi Alexandr, 

This can be done to a limited degree, probably not good enough for your use case. 

This will most likely be possible once we implement advanced filtering options - unfortunately, I don't know exactly when that will actually happen (it's on the roadmap but we are currently working on another feature ahead of filters).

0 Kudos
Alexandr_Kharch
Explorer

Maybe i can use log exporter tool in custom script for sending messages?

0 Kudos
Yonatan_Philip
Employee Alumnus
Employee Alumnus

Hi,

I'm not sure I understand the question. I suspect the answer is no. The Log Exporter uses the indexing infrastructure (that Checkpoint log servers use). It reads *.fw log files, but instead of 'indexing' them it sends the logs to the interface send queue.

I don't see how you can insert scripts into this chain, nor can this run on a server without the indexing infrastructure (e.g. log servers)

Maarten_Sjouw
Champion
Champion

Jonathan,

Something that is not completely clear to me, when we want to use TLS and we have a official cert at the other end, do I still need to created local files to allow this to be used? Or can I just tell log exporter to use TLS?

Regards, Maarten
0 Kudos
Yonatan_Philip
Employee Alumnus
Employee Alumnus

The Log Exporter uses mutual authentication - both sides need to authenticate each other.

When we were looking at the TLS implementation of other vendors we noticed that some of them use single sided authentication, but after considering the issue we decided to err on the side of more security and implemented mutual authentication.

Hope that answers your question. it's always a bit difficult for me to address TLS related questions as that's not my area of expertise, and while I was involved in the TLS discussions during the implementations, I was mostly on the sidelines of those discussions and left it to the relevant experts to do the heavy lifting.

HTH 

 Yonatan 

0 Kudos
Jeff_Gao
Advisor

Log_export can export all software blade log to external syslog server ?

0 Kudos
Yonatan_Philip
Employee Alumnus
Employee Alumnus

Yes.

The Log Exporter can export everything in the fw.log file regardless of the content.

It basically treats everything in the log payload as an alphanumeric string.

All the adaptations, mappings, filters, etc. are all based on string/text manipulation regardless of the content ("blade").

HTH 

 Yonatan 

0 Kudos
Anton_Smirnoff
Participant

Hi Yonatan, I just configured log exporter so send logs via syslog to SIEM server, however when log sent I don't see protocol field in the log, when sent vi LEA  saw for example - protocol=UDP etc. but now I get only proto=6, proto=17 etc. How do I convert it to protocol name? Is there any proto(number) to protocol(actual protocol name) mapping exist?

0 Kudos
_Val_
Admin
Admin

0 Kudos
Timothy_Hall
Champion
Champion

Also the /etc/protocols file on Gaia/Linux.

--
Second Edition of my "Max Power" Firewall Book
Now Available at http://www.maxpowerfirewalls.com

Gateway Performance Optimization R81.20 Course
now available at maxpowerfirewalls.com
0 Kudos
Anton_Smirnoff
Participant

Thanks!

0 Kudos
Yonatan_Philip
Employee Alumnus
Employee Alumnus

For those who missed it, Dan has officially announced the new Log Exporter update with better Splunk integration.

*New* Splunk App for Check Point Logs 

_Val_
Admin
Admin

Will it work on Standalone machine?

0 Kudos
Maarten_Sjouw
Champion
Champion

I have it running on a standalone logserver.

Regards, Maarten
0 Kudos
Chris_W
Participant

Is it possible to run the LogExporter only on a Log Correlation Unit?

0 Kudos
Yonatan_Philip
Employee Alumnus
Employee Alumnus

You can run the Log Exporter on any server where you can enable the Logging blade (Management, Log server, SmartEvent, etc.).

The Log Exporter uses the Indexing infrastructure so that infrastructure has to be installed on the server. (it doesn't have to actually be active, just needs to be installed. So even if you aren't actively using the server as an Indexer, as long as you have the option to enable the blade that's good enough).

HTH

 Yonatan 

0 Kudos
Luca_Martinis
Employee Alumnus
Employee Alumnus

Hi, I'd need to add a string at the beginning of the exported logs, is it possible?

Not working on RSA NetWitness / Security Analytics:

"CEF:0|Check Point|VPN-1 & FireWall-1|Check Point|Log|domain-udp|Unknown|act=Accept destinationTranslatedAddress=0.0.0.0 destinationTranslatedPort=0 deviceDirection=0 .....

Working fine on RSA NetWitness / Security Analytics:

"<1> CEF:0|Check Point|VPN-1 & FireWall-1|Check Point|Log|domain-udp|Unknown|act=Accept destinationTranslatedAddress=0.0.0.0 destinationTranslatedPort=0 deviceDirection=0 .....

the addition of the string "<1>" at the beginning of the exported log is needed in order to have the exported log correctly ingested and parsed in the RSA SIEM.

Many thanks

kind regards

Luca

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events