Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Yonatan_Philip
Employee
Employee

Log Exporter guide

Hello All,

We have recently released the Log Exporter solution.
A few posts have already gone up and the full documentation can be found at sk122323.

However, I've received a few questions both on and offline and decided to create a sort of log exporter guide.

But before I begin I’d like to point out that I’m not a Checkpoint spokesperson, nor is this an official checkpoint thread.

I was part of the Log Exporter team and am creating this post as a public service.

I’ll try to only focus on the current release, and please remember anything I might say regarding future releases is not binding or guaranteed.
Partly because I’m not the one who makes those decisions, and partly because priorities will shift based on customer feedback, resource limitations and a dozen other factors. The current plans and the current roadmap is likely to drastically change over time.

And just for the fun of it, I’ll mostly use the question-answer format in this post (simply because I like it and it’s convenient).

 

Log Exporter – what is it?

Performance

Filters

Filters: Example 1

Filters: Example 2

Gosh darn it, I forgot something! (I'll edit and fill this in later)

Feature request

145 Replies
Luca_Martinis
Employee Alumnus
Employee Alumnus

Solution provided as follows from the great Kobi 😉

Yes we have this option:

Go to CEFFormatDefinition.xml in $EXPORTERDIR/targets/<target name>/conf

 

Change the line:

<header_format>{}|{}|{}|{}|{}|{}|{}|</header_format>

To be:

<header_format>{} {}|{}|{}|{}|{}|{}|{}|</header_format>

 

In addition, under the line:

<headers>

Add this:

<header>  <default_value>&lt;1&gt;</default_value>  <assign_order>init</assign_order>     </header>

 

Save the file and restart the exporter.

0 Kudos
Will_Hargreaves
Employee
Employee

Hi all, 

Default behaviour appears to be that suppressed logs are not exported to 3rd party SIEM by LogExporter. Is there a way to modify this so that supressed logs ARE exported? 

FYI, the test bed we're working on is RSYSLOG.

Cheers,

Will

Hrvoje_Brlek
Contributor

Hi,

 

Any help regarding this question? 

I am exporting logs to Splunk with semi-unified mode on, but we are getting all the logs not only the supressed ones.

For example for one loguid we are getting 4 different logs. As in the question mentioned here: https://community.checkpoint.com/t5/Logging-and-Reporting/Log-exporter-not-summary-logging-to-one-ev...

Would be very helpful if it is possible just to export the last summarized event...

Thanks

0 Kudos
Ben_Chong
Employee
Employee

Can we filter logs that are being sent out through log exporter, by rules ID or name? For example, I would like to have only outbound traffic to internet to be sent out through log exporter. I understand we can modify the targetConfiguration.xml file, and the only fields that can be filtered are as the following:

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['product']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['__policy_id_tag']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['inzone']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['outzone']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['service_id']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['src']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['s_port']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['dst']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['service']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['proto']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['xlatesrc']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['xlatedst']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['xlatesport']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['xlatedport']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['nat_rulenum']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['nat_addtnl_rulenum']

[log_indexer 23822 4107270976]@CP-SMS[1 Mar 8:09:34] Read Log Format field name:['match_table']

Let me know if there is any other way. thanks.

Yonathan_Grunew
Participant

Hello,

So nice to see your investment into this topic, Yonathan.

We're having some trouble configuring Log Exporter to work with qRadar.

we have updated our Management server to r80.10 JHF 154 just to install new log exporter following PS recommendation. 

We have followed the SK, but the TLS instructions aren't so clear - what certificate\keys goes where, some weird symbols and notes scattered etc.

I think the Qradar screenshot could use some values in it.

has anyone managed to configure this with qRadar and mutual TLS?

0 Kudos
Richard_Amos
Participant

Hi,

I'm trying to filter using the predefined TP "product" with the filter-blade-in option per the example given in sk122323.  Currently all logs are being exported.  I get the following output:

cp_log_export set name <name obfuscated> filter-blade-in "TP"
Error: Argument [filter-blade-in] is undefined for command: [set]

This is on an R80.20 GA w/JHFA 33 open management server with logging.

One other point of clarification - the description for TP in the sk does not include IPS.  I only see that in the EndPoint description.  Was this an omission in the description or is it really not included in the TP filter?

Thanks for the help!

0 Kudos
Dror_Aharony
Employee
Employee

Hi Richard,

Sadly, the new log-exporter filtering feature isn't yet supported on R80.20 / R80.30.

from the official log-exporter sk122323 (Installation section):

"Note: Filtering ability is not integrated to R80.20 and R80.30 yet, this SK will be updated when it will be supported."

Coming soon...

 

*In-general, TP filter includes all Threat blades (including IPS blade).

0 Kudos
Yabin_Zhang
Employee Alumnus
Employee Alumnus

I have a case about integration with Aruba ClearPass , Aruba hope checkpoint SMS send syslog to ClearPass ,and give me a conf file like this。 which XML file should I edit? and which  Field?

CheckPoint_IngressEvent.xml:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<TipsHeader exportTime="Mon May 27 16:36:00 CST 2019" version="6.7"/>
<IngressEvents>
<IngressEvent>
<Vendor>Check Point</Vendor>
<Description>Check Point log message</Description>
<FormatName>CheckPoint-Log</FormatName>
<Format>TIME LOG_TYPE ORIGIN SERVICE PID ACTION SRC_INTERFACE MESSAGE</Format>
<Prefix>CheckPoint-Log</Prefix>
<Enabled>false</Enabled>
<Sample>Mon Jul 20 15:56:36 Log host CPLogToSyslog: 49154 redirect &lt;eth1 web_client_type: Firefox; resource: http://sc1.checkpoint.com/za/images/threatwiki/pages/testantibotblade.html; src: 10.70.11.11; dst: 194.29.36.43; proto: 6; session_id: {0x55acf003,0x1,0xb4617ac,0xc0000002}; Protection name: Check Point - Testing Bot; malware_family: Check Point; Confidence Level: 5; severity: 2; malware_action: Communication with CandC; rule_uid: {6AA76C68-D45C-4E78-BAD1-34C42548BF41}; Protection Type: URL reputation; malware_rule_id: {645B69FE-85AC-F748-AA79-5652BF58BF6A}; protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 10.70.11.11; scope: 10.70.11.11; aba_customer: Default; date: 20Jul2015; hour: 15:56:35; type: log; Interface: &gt; eth1; product: Anti Malware; service: 8080; s_port: 39490;</Sample>
<Filter>filter {
grok {
match =&gt; { 'message' =&gt; '%{SYSLOGTIMESTAMP:time}%{SPACE}%{WORD:log_type}%{SPACE}%{WORD:origin}%{SPACE}%{WORD:service}:%{SPACE}%{WORD:pid}%{SPACE}%{WORD:action}%{SPACE}%{DATA:src_interface} %{GREEDYDATA:syslog_message}'}
add_tag =&gt; [ "CP" ]
}
if("CP" in [tags]){
mutate {
replace =&gt; [ '@message', '%{syslog_message}' ]
}
kv {
source =&gt; '@message'
prefix =&gt; 'Event:CheckPoint-Log:'
field_split =&gt; ';'
value_split =&gt; ':'
trim =&gt; ' '
trimkey =&gt; ' '
}
mutate {
remove_field =&gt; ['@version','path','syslog_message','@message','message']
add_field =&gt; [ 'Event:Event-Name', '%{service}' ]
add_field =&gt; [ 'Event:Timestamp', '%{time}' ]
add_field =&gt; [ 'Event:Pattern-Name', 'CheckPoint-Log' ]

}
ruby {
code =&gt; "
data = event.clone.to_hash;
data.each do |k,v|
if (k != '@timestamp' and !k.start_with?('Event:') and !k.start_with?('@'))
newFieldName = 'Event:CheckPoint-Log:'+ k
event[newFieldName] = v
event.remove(k)
end
end
tstamp = Time.now.to_i
tstamp_str = Time.at(tstamp).strftime('%Y-%m-%d %H:%M:%S')
event['Event:Timestamp'] = tstamp_str
"
}
}
}</Filter>
<FieldMapping>
<Field AllowedValues="" DataType="Time" Name="time"/>
<Field AllowedValues="" DataType="String" Name="log_type"/>
<Field AllowedValues="" DataType="String" Name="origin"/>
<Field AllowedValues="" DataType="String" Name="pid"/>
<Field AllowedValues="" DataType="String" Name="action"/>
<Field AllowedValues="" DataType="String" Name="service"/>
<Field AllowedValues="" DataType="String" Name="src_interface"/>
<Field AllowedValues="" DataType="String" Name="web_client_type"/>
<Field AllowedValues="" DataType="String" Name="resource"/>
<Field AllowedValues="" DataType="String" Name="src"/>
<Field AllowedValues="" DataType="String" Name="dst"/>
<Field AllowedValues="" DataType="String" Name="proto"/>
<Field AllowedValues="" DataType="String" Name="session_id"/>
<Field AllowedValues="" DataType="String" Name="Protectionname"/>
<Field AllowedValues="" DataType="String" Name="malware_family"/>
<Field AllowedValues="" DataType="String" Name="ConfidenceLevel"/>
<Field AllowedValues="" DataType="String" Name="severity"/>
<Field AllowedValues="" DataType="String" Name="malware_action"/>
<Field AllowedValues="" DataType="String" Name="rule_uid"/>
<Field AllowedValues="" DataType="String" Name="ProtectionType"/>
<Field AllowedValues="" DataType="String" Name="malware_rule_id"/>
<Field AllowedValues="" DataType="String" Name="protection_id"/>
<Field AllowedValues="" DataType="String" Name="log_id"/>
<Field AllowedValues="" DataType="String" Name="proxy_src_ip"/>
<Field AllowedValues="" DataType="String" Name="scope"/>
<Field AllowedValues="" DataType="String" Name="aba_customer"/>
<Field AllowedValues="" DataType="String" Name="date"/>
<Field AllowedValues="" DataType="String" Name="hour"/>
<Field AllowedValues="" DataType="String" Name="Interface"/>
<Field AllowedValues="" DataType="String" Name="product"/>
<Field AllowedValues="" DataType="String" Name="s_port"/>
</FieldMapping>
<GenericFieldMapping>
<Field GenericName="Event-Name" Name="service"/>
<Field GenericName="Timestamp" Name="time"/>
</GenericFieldMapping>
</IngressEvent>
</IngressEvents>
</TipsContents>
0 Kudos
S_E_
Advisor

hi,

interesting feature.

2 questions regarding this one:

 

1.  is it possible to export the main CMA ? (content of /opt/CPmds-R80.30/log)

      I tried the command but a warning appears: Failed to change env to customer: <IP.IP.IP.IP> 

2. Is it possible to export a single domain to multiple destinations? target-server 1 & target-server 2

 

Thanks

Best Regards

 

 

 

0 Kudos
Dror_Aharony
Employee
Employee

Hi S_E_ (Nickel),

1. Yea. You just need to use the domain-server mds for the mds/global level.

      "On MDS/MLM: domain-server argument is mandatory, you can use 'mds' as the value for domain-server in order to export mds level audit logs"

2. Yea. You just need to add another log-exporter on that same domain-server, but with a different target-server <IP>.

      quick shortcut way to create another identical exporter is simply copying entire folder (It'll also register it as new. Simply edit what you need afterwards, by either using the set command or manually) by:

       mdsenv <relevant domain-name/IP>

       cp -rf $EXPORTERDIR/targets/<name> $EXPORTERDIR/targets/<new_name>

MiteshAgrawal15
Participant

Hi @Dror_Aharony ,

We aren't receiving audit logs using the Log exporter guide. You mentioned that the domain-server argument needs to be added while configuring the destination. Please help how can I confirm whether the domain-server argument is added in my configuration.

 

My current configuration is given below which I got by running command cp_log_export show "name":

name: ArcSightLog
enabled: true
target-server: (AGENT SERVER IP)
target-port: 514
protocol: udp
format: cef
read-mode: semi-unified

 

Regards,

Mitesh Agrawal

0 Kudos
Dror_Aharony
Employee
Employee

domain-server is only needed/required on an MDS server.

so if you don't see it, then it's not an MDS, right?

 

in-general, you simply add another flag: domain-server <CMA>

# cp_log_export add/set....domain-server <CMA/DMS>

 

 

0 Kudos
cdooer
Participant

Can you tell us if this has been resolved yet?

0 Kudos
phlrnnr
Advisor

Feature request - it seems that the 'target-server' must be an IP address.  It would be great if this can be updated to either a domain name or an IP address.  That would allow for flexibility when a customer is controlling which syslog server you are hitting via DNS.  Thanks for considering.

0 Kudos
Dan_Zada
Employee Alumnus
Employee Alumnus

Hi,
Thank you for your feedback. This is already is our todo list.
Stay tuned for updates 🙂
0 Kudos
Raymondn
Contributor

Hi there,

Regarding to "System Monitor" type of logs, what can I expect to see?

More background about my question.

We are on R80.30.  We have been using the old (classic/legacy) way to export CP logs to Splunk via the OPSEC LEA connection.

Been working fine.  For "system" type of messages, I can see log when firewall policy being pushed with the admin username.  I also able to see some messages regarding to high CPU usage or cluster status alert, similar to those "control" or "alert" type messages I see in the native Smart Console Log view.

 

Last week we switched to use the Log Exporter to Splunk (we also got the Check Point for Splunk apps installed).

What I am trying to figure out is how to get back those "policy installation" and "system status" log in Splunk.

I manage to find the "policy installation" log via Splunk (index="network_firewalls" source=tcp:11002
sys_message="installed*"), but the log didn't include the admin username.

But for other system status/alert type of log, I am not able to find them in Splunk.

 

First question is if what I am trying to do is available under LogExporter method?

If so, which direction or field values I should search in Splunk?

I see there is a field "product=System Monitor" but it doesn't seem contains much useful system log messages.

 

Thanks in advance.

0 Kudos
Dror_Aharony
Employee
Employee

All such logs should be exported to splunk assuming you don't have any filter.

if you see them in the SmartConsole's Logs view for this Log-Server/Management.

 

product=System Monitor should work (translated to splunk matching filter).

 

Can you share an example pic (or copy fields) of a system monitor log you'd like to see, but is missing or cannot be found on your splunk using log-exporter?

 

 

0 Kudos
LostBoY
Advisor

Hello Yonaton,

This is an extremely helpful post for newbies like me..thank you for this..

i am having some trouble with setting up the log exporter as follows:

The scenario is to to transfer logs from R80.40 mgmt server to a syslog server..that syslog server will be integrated with datadog SIEM..i have setup the log exporter in r80.40 (i guess in r80.40 there is no need to install log exporter and its already there)..the issue is at the syslog end complete connection logs are not visible..i can just see the process id of log exporter deamon and the mgmt server hostname..is there anything else i need to do to transfer all the connection logs to syslog ?

Thanks

0 Kudos
Shay_Hibah
Employee Alumnus
Employee Alumnus

Hi @LostBoY 

On R80.40 Log Exporter feature is already integrated and you can use it without any other fix installation.

When you configure Log Exporter, by default all logs (both security and audit) are exported to the target server.

In order to check where the problem is, I would suggest you to:

1. Make sure the logs can be seen in your Log Server by connecting to this server via SmartConsole and make sure you are able to see them.

2. Check if these logs can be found on your syslog server.

In case it doesn't help to find the issue, I would suggest you to open a ticket for a further investigation.

If you would like to, you can upload / send me your exporter directory and I will be able to take a look at your exporter configuration to see if I found any errors.

Shay

0 Kudos
LostBoY
Advisor

Thanks for the reply... my log server is the mgmt server and i can see the logs via smartconsole..

 

At the syslog side.. the process id of log exporter is visible and mgmt hostname is being populated but there arent any traffic logs..

Which exact directory do i need to fetch and where should i upload it ? 

0 Kudos
Shay_Hibah
Employee Alumnus
Employee Alumnus

Zip the directory $EXPORTERDIR/targets<your_exporter_name>

You can upload it to here or send me by email: shayhi@checkpoint.com and we will take it from there.

0 Kudos
bigjim
Explorer

Hello everybody
I have a problem with Log Exporter. So i am taking my chance here .😉
my management / logs server is in R80.40 build 126
the daemon is Ok and sends the logs to my SIEM (logstash) on a TCP / 6032 listening port
the problem is that the TCP stack explodes after several thousand unclosed sessions on the SIEM server.
Why does Log_exporter open all these sessions instead of just one?
How do you tell him to close them if they don't use them anymore?
Thank you in advance for your help.

Here is my basic activation line: cp_log_export add name ISEMLOG target-server x.x.x.x target-port 6032 protocol tcp format cef --apply-now

0 Kudos
Ido_Shoshana
Employee
Employee

Hi,

We are currently checking the issue.

Can you say if you had network issues in the past?

looks like if there is a network issue the exporter closes the connections on its side and tries to open new connections until it succeeds, but on SIEM side the old connections are also still open (still under investigation).

 

0 Kudos
Miri_Ofir
Employee
Employee

Hi @bigjim 

My name in Miri Ofir, I'm the group manager in charge of Logging products.

In regards to many open connections on SIEM, is it a new issue in your log exporter that you started to see recently?

Usually, based on previous cases, the problem of abandoned connection was something on the SIEM side that didn't clear them. I suggest to check this direction. What is the destination type? 

0 Kudos
bigjim
Explorer

Hi @Miri_Ofir and @Ido_Shoshana 

we have no network issue between SIEM and log server : 2 servers on  seperate Vlan routed on the same core DC. just 1 hop. No FW. 

It 's a new issue because this is the first time we set up log-exporter.

The SIEM is based on a Red Hat Enterprise Linux release 8.3 + Logstash 7.14.2 for logs collection

 

0 Kudos
Miri_Ofir
Employee
Employee

Hi @big

Do you know if you have an option to configure on the SIEM to close abounded connections on it's own after certain TTL?

Log Exporter opens a new connection only in case of existing connection is failing, if you are sure the I/S is good and the Syslog server is configured properly, I suggest to work with our support to troubleshoot the problem.

 

0 Kudos