R80.10 Syslog Exporter

[HeikoAnkenbrand]

Via Check Point Support you get a Syslog exporter for SIEM applications for R80.10 Managment.

Which allows an easy and secure method for exporting CP logs over syslog. Exporting can be done in few standard protocols and formats.

Log Exporter supports:

• Splunk
• Arcsight
• RSA
• LogRhythm
• QRadar
• McAfee

Log Exporter is a multi-threaded daemon service, running on a log server. Each log that is written on the log server is read by the log exporter daemon, transformed into the desired format and mapping, and then sent to the end target.

Installation on R80.10 Jumbo Hotfix Take 56 or higher.

Syntax:

# cp_log_export add name <name> [domain-server <domain-server>] target-server <target-server> target-port <target-port> protocol <(udp|tcp)> [optional arguments]

Command Name	Command Description
add	Deploy a new Check Point logs exporter.
set	Updates an exporter's configuration.
delete	Removes an exporter.
show	Prints an exporter's current configuration.
status	Shows an exporter's overview status.
start	Starts an exporter process
stop	Stops an exporter process.
restart	Restarts an exporter process.
reexport	Resets the current position, and re-exports all logs per the configuration.

Regards,

Heiko

[Biju_Nair]

Please see this snapshot.

Also when i try to delete, it didn't allow me to delete it. Should i try to delete the EXPORTERDIR manually.

After deleting the EXPORTERDIR manually, will it create the EXPDIR again automatically.(After cpstart).

[Yonatan_Philip]

Hello,

Since the $EXPORTERDIR wasn't defined at the time of deployment, the exporter was deployed to /opt/CPrt-R80/log_indexer/targets/ (you can see the location in the show status output).

Now that the $EXPORTERDIR variable is defined it's looking for it (and trying to delete it) in the 'correct' location.

That's why I said you might have to manually delete it and then redeploy it.

I would use 'rm -r /opt/CPrt-R80/log_indexer/targets/*'  (as with any other time that you use rm with a wild card, I would advise to exercise caution and make sure you are deleting the right location/content and nothing else).

I think we could have and should have handled this scenario better. This sort of mixup and confusion is why this should have been blocked in my opinion (I already have an open RFE to block deployment in case the $EXPORTERDIR variable is not defined).

Edit: Just noticed that the command should be: cp_log_export delete name XXXX  (you're missing the name parameter)

HTH 

 Yonatan 

[Biju_Nair]

Hey Yonatan, thank you so much. it worked for me.

Even though adding the corrected parameter(name) didn't help, but removing the target directory and recreating it helped for me.

[Hugo_vd_Kooij]

Expect new ports with R80.20. There is something going on on port 4434 that is web related.

[Yonatan_Philip]

Hello Hugo,

I saw your post on regarding the R80.20 M1 release.

I have to admit I wasn't aware of any issue with the ports and I've asked the relevant person to look into it.

I'm not familiar with any planned change to the log exporter ports in R80.20.

Nor am I familiar with any new issue with the Log Exporter on R80.20.

If you are aware of any specific issue with the Log Exporter on R80.20 M1 please send me an email with details to (edited as we are already GA for R80.20)

Thanks!

 Yonatan 

[phlrnnr]

What are the performance implications of installing the Log Exporter?  What is the typical cpu/memory footprint?

[Yaakov_Ohayon]

Each log exporter can consume up to 1 CPU when exporting in full capacity.

[phlrnnr]

I installed a log exporter on a fairly busy log server.  The log_exporter process immediately jumped to ~100% cpu (1 core).  Does this mean I need to install multiple log_exporter processes?  If so, how do I do that? 

Are there any troubleshooting commands I can use to see if the log_exporter process is not able to send all logs to the syslog server?

[Yonatan_Philip]

You don't really need to do any troubleshooting just yet - a CPU spike is expected after enabling the process for the first time.

The process will export all logs going back (by default) 1 day and will start exporting them at the maximum possible rate until it empties the queue, at which point it will go into the 'steady state' of exporting logs as they arrive.
This is the expected behavior. If the process goes down for any reason it will not lose any logs as we save a record of which logs were exported, and any that were not will be exported as soon as the process is back up and running. 

Depending on how many logs you generate per day this can take anywhere from a few seconds to a few hours.

From the screenshot, it looks like the process has been running for 4 hours already which leads me to believe you probably generate a large number of logs per day.

If the process is still at 100% CPU after a few more hours I would check the elg file to see what is the current, average and total amount of logs exported (the 'cp_log_export status' command will also give you the path to the elg file).

If the CPU load persists for more than a day, this might be an indication of a problem and you should investigate further and/or open a support ticket.

HTH 

 Yonatan 

[phlrnnr]

Thank you very much for your quick reply!  Yes, after a few hours, things seem to have calmed down, and the process is ~ 15%.  Thanks for the explanation!

[Comsec_Gsim]

We tried to use Log Exporter following SK122323, but it doesn't work as expected.

R80.10 with jumbo take 112

cp_log_export status

name: syslog-prova
     status: Running (24439)
     last log read at: 5 Nov 16:52:30
     debug file: /opt/CPrt-R80/log_exporter/targets/syslog-prova/log/log_indexer.elg

log_indexer.elg seems to report reading data, but doesn^t send anything


[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:15] Files read rate [log] : Current=44 Avg=1649 MinAvg=54 Total=2492257 buffers (0/0/0/0)

[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:15] Sent current: 0 average: 0 total: 0

[log_indexer 24439 3872390032]@meteoam-center2[5 Nov 16:25:19] Files read rate [adtlog] : Current=0 Avg=0 MinAvg=0 Total=12 buffers (0/0/0/0)

[log_indexer 24439 3872390032]@meteoam-center2[5 Nov 16:25:19] Sent current: 0 average: 0 total: 0

[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:20] Files read rate [log] : Current=42 Avg=1644 MinAvg=54 Total=2492467 buffers (0/0/0/0)

[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:20] Sent current: 0 average: 0 total: 0

[log_indexer 24439 3872390032]@meteoam-center2[5 Nov 16:25:24] Files read rate [adtlog] : Current=0 Avg=0 MinAvg=0 Total=12 buffers (0/0/0/0)

[log_indexer 24439 3872390032]@meteoam-center2[5 Nov 16:25:24] Sent current: 0 average: 0 total: 0

[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:25] Files read rate [log] : Current=49 Avg=1638 MinAvg=54 Total=2492708 buffers (0/0/0/0)

[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:25] Sent current: 0 average: 0 total: 0

[Comsec_Gsim]

cp_log_export show

name: syslog-prova
     enabled: true
     target-server: 172.22.223.71
     target-port: 514
     protocol: udp
     format: syslog

[Comsec_Gsim]

Logs from Gaia to the same server instead work as expected.

meteoam-center2> show syslog all
Syslog Parameters:
    Remote Address 172.22.223.71
        Levels info
    Auditlog permanent

[Yonatan_Philip]

Your output actually indicates that everything is working as expected and that you have already exported ~2.5M logs at an average rate of 1.6K logs/sec:

[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:25] Files read rate [log] : Current=49 Avg=1638 MinAvg=54 Total=2492708 buffers (0/0/0/0)

You have also exported 12 audit logs (I'm assuming this was the value that confused you - the elg has information for both fw.log and fw.adtlog):

[log_indexer 24439 3872390032]@meteoam-center2[5 Nov 16:25:24] Files read rate [adtlog] : Current=0 Avg=0 MinAvg=0 Total=12 buffers (0/0/0/0)

A useful debugging tool is the following command:

# tcpdump port 514 -s0 -A

[Expert@MDS-72:0]# tcpdump port 3010 -s0 -A
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:50:58.754248 IP MDS-72.61205 > 192.168.32.76.gw: P 1:297(296) ack 1 win 6 <nop,nop,timestamp 158701155 620400757>E..\..@.@..... H.. L......{.MJ......"N.....u.c$..uCEF:0|Check Point|Security Gateway/Management|Check Point|Log|Log|Unknown|deviceDirection=0 msg=Contracts outcome=Started rt=1541504854000 loguid={0x5be18022,0x0,0x4820a8c0,0x3d1779fc} origin=192.168.32.72 sequencenum=1 version=5 product=Security Gateway/Management update_service=1 version=1.0

13:50:58.754322 IP MDS-72.61205 > 192.168.32.76.gw: . 297:1745(1448) ack 1 win 6 <nop,nop,timestamp 158701155 620400757>E.....@.@..X.. H.. L......|.MJ.......3.....u.c$..uCEF:0|Check Point|Security Gateway/Management|Check Point|Log|Log|Medium|cp_severity=Medium deviceDirection=0 msg=Contracts outcome=Failed rt=1541504854000 loguid={0x5be18022,0x1,0x4820a8c0,0x3d1779fc} origin=192.168.32.72 sequencenum=2 version=5 failure_impact=Contracts may be out-of-date product=Security Gateway/Management reason=Server replied with no results. update_service=1 version=1.0

This will allow you to see the actual logs in real time as they are being exported.

If you are also sending OS logs or sending logs using other methods on the relevant port you might want to add more filters to the tcpdump command.

HTH 

 Yonatan 

[TOM_MORAN]

Hi i am trying to change the format from syslog to generic 

i have tried the below

what am i missing ?

any help appreciated

[Expert@gw-920ce3:0]# cp_log_export set format generic
Error: Missing mandatory argument <name> for command set

[Expert@gw-920ce3:0]# cp_log_export delete syslogser --apply-now
Error: Argument [syslogser] is undefined for command: [delete]

[DeletedUser]

just missing the name.... 

[Expert@R80_M:0]# cp_log_export set name myCEF format generic
Export settings for myCEF has been changed successfully
To apply the changes run: cp_log_export restart name myCEF

[Expert@R80_M:0]# cp_log_export delete name myCEF --apply-now
Stopping log_exporter for: myCEF
Removing /opt/CPrt-R80.20/log_exporter/targets/myCEF

[TOM_MORAN]

Thank you that worked very appreciated

[Alex_Tooze]

We have a new deployment coming up, which will require SPLUNK feeds for traffic and audit logs. We already have both OPSEC & LogExporter defined on our existing R80.10 platform. We use OPSEC LEA for SPLUNK, and LogExporter for a different SIEM system.

As LogExporter is currently installed as an HFA, it is an additional overhead when carrying out upgrades.

Is there any big reason why we should be using one of these solutions over the other?

Thanks.

[PhoneBoy]

Log Exporter should be independent of HFAs

[Alex_Tooze]

It isn't, as far as I am aware. In order to install a new JHFA, it is currently necessary to remove LogExporter, then previous JHFA, before installing the new JHFA and reinstalling LogExporter.

The main point of my question, though, is whether we should be using LogExporter in preference to an OPSEC LEA connector, or stick with tried-and-tested OPSEC.

[DeletedUser]

From Check Point’s perspective Log Exporter is the preferred integration going forward. Having worked with LEA for a while would say LEA provides more hooks, but lacks one key thing that Log Exporter offers, mapping Check Point logs to another format; CEF, LEEF, Splunk (CIM), etc.

Aside from LEA's lack of 64 bit support, one of the biggest challenges an integrator may have is mapping Check Point fields to their taxonomy. Currently our taxonomy isn’t well defined or uniform across our products, i.e. one product may use a different field name from another product. We're working to better define our log fields (see Threat Prevention Log Field Documentation ), but Log Exporter can also help by normalizing these fields into a common format. For example see Log Exporter CEF Field Mappings . 

Our Field Names	CEF Field Name
redirect_url	request
resource	request
url	request

P.S. regarding the upgrade question, you're probably aware of this, for others see (sk127653) How to backup and restore Log Exporter configuration on upgrade to R80.20/R80.20.M1 or as part of Ju... 

[Andras_Nagy]

Hello,

I just recognized that after enabling Log Exporter on R80.10 management the feature started to export the logs of the past one day. After this has been completed, actual logs are exported as expected.

The problem is that the /opt/CPrt-R80/log_exporter/targets/SIEM/log/log_indexer.elg file is continuoulsy growing at about a 500-600bytes/5s pace under "normal" circumstances (=no error messages just info about the normal operation of Log Exporter) when the logs are exported.

You can see an example screenshot of the logs below.

First question: this growth consumes disk capacity under /opt (/dev/mapper/vg_splat-lv_current). Will this log_indexer.elg file deleted or archived somehow automatically or do I have to do it manually?

Second question: what does the message containing "SyslogTCPSender::shouldRetry: Socket: [14] was write-blocked" mean? Is it an error or..?

Thank you in advance!

[Dror_Aharony]

That is the current design.

1. As all debug logs, it allows only 10 of these & then the oldest one is deleted, so no worries about disk-space.

2. This message is okay, it's meant for internal debugging purposes.

[Jeff_Gao]

@HeikoAnkenbrand How to export gateway's all secure logs to syslog server?sk87560 just only export firewall to syslog server and can not export secure logs to syslog server.