cancel
Showing results for 
Search instead for 
Did you mean: 
Post a Question
Biju_Nair
Nickel

Re: R80.10 Syslog Exporter

Hey Yonatan, thank you so much. it worked for me.

Even though adding the corrected parameter(name) didn't help, but removing the target directory and recreating it helped for me.

Re: R80.10 Syslog Exporter

Expect new ports with R80.20. There is something going on on port 4434 that is web related.

0 Kudos

Re: R80.10 Syslog Exporter

Hello Hugo,

I saw your post on regarding the R80.20 M1 release.

I have to admit I wasn't aware of any issue with the ports and I've asked the relevant person to look into it.

I'm not familiar with any planned change to the log exporter ports in R80.20.

Nor am I familiar with any new issue with the Log Exporter on R80.20.

If you are aware of any specific issue with the Log Exporter on R80.20 M1 please send me an email with details to (edited as we are already GA for R80.20)

Thanks!

 Yonatan 

phlrnnr
Copper

Re: R80.10 Syslog Exporter

What are the performance implications of installing the Log Exporter?  What is the typical cpu/memory footprint?

0 Kudos

Re: R80.10 Syslog Exporter

Each log exporter can consume up to 1 CPU when exporting in full capacity.

0 Kudos
phlrnnr
Copper

Re: R80.10 Syslog Exporter

I installed a log exporter on a fairly busy log server.  The log_exporter process immediately jumped to ~100% cpu (1 core).  Does this mean I need to install multiple log_exporter processes?  If so, how do I do that? 

Are there any troubleshooting commands I can use to see if the log_exporter process is not able to send all logs to the syslog server?

output from

0 Kudos

Re: R80.10 Syslog Exporter

You don't really need to do any troubleshooting just yet - a CPU spike is expected after enabling the process for the first time.

The process will export all logs going back (by default) 1 day and will start exporting them at the maximum possible rate until it empties the queue, at which point it will go into the 'steady state' of exporting logs as they arrive.
This is the expected behavior. If the process goes down for any reason it will not lose any logs as we save a record of which logs were exported, and any that were not will be exported as soon as the process is back up and running. 

Depending on how many logs you generate per day this can take anywhere from a few seconds to a few hours.

From the screenshot, it looks like the process has been running for 4 hours already which leads me to believe you probably generate a large number of logs per day.

If the process is still at 100% CPU after a few more hours I would check the elg file to see what is the current, average and total amount of logs exported (the 'cp_log_export status' command will also give you the path to the elg file).

If the CPU load persists for more than a day, this might be an indication of a problem and you should investigate further and/or open a support ticket.

HTH 

 Yonatan 

0 Kudos
phlrnnr
Copper

Re: R80.10 Syslog Exporter

Thank you very much for your quick reply!  Yes, after a few hours, things seem to have calmed down, and the process is ~ 15%.  Thanks for the explanation!

Re: R80.10 Syslog Exporter

We tried to use Log Exporter following SK122323, but it doesn't work as expected.

R80.10 with jumbo take 112

cp_log_export status

name: syslog-prova
     status: Running (24439)
     last log read at: 5 Nov 16:52:30
     debug file: /opt/CPrt-R80/log_exporter/targets/syslog-prova/log/log_indexer.elg

log_indexer.elg seems to report reading data, but doesn^t send anything


[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:15] Files read rate [log] : Current=44 Avg=1649 MinAvg=54 Total=2492257 buffers (0/0/0/0)

[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:15] Sent current: 0 average: 0 total: 0

[log_indexer 24439 3872390032]@meteoam-center2[5 Nov 16:25:19] Files read rate [adtlog] : Current=0 Avg=0 MinAvg=0 Total=12 buffers (0/0/0/0)

[log_indexer 24439 3872390032]@meteoam-center2[5 Nov 16:25:19] Sent current: 0 average: 0 total: 0

[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:20] Files read rate [log] : Current=42 Avg=1644 MinAvg=54 Total=2492467 buffers (0/0/0/0)

[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:20] Sent current: 0 average: 0 total: 0

[log_indexer 24439 3872390032]@meteoam-center2[5 Nov 16:25:24] Files read rate [adtlog] : Current=0 Avg=0 MinAvg=0 Total=12 buffers (0/0/0/0)

[log_indexer 24439 3872390032]@meteoam-center2[5 Nov 16:25:24] Sent current: 0 average: 0 total: 0

[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:25] Files read rate [log] : Current=49 Avg=1638 MinAvg=54 Total=2492708 buffers (0/0/0/0)

[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:25] Sent current: 0 average: 0 total: 0

0 Kudos

Re: R80.10 Syslog Exporter

cp_log_export show

name: syslog-prova
     enabled: true
     target-server: 172.22.223.71
     target-port: 514
     protocol: udp
     format: syslog

0 Kudos

Re: R80.10 Syslog Exporter

Logs from Gaia to the same server instead work as expected.

meteoam-center2> show syslog all
Syslog Parameters:
    Remote Address 172.22.223.71
        Levels info
    Auditlog permanent

0 Kudos

Re: R80.10 Syslog Exporter

Your output actually indicates that everything is working as expected and that you have already exported ~2.5M logs at an average rate of 1.6K logs/sec:

[log_indexer 24439 4007656336]@meteoam-center2[5 Nov 16:25:25] Files read rate [log] : Current=49 Avg=1638 MinAvg=54 Total=2492708 buffers (0/0/0/0)

You have also exported 12 audit logs (I'm assuming this was the value that confused you - the elg has information for both fw.log and fw.adtlog):

[log_indexer 24439 3872390032]@meteoam-center2[5 Nov 16:25:24] Files read rate [adtlog] : Current=0 Avg=0 MinAvg=0 Total=12 buffers (0/0/0/0)

A useful debugging tool is the following command:

# tcpdump port 514 -s0 -A

[Expert@MDS-72:0]# tcpdump port 3010 -s0 -A
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:50:58.754248 IP MDS-72.61205 > 192.168.32.76.gw: P 1:297(296) ack 1 win 6 <nop,nop,timestamp 158701155 620400757>E..\..@.@..... H.. L......{.MJ......"N.....u.c$..uCEF:0|Check Point|Security Gateway/Management|Check Point|Log|Log|Unknown|deviceDirection=0 msg=Contracts outcome=Started rt=1541504854000 loguid={0x5be18022,0x0,0x4820a8c0,0x3d1779fc} origin=192.168.32.72 sequencenum=1 version=5 product=Security Gateway/Management update_service=1 version=1.0

13:50:58.754322 IP MDS-72.61205 > 192.168.32.76.gw: . 297:1745(1448) ack 1 win 6 <nop,nop,timestamp 158701155 620400757>E.....@.@..X.. H.. L......|.MJ.......3.....u.c$..uCEF:0|Check Point|Security Gateway/Management|Check Point|Log|Log|Medium|cp_severity=Medium deviceDirection=0 msg=Contracts outcome=Failed rt=1541504854000 loguid={0x5be18022,0x1,0x4820a8c0,0x3d1779fc} origin=192.168.32.72 sequencenum=2 version=5 failure_impact=Contracts may be out-of-date product=Security Gateway/Management reason=Server replied with no results. update_service=1 version=1.0

This will allow you to see the actual logs in real time as they are being exported.

If you are also sending OS logs or sending logs using other methods on the relevant port you might want to add more filters to the tcpdump command.

HTH 

 Yonatan 

Re: R80.10 Syslog Exporter

Hi i am trying to change the format from syslog to generic 

i have tried the below

what am i missing ?

any help appreciated

[Expert@gw-920ce3:0]# cp_log_export set format generic
Error: Missing mandatory argument <name> for command set

[Expert@gw-920ce3:0]# cp_log_export delete syslogser --apply-now
Error: Argument [syslogser] is undefined for command: [delete]

0 Kudos

Re: R80.10 Syslog Exporter

just missing the name.... 


[Expert@R80_M:0]# cp_log_export set name myCEF format generic
Export settings for myCEF has been changed successfully
To apply the changes run: cp_log_export restart name myCEF

[Expert@R80_M:0]# cp_log_export delete name myCEF --apply-now
Stopping log_exporter for: myCEF
Removing /opt/CPrt-R80.20/log_exporter/targets/myCEF

Re: R80.10 Syslog Exporter

Thank you that worked very appreciated

Alex_Tooze
Nickel

Re: R80.10 Syslog Exporter

We have a new deployment coming up, which will require SPLUNK feeds for traffic and audit logs. We already have both OPSEC & LogExporter defined on our existing R80.10 platform. We use OPSEC LEA for SPLUNK, and LogExporter for a different SIEM system.

As LogExporter is currently installed as an HFA, it is an additional overhead when carrying out upgrades.

Is there any big reason why we should be using one of these solutions over the other?

Thanks.

0 Kudos
Highlighted
Admin
Admin

Re: R80.10 Syslog Exporter

Log Exporter should be independent of HFAs

Alex_Tooze
Nickel

Re: R80.10 Syslog Exporter

It isn't, as far as I am aware. In order to install a new JHFA, it is currently necessary to remove LogExporter, then previous JHFA, before installing the new JHFA and reinstalling LogExporter.

The main point of my question, though, is whether we should be using LogExporter in preference to an OPSEC LEA connector, or stick with tried-and-tested OPSEC.

0 Kudos

Re: R80.10 Syslog Exporter

From Check Point’s perspective Log Exporter is the preferred integration going forward. Having worked with LEA for a while would say LEA provides more hooks, but lacks one key thing that Log Exporter offers, mapping Check Point logs to another format; CEF, LEEF, Splunk (CIM), etc.

Aside from LEA's lack of 64 bit support, one of the biggest challenges an integrator may have is mapping Check Point fields to their taxonomy. Currently our taxonomy isn’t well defined or uniform across our products, i.e. one product may use a different field name from another product. We're working to better define our log fields (see Threat Prevention Log Field Documentation ), but Log Exporter can also help by normalizing these fields into a common format. For example see Log Exporter CEF Field Mappings . 

 

Our Field Names

CEF Field Name

redirect_url

request

resource

request

url

request

P.S. regarding the upgrade question, you're probably aware of this, for others see (sk127653How to backup and restore Log Exporter configuration on upgrade to R80.20/R80.20.M1 or as part of Ju...