cancel
Showing results for 
Search instead for 
Did you mean: 
Create a Post

Re: Log Exporter guide

What I mean by that is that the LEA (OPSEC) connection currently is used to send instant log information to other systems that are mainly used for SIEM solutions. As I have seen some entries on LogEport mentioning similar functionality, but with additional lfiltering on top, it looks to me all the info getting to the log can be exported as well.

Currently the LEA connection is missing out some of the information, some log-in records ie and also not all fields seem to reach the external system.

So that's what I mean by it.

Regards, Maarten.

Regards, Maarten
0 Kudos
Admin
Admin

Re: Log Exporter guide

I believe the plan is to build new functionality/enhancements into the Log Exporter tool versus extend LEA. 

That doesn't mean LEA goes away, but different and more integrations will be possible with Log Exporter over time.

Re: Log Exporter guide

One of the problems currently with LEA is that when you want to setup a new connection at this moment, this connection will be using a SHA256 certificate and most SIEM systems did not update the LEA client to support the 256 certificates.

This is a reason why currently very few LEA connections will be used any longer for setting up a new connection.

Regards, Maarten
0 Kudos
Employee+
Employee+

Re: Log Exporter guide

I was recently asked about JSON support for the log exporter, and my initial reaction was that it isn't supported in the current release. 

However, after looking into this a bit more, I found that although some basic manipulation of the configuration files I'm able to reformat the logs into a format that passes JSON validators.

However, just because the output passes validators doesn't actually make it useful.

It's formatted as json, but a very basic and simplified form of json.
An example log would look like this:

{"action":"Drop", "ifdir":"inbound", "ifname":"eth1", "loguid":"{0x0,0x0,0x0,0x0}", "origin":"192.168.32.91",  "time":"1522592600", "version":"5", "dst":"10.32.30.255", "message_info":"Address spoofing", "product":"VPN-1 & FireWall-1", "proto":"17", "s_port":"137", "service":"137", "src":"10.32.30.20", "":""}

One thing that should be noted is that we do have duplicate keys, and while this is compliant with the RFC, it is not recommended (but can't be helped with the current release). 

I was wondering if anyone has a use case where such json output would be useful, and if so what specifically are the requirements? does it just have to just be json compliant or is there anything specific that you'd be looking for?

For those that are interested here is the configuration I used to transformat the log to a json format:

<start_message_body>{"</start_message_body>
<end_message_body>":""}</end_message_body>
<message_separator>&#10;</message_separator><!-- &#10;=='\n' -->
<fields_separatator>, "</fields_separatator>
<field_value_separatator>":</field_value_separatator>
<value_encapsulation_start>&quot;</value_encapsulation_start>
<value_encapsulation_end>&quot;</value_encapsulation_end>

Note: It adds an empty "":"" at the end of the log for convenience.

I'd be interested to hear feedback about json related use cases.

Regards,

 Yonatan  

0 Kudos
Vladimir
Pearl

Re: Log Exporter guide

Without thinking on this subject at length, one of the possible use cases will be integration with cloud services native logging and enforcement systems. We could, conceivably, output the json formatted events to the AWS CloudWatch to have native integrated metrics, alarms etc..

0 Kudos
Employee+
Employee+

Re: Log Exporter guide

But would any of those applications actually be able to make sense of the data?

Just because it can officially be called 'json' doesn't mean it's can actually be parsed into useful data. That's what I meant when I asked if there are any requirements other than just being 'json'?

0 Kudos
Vladimir
Pearl

Re: Log Exporter guide

It looks like in AWS' case, they do not much care what data you are logging to them, so long as it is in JSON:

Amazon CloudWatch Logs JSON Log Format Support 

You pretty much defining filters in CloudWatch yourself to look for patterns in your logs and define metrics, alerts and events based on those:

Filter and Pattern Syntax - Amazon CloudWatch Logs 

0 Kudos

Re: Log Exporter guide

Hi Yonatan Philip‌, 

I have a JSON requirement. I'd like to output the messages in cee-enhanced format in order for me to parse the message using rsyslog's mmjsonparse. On my rsyslog server I reformat the message into LEEF, CEF or JSON depending on the tool I'm forwarding too (in most cases all three) and ship it on.

Using your example above I inserted @cee: and some of the messages are successfully parsed:

@cee: {"action":"Reject", "flags":"2304", "ifdir":"inbound", "ifname":"daemon", "loguid":"{0x0,0x0,0x0,0x0}", "origin":"10.6.41.170", "time":"1530153441", "version":"1", "dst":"172.30.235.66", "encryption_failure:":"no response from peer.", "fw_subproduct":"VPN-1", "peer_gateway":"10.12.130.249", "proto":"6", "reject_category":"IKE failure", "rule":"0", "s_port":"46792", "scheme:":"IKE", "service":"18192", "src":"10.6.41.196", "vpn_feature_name":"IKE", "":""}

but others are not, probably due to the part in red:

@cee: {"action":"Accept", "flags":"51460", "ifdir":"outbound", "ifname":"eth0", "loguid":"{0x5b3466e3,0x5f4b0001,0x612906c4,0x7b6}", "origin":"10.6.41.97", "time":"1530160867", "version":"1", "__policy_id_tag":"product=VPN-1 & FireWall-1[db_tag={1AF555CB-2329-464E-986C-7D2991E1C63A};mgmt=sma-nscma001;date=1481830965;policy_name=Development_ISG_current\]", "dst":"10.6.41.171", "message_info":"Implied rule", "nat_addtnl_rulenum":"0", "nat_rulenum":"0", "product":"VPN-1 & FireWall-1", "proto":"17", "rule":"0", "s_port":"123", "service":"123", "service_id":"ntp-udp", "smartdefense_profile":"No Protection", "src":"10.6.41.97", "xlatedport":"0", "xlatedst":"0.0.0.0", "xlatesport":"10135", "xlatesrc":"10.6.41.99", "":""}

If you can see a better way to achieve this then I'd be interested to learn in.

Additionally, can you elaborate on which keys are duplicated?

Cheers,

Simon

0 Kudos
Highlighted
Employee+
Employee+

Re: Log Exporter guide

Hello Simon,

This is just a guess, but it looks to me as if the issue is probably the square brackets inside the log - that is inside the value fields (as part of the log data).

The way to resolve it is by escaping. Inside the format definition file, you have a section for escaping which looks like this:

<escape_chars>
<char><orig>\</orig> <!--MUST be first to prevent double escaping --> <escaped>\\</escaped></char>
<char><orig>"</orig><escaped>\"</escaped></char>
<char><orig>]</orig><escaped>\]</escaped></char>
<char><orig>&#10;</orig><escaped> </escaped></char>
</escape_chars>

(might have different values depending on the format you're using).

If you notice in the example above we escape the closing square brackets (]) with a backslash (\]), but not the opening square brackets. I can see the same thing in your log - which effectively means you opened the brackets and never closed it.

I'm sitting here scratching my head trying to remember if we did this on purpose or if this is an oversight/bug.

If you could test this and let me know if it helped I'd appreciate it, and in the meantime, I'll ask someone to double check if this is a bug.

Looking at the cee-enhanced documentation you might have to escape the curly brackets as well '{' '}'.

Note that some applications ignore backslash escaping - in such cases instead of escaping with a backslash, simply replace the bracket with another character of your choosing. Bear in mind that this changes the actual values, so if you pipe the values and/or parse them this needs to be taken into account.

Regarding duplicate values - there are several scenarios where this can happen, the easiest way to see an example is to create another layer (on an R80.10 GW).

The log will hold duplicate keys for many values such as rule name, layer name etc.

Edit: I checked, and escaping only the ending bracket was on purpose. We wished to minimize changes to the payload to the absolute minimum. We had to escape the closing bracket to avoid log corruption, but the opening bracket didn't have the same effect.

However, since it seems that this might impact an external (3'rd party) log parser, I'll open an RFE to update the default settings to escape both the opening and closing brackets.

HTH 

 Yonatan 

Re: Log Exporter guide

Hi Yonatan Philip‌,

After much head scratching today I finally got this to parse via mmjsonparse Smiley Happy

As you suggested I escaped/replaced the square brackets. My syslog.xml contains:

<start_message_body>@cee: {"</start_message_body>
<end_message_body>localinfo":"myceelog"}</end_message_body>
<message_separator>&#10;</message_separator><!-- &#10;=='\n' -->
<fields_separatator>,"</fields_separatator>
<field_value_separatator>":</field_value_separatator>
<value_encapsulation_start>&quot;</value_encapsulation_start>
<value_encapsulation_end>&quot;</value_encapsulation_end>
<escape_chars>
   <!--MUST be first to prevent double escaping -->
   <char><orig>\</orig><escaped>\\</escaped></char>
   <char><orig>"</orig><escaped>_</escaped></char>
   <char><orig>[</orig><escaped>_</escaped></char>
   <char><orig>]</orig><escaped>_</escaped></char>
   <char><orig>&#10;</orig><escaped></escaped></char>
</escape_chars>

The main obstacle was the message format, I had to force the timestamp to comply with RFC 5424 - The Syslog Protocol timestamp (I'm on R77.30 Mgmt perhaps that's why?):

<name>format_timestamp</name>
   <args>
      <arg key="format" value="%Y-%m-%dT%H:%M:%SZ"/>
   </args>

Interesting bits of information are:

  • some keys contain ":" but this didnt effect the jsonparser as I thought it might.
  • "{" "}" "=" ":" ";"  - all of which appear in the 'values' strings did not require escaping.

It would still be very useful if cee was a supported output format for future versions. For example, the key "policy_id_tag" contains a list for the value:

"__policy_id_tag": "product=VPN-1 & FireWall-1_db_tag={1E295337-3B29-FD42-8155-FC9FC0CE93D6};mgmt=my_cma_name;date=1530151258;policy_name=Standard_",

which would be better displayed as:

"__policy_id_tag": { "product": "VPN-1 & FireWall-1", "db_tag": "{1E295337-3B29-FD42-8155-FC9FC0CE93D6}", "mgmt": "my_cma_name", "date":"1530151258", "policy_name":"Standard" },

Thanks again for the quick support. Looking forward to working with this product much more in the coming weeks!

Cheers,

Simon

Employee+
Employee+

Re: Log Exporter guide

Hi Simon,

I'm glad to hear that you got it working!

Regarding the timestamp - we did indeed have a bug with the timestamp not being RFC compliant.

We actually released a minor update a few days ago with a few critical bug fixes - the two main bugs fixed were the time stampe issue and the default insatllation folder on R77.30 MDS servers.

Could you please verify if you're running the newer take of the log exporter?

If you're running the new take and still seing an issue with the time stamp format please let me know.
When you say that we should support cee I'm assuiming you mean add json format support?

We've gotten this request a few times and have it on the roadmap, but as far as I know it's somewhat down the list.

Although the numebr of such requests is growing so this might change.

HTH 

 Yonatan

0 Kudos

Re: Log Exporter guide

Hi Yonatan Philip,  

I’m using the T25 release.

And yes support for JSON would be the best option for me.

 

As I use an rsyslog server to aggregate/normalise our logs, the way I see it, once you have JSON you can craft any other format you like.

We run more than one brand of SIEM so by using JSON I can provide logs to each vendor. 

 

Cheers, Simon

0 Kudos
Employee+
Employee+

Re: Log Exporter guide

Regarding the JSON - it's becoming apparent that JSON support along with HTTP post might be something we will have to look into.

I've passed this feedback along to the relevant parties.

Regarding the timestamp issue - I passed it to R&D to investigate. It should have been fixed in take 25, we'll look into this.

0 Kudos
RPdeBeer
Ivory

Re: Log Exporter guide

Hi,

Is it possible to have the CMA's IP address as origin IP instead of the MDS IP?

Greetings,

R.P. de Beer

Employee+
Employee+

Re: Log Exporter guide

Hello Rutger-Paul,

We are aware of this issue. It was a limitation on the first release.

Because domains are actually aliases (on the network level) the exporter sends out the logs with the server’s IP which is the MDS IP.

This is addressed in R80.20, and we will possibly also address this in the next release for R80.10 as well.

Regards,

 Yonatan 

Biju_Nair
Nickel

Re: Log Exporter guide

yes i am also facing the same issue, the IP address shown in Arcsight server(syslog) is the MDS IP and not the CMA IP.

I was in a impression that i did something wrong while configuring  it. Because of the mdsenv does not show that you entered into the CMA. however i was able to find the config.xml file in the "target" directory showing that the new target was created in teh correct location.

Please do let us know if this is resolved in which Take of 80.10

(I am currently running T103 of 80.10)

Employee+
Employee+

Re: Log Exporter guide

While I don't currently have any definite information on this, I'm assuming it will be included in the next update of the log exporter hotfix.

I'm not sure when this will be ready/released. I believe that the current focus is integrating the log exporter into R80.20, but since I'm not directly involved in that effort I don't know for sure.

HTH

 Yonatan

0 Kudos
Ivo_Marques
Nickel

Re: Log Exporter guide

Hi Yonatan,

Do you know if "CMA IP" features was already integrated on the new version of Nov 6th? I could understand "what's new" in this new version.

Thanks,

Ivo

0 Kudos
Dan_Roddy
Copper

Re: Log Exporter guide

Hi Yonatan,

Would you mind telling me what format to use to send logs from R77.30.03 endpoint management server to R80.10 gateway management server?  It would be helpful if you could provide an example using this script.

cp_log_export add name <name> [domain-server <domain-server>] target-server <target-server> target-port <target-port> protocol <(udp|tcp)> format <(syslog)|(cef)> [optional arguments]

Thank you,

D. Roddy

Employee+
Employee+

Re: Log Exporter guide

HI Dan,

I'm sorry to say that we did not test log exporter with R77.30.03. 

 

I'm not familiar with this use case (sending logs from checkpoint to checkpoint), and to be honest, I personally have no experience with R77.30.03 (or endpoint in general).

I'll try and consult with some of my colleagues from endpoint but at this point I think that the answer is that it's not officially supported.

 

I think it's probably good advice to also request this through the regular channels (SE/RFE/solution center/etc.).

There is always a long laundry list of feature to add and test, prioritizing between them is often influenced by customer feedback and official requests. 

 

Regards,

 Yonatan 

0 Kudos

Re: Log Exporter guide

Hi Yonatan,

Just a quick question, i'm i right in saying log exporter only installs on your management server not on your gateways?

0 Kudos
Employee+
Employee+

Re: Log Exporter guide

Hi Conor,

The log exporter reads the logs from the local machine ($FWDIR/log by default) and exports them.

It relies on the indexing infrastructure to read the logs, which means the server has to have this infrastructure installed for it to be a valid target, which excludes gateways.

 

The generic answer would be – install the log exporter on the log server that stores the logs you wish to export.

Regards,

 Yonatan 

Re: Log Exporter guide

Can I get medium to high IPS events in syslog?

0 Kudos
Employee+
Employee+

Re: Log Exporter guide

Hello John,

I actually addressed something very similar in the FAQ:

That’s great! But I don’t really need the Application Control logs either. I only want the IPS logs to be sent. How can I do that?

Unfortunately, you can’t do that in this release.

You can't filter by specific blade or by specific action/severity/protection etc.

You can only filter by the field itself, not by the specific content of the field.

So while you can't set up a filter saying 'blade=IPS' you can set up a filter that says 'protection_name is required' which will mean that only logs with the 'protection_name' field will be sent.

But again, you will not be able to set up a filter that says 'severity=high'.

Those types of filters are on the roadmap but are not yet supported.

Yonatan 

0 Kudos

Re: Log Exporter guide

Hi,

Is this tool able to export the logs in checkpoint log format to an external log server to view in smartview tracker/smartlog without using syslog to checkpoint log parser?

Thanks.

Regards

Wilson

0 Kudos
Employee+
Employee+

Re: Log Exporter guide

Hello Swee,

Let me start by saying we did not test this use case. Everything I'm going to say is my opinion, but I could be mistaken as I haven't actually tried this.

The tool can export logs in the checkpoint format ('as is'). The difficulty isn't on the exporting side but on the receiving side. 

The receiving end will not be able to recognize that these logs are arriving from a 'legal' checkpoint server (with a SIC connection). The only way for it to accept those logs would be as a syslog. And while this is possible (there is a checkbox on the log server that enables it to receive syslogs), the logs will be treated as syslog.

That means that the log server will generate a syslog log and the payload will be put into the message field.

So while the data will be sent, it will not be parsed as a checkpoint log but rather as raw data.

Yonatan 

0 Kudos

Re: Log Exporter guide

Hi Yonatan,

Thanks for your reply.

If this is the case, does Checkpoint has any syslog to checkpoint log parser as per sk55020?

My external log server is not SIC with the management server that is why we are looking at sk55020.

I tried to use the log parser editor, but there are too many log fields in the syslog as compared to other syslogs and it varies with different versions. eg. r77.30 and r80.10.

Thanks.

Regards

Wilson

0 Kudos
Employee+
Employee+

Re: Log Exporter guide

Hi Wilson,

Again most of what I'm about to say is my own personal opinion.

I see a lot of issues with allowing a log server to accept logs from a 3rd party source without SIC or another type of authentication. It opens up your environment for log injection from unsanctioned sources.

This is why we added the TLS option to the log exporter. This was a feature that we had to develop for the log exporter specifically. this means that we are able to send logs using this mechanism. But for incoming logs, the mechanism in use is different.

This means that something will likely have to be created to either add SIC support to the log exporter somehow or add incoming TLS support to the log server itself, or something else along those lines.

There might be other options that I'm not considering or a way to easily manipulate the configuration to use an existing option, but it seems to me that it won't work (securely) out of the box and will need some development effort to make it work.

I'll raise this use case as something that we should look into as a roadmap item. I would also suggest that you try to ask for this through official channels  (SE/Solution Center/etc.). First off they might have suggestions that I haven't considered or options that I'm unaware of, and secondly, if enough such requests are made by customers it can have a major impact on prioritization of feature development.

Regards,

 Yonatan 

0 Kudos

Re: Log Exporter guide

Hi guys, Im using arcsight and Im having trouble setting this up.

I think its on the connector side, log exporter is reporting:

 

[log_indexer 23554 4107553680]@SmartEvent-vm[6 Jun 17:17:15] SyslogTCPSender::connect: Failed to initialize socket (x.x.x.x:514)

[log_indexer 23554 4107553680]@SmartEvent-vm[6 Jun 17:17:15] TcpTlsSender::connect: Failed to create socket.

 

The added lines to agent.properties should be ?

agents[0].syslogng.tls.keystore.file=user/agent/syslog-ng.p12
agents[0].syslogng.tls.keystore.alias=syslogng-alias

or

 

syslogng.tls.keystore.file=user/agent/syslog-ng.p12
syslogng.tls.keystore.alias=syslogng-alias

I've tried both ways, and no go. 

Also where should I put syslog-ng.p12 file ?

0 Kudos
Employee+
Employee+

Re: Log Exporter guide

Hello Rodrigo,

This question should probably be directed to an ArcSight expert (which I am definitely not).

I believe that the syslog-ng.p12 file should be located in your <connector folder>/current/user/agent/ folder (but I think you can use other locations if you specify it in the setting).

I think that you should use the commands as the sk instructs:

syslogng.tls.keystore.file=user/agent/syslog-ng.p12

syslogng.tls.keystore.alias=syslogng-alias

If it sounds as if I'm unsure of my answers, that's because I'm unsure of my answers Smiley Happy

I would definitely consult with an ArcSight expert on these points.

I would also suggest that this might be better handled via a support ticket. Debugging TLS connection is extremely difficult (at least for me...) under even ideal circumstances, and a forum post is less than ideal medium for debugging.

HTH 

 Yonatan 

0 Kudos