Log Exporter Filtering

[Dan_Zada]

Hello all,

I'm happy to inform you that we added a new feature to the log exporter - the ability to filter logs.

Starting today, you will be able to configure which logs will exported, based on fields and values, including complex statements.

More information, including basic and advanced filtering instructions, can be found in SK122323.

If you have any question or comment, let me know.

Thanks!

Dan.

[Danny]

Great innovation!

[Maarten_Sjouw]

Very good addition, many people have asked for it.

Is there anywhere a list of the field names can be found with possibly an explanation?

[Dan_Zada]

We are working on one.

Stay tuned

[Tom_Cripps]

Hi Dan,

I'm trying to filter on Source IP, can the filter be of a network group at all? or even subnets?

Thanks,

Tom

[Jerry]

oh yes indeed great stuff! Thanks guys 🙂 well done!

[Dan_Zada]

Hi @Maarten_Sjouw,
I'm happy to share with you (and with everyone else) that we released the full mapping for Check Point logs fields:
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

[Maarten_Sjouw]

Great Dan, that really helps to sort out what we can do with it.

[Peter_Baumann]

Hi all,

A customer has tried to do a filtering for the origin so that he only get the logs sent from one firewall in the config.
It was not possible to use the following:

[Expert@mgmt1:0]# cp_log_export set name Filtered filter-origin-in "fw-test"
Error: Argument [filter-origin-in] is undefined for command: [set]

[Expert@mgmt1:0]# cp_log_export add name splunk domain-server fwadm.local filter-origin-in fw-test
Error: Argument [filter-origin-in] is undefined for command: [add]

[Expert@mgmt1:0]# cp_log_export set name splunk domain-server fwadm.local filter-origin-in fw-test
Error: Argument [filter-origin-in] is undefined for command: [set]

The documentation is very confusing without some adequate examples.

Does someone have examples for origin and/or action filtering?

Thanks

[Garrett_Anderso]

would be nice to understand what is used to QA the "filtering" features.  at min, this would provide good examples for use.

Also, since sk122323 references "newly added feature" of filtering, this implies it requires a specific build of Log Exporter.  I don't see any references to build numbers and/or whether the "filter" feature will be added as part of HFA, etc. 

In other words:   will the "filter" feature Log Exporter require a patch even for platforms like R80.20 (where Log Exporter is native)?

[PhoneBoy]

Where Log Exporter is integrated (R80.20), the logical method for updating it is via the Jumbo Hotfix mechanism.
And, as it turns out, there's a note in the Log Exporter SK that says this is NOT included in R80.20 as of yet...

[Garrett_Anderso]

thanks D. appreciate it.

[Dror_Aharony]

Hi Peter,

Your command is absolutely correct. It should work (as the examples are accurate).

Which CP-Version & Exporter-HF version are you using (if any)?

Keep in mind, that the new filtering options are currently only available for R77.30 & R80.10 (with the latest exporter-HF), so I'd guess you simply don't have the filtering feature on your server yet, that is probably the source of confusion here.

It should be available on R80.20/.30 JHFs pretty soon...

[Peter_Baumann]

Hi Dror,

Thanks for the update.
The customer is using R80.20 Take 47.
It seems there's no version visible in cp_log_export, therefore here's the MD5 checksum:

11f4776c10b7b02d67ba72ee5cd00953 /opt/CPrt-R80.20/bin/cp_log_export

[Dror_Aharony]

Hi Peter,

as I suspected, the entire filtering feature for log-exporter isn't available in R80.20 yet. 

Coming soon...

[Garrett_Anderso]

hello @Dror_Aharony 

thanks for the updates and insight on JHA requirements for LogExporter filtering for R80.20/.30.     I would expect to find that information in SK122323.

The documentation enhancement request is valid to have examples inserted into filter section of SK122323.    I suggest not only what works, but more importantly, what doesn't work. 

Example for cp_log_export usages:

cp_log_export set name <name> filter-action-in "value1,value2"
cp_log_export set name <name> filter-origin-in "value1,value2"
cp_log_export set name <name> filter-blade-in "value2"

* The name of the field to filter on should be the mapped name in case it is changed in the mapping XML, or the original raw name if it is not mapped.

* The value of the field to filter on should be the raw field values.

[Dror_Aharony]

Hi Garrett,

We've added a few examples to the sk (SK122323) of using the log-exporter with the new filtering feature.

Hope it helps a bit.

[Garrett_Anderso]

@Dror_Aharony thank you for the update!!   much appreciated.   -GA

[Svetlana_Klasni]

Hi Dan,

I tried to filter logs on CP SMS R80.20, Gaia Kernel Version: 3.10, Jumbo Hotfix General availability (Take 87), but without success:

 cp_log_export set name RSYSLOG-TCP filter-action-in "Drop"
Error: Argument [filter-action-in] is undefined for command: [set]

The sending logs to our RSYSLOG server was configured earlier:

[Expert@CASCPSMS:0]# cp_log_export show

name: RSYSLOG-TCP
enabled: true
target-server: 10.0.9.99
target-port: 5519
protocol: tcp
format: syslog
read-mode: raw

Why I do not have  use filter-action-in argument for cp_log_export command?

Best regards,

                 Svetlana

[Maarten_Sjouw]

Svetlana from SK 122323:
R80.20
Log Exporter is already integrated in R80.20. There is no need to install in it a dedicated package.

Note: Filtering ability is not integrated to R80.20 and R80.30 yet, this SK will be updated when it will be supported.

[genisis__]

Have a question.

We know that log exporter can be used to export logs to an external logging server, however what I would like to do is export IPS and Threat Prevention logs from originating from specific GWs or CMA.

Example:

Scenario I:

SmartEvent is used in our environment which clearly is already consolidating entries; however this is consolidating entries for every client our Provider.  We need to export a specific customers entries to any external collector such as ArcSight.

Is there a way to export logs from a specific CMA or GWs from here?

Scenario II:

From the CMA how can we extract IPS/Threat Prevention logs to an external logging server (ArcSight I believe).  Ideally logs that originate from specific GWs.

The SK122323  above does not really show me any examples that make sense to me.  We are running R80.20 with HFA103 as well.

[Dror_Aharony]

the log-exporter filtering ability allows you to do just that:

You can activate/use an exporter that runs on that specific CMA (as the Log-Server), that will only export logs received by it - that CMA.

to export logs from a specific GW only, simply use the origin field, as in this example:

cp_log_export set name <name> filter-origin-in "GW_name"

Good Luck!

[Maarten_Sjouw]

@Dror_Aharony can this also be used for a group of gateways?

[Dror_Aharony]

that depends, who's the origin GW that appears in the logs?

if they are separate, which I believe they are, then you need to write all of them, as in the sk's example ("GW1,GW2,GW3")

cp_log_export set name <name> filter-origin-in "GW1,GW2,GW3"

You can verify in the FilterConfiguration.xml that there's an OR in that field (origin) with all your matching values. Should look like that:

                <field name="origin" operator="or">
                        <value operation="eq">GW1</value>
                        <value operation="eq">GW2</value>
                        <value operation="eq">GW3</value>
                </field>

[Maarten_Sjouw]

Ok, well that will be a bit of a hassle when you add multiple gateways every week, While a group of origin gateways would just be updated when the gateway is added in the SmartConsole.

[genisis__]

In our case we only have a certainly set of GWs we require the logs from, however we only want IPS/TP logs be to exported.

<field name="origin" operator="or">
             <value operation="eq">GW1</value>
             <value operation="eq">GW2</value>
             <value operation="eq">GW3</value>

</field>

So the above is great! just need to expand on this to determine only IPS/TP logs from the above?

[Dror_Aharony]

Yea, simply add the IPS/TP as in the sk examples, like this for both TP & Multi-GWs.

cp_log_export set name <name> filter-blade-in "TP" filter-origin-in "GW1,GW2,GW3"

or

   cp_log_export set name <name> filter-blade-in "SmartDefense" filter-origin-in "GW1,GW2,GW3"

 change/combine them both if needed. 

[genisis__]

Will this same process work on a SmartEvent appliance?

ie. origin GWs only would be required as the SmartEvent should only be storing consolidated events.

[Dror_Aharony]

exactly the same for using log-exporter on a SmartEvent too, for any exporter.

[Maarten_Sjouw]

@Dror_Aharony, @Dan_Zada any idea when the filtering will also be available on the R80.30 release?