Check Point Log Export - filtering field

[andy_currigan]

we want to export checkpoint logs to a syslog server using the cp log exporter, for privacy reasons we want to remove certain sensitive fields such as username.

According to the sk122323"The filtering feature allows to decide which logs will be exported based on values from the various fields on the raw log but not to filter an entire field, anyone has some experience? 

Is it possible?

tks

[Daniel_]

I tried it and AFAIK you can just "tell" the system wich logs you would like to send. So you can't filter fields which is quite annoying. So if you are going to activate a new blade or upgrade your system (and fields change) you have to remember that you have to modify log exporter....

[Dror_Aharony]

You can filter-out fields by field name, however you wish, but for multiple fields you'll need to edit it via CLI in the relevant configuration.xml file:

using the <exported>true/false attribute to determine whether you wish to filter-in only this field, or filter-out only this field

& via the general <exportAllFields>false/true depending whether you wish to start by filtering-out specific fields from all, or filter-in each field starting from nothing.

Hope that's clear.

See the log-Exporter sk122323's section below.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

Field Mapping Configuration XML section

<exported></exported>

[optional] You can filter out specific fields by using the 'exported' true/false tag in the mapping configuration file. Alternatively, if the 'exportAllFields' tag in the 'targetConfiguration.xml' file is set to false, only those fields which are listed in the mapping file will be exported..

true \ false

[andy_currigan]

 [optional] You can filter out specific fields by using the 'exported' true/false tag in the mapping configuration file. Alternatively, if the 'exportAllFields' tag in the 'targetConfiguration.xml' file is set to false, only those fields which are listed in the mapping file will be exported.. 

ok in this case we'll set the exportAllFields to false but where can I find the mapping file in order to set the fields we want to export?

thanks

[Dror_Aharony]

in the relevant Field formatting mapping file. either in the general FieldsMapping.xml or the relevant format's fields mapping xml file.

Use & Edit (backup original) your chosen format's mapping file in the targetConfiguration.xml (format fields mapping files are under ./conf folder)

<mappingConfiguration>...

[Daniel_]

Thanks for the hint.

So I tried it with targetConfiguration.xml

<mappingConfiguration>myfilter.xml</mappingConfiguration>
<exportAllFields>true</exportAllFields>

And myfilter.xml

<?xml version="1.0" encoding="utf-8"?>
<fields>
<field><origName>origin_sic_name</origName><exported>false</exported><required>false</required></field>
<field><origName>version</origName><exported>false</exported><required>false</required></field>
<field><origName>loguid</origName><exported>false</exported><required>false</required></field>
<field><origName>logid</origName><exported>false</exported><required>false</required></field>
<field><origName>flags</origName><exported>false</exported><required>false</required></field>
<field><origName>originsicname</origName><exported>false</exported><required>false</required></field>
<field><origName>__policy_id_tag</origName><exported>false</exported><required>false</required></field>
<field><origName>layer_uuid</origName><exported>false</exported><required>false</required></field>
<field><origName>match_id</origName><exported>false</exported><required>false</required></field>
<field><origName>parent_rule</origName><exported>false</exported><required>false</required></field>
<field><origName>service_id</origName><exported>false</exported><required>false</required></field>
<field><origName>sequencenum</origName><exported>false</exported><required>false</required></field>
<field><origName>nat_addtnl_rulenum</origName><exported>false</exported><required>false</required></field>
</fields>

It's better then my first try but as you can see I need a lot more filters. IMHO by default it filters some fields already.

BUT I can not remove some fields like "match_id", "parent_rule" and "layer_uuid" even I have configured them to exported false.

Any ideas? Running MDS on 80.30 with take 191

[Dror_Aharony]

those are table fields, that need the right <table> enveloping it (similar to application fields in other format examples).

for these fields, Add the <match_table> tableName to envelop these fields.

<table>

Some fields will appear in tables depending on the log format. This information can be found in the elg log - one entry for every new field. A field can appear in multiple tables, each distinct instance is considered as a new field.

like this:

<table><tableName>match_table</tableName>

  <fields>

...

  </fields>
</table>

Full CEF example
cat ./conf/CefFieldsMapping.xml | grep match_table -A 30
<table><tableName>match_table</tableName>
  <fields>
<field><origName>appi_name</origName><dstName>cs6</dstName><dstLabel>cs6Label</dstLabel><dstLabelVal>Application Name</dstLabelVal></field>
<field><origName>matched_category</origName><dstName>cs5</dstName><dstLabel>cs5Label</dstLabel><dstLabelVal>Matched Category</dstLabelVal></field>
<field><origName>app_sig_id</origName><dstName>flexString1</dstName><dstLabel>flexString1Label</dstLabel><dstLabelVal>Application Signature ID</dstLabelVal></field>
<field><origName>category</origName><dstName>cs2</dstName><dstLabel>cs2Label</dstLabel><dstLabelVal>Category</dstLabelVal></field>
<field><origName>rule_name</origName><dstName>cs2</dstName><dstLabel>cs2Label</dstLabel><dstLabelVal>Rule Name</dstLabelVal></field>
  </fields>
</table>

[Daniel_]

Thanks for the hint but I didn't got it 🙄

If I would like to filter layer_uuid: I took a look to sk144192 and see:

So I tried it with you match_table:

<?xml version="1.0" encoding="utf-8"?>
<table>
        <tableName>match_table</tableName>
        <fields>
                <field><origName>origin_sic_name</origName><exported>false</exported><required>false</required></field>
                <field><origName>version</origName><exported>false</exported><required>false</required></field>
                <field><origName>loguid</origName><exported>false</exported><required>false</required></field>
                <field><origName>logid</origName><exported>false</exported><required>false</required></field>
                <field><origName>flags</origName><exported>false</exported><required>false</required></field>
                <field><origName>originsicname</origName><exported>false</exported><required>false</required></field>
                <field><origName>__policy_id_tag</origName><exported>false</exported><required>false</required></field>
                <field><origName>layer_uuid</origName><exported>false</exported><required>false</required></field>
                <field><origName>match_id</origName><exported>false</exported><required>false</required></field>
                <field><origName>parent_rule</origName><exported>false</exported><required>false</required></field>
                <field><origName>service_id</origName><exported>false</exported><required>false</required></field>
                <field><origName>sequencenum</origName><exported>false</exported><required>false</required></field>
                <field><origName>nat_addtnl_rulenum</origName><exported>false</exported><required>false</required></field>
        </fields>
</table>

But it didn't work (and I didn't understand it in detail):
ExportConfiguration::init failed to parse /opt/CPmds-R80.30/customers/<customer>/CPrt-R80.30/log_exporter/targets/<target>/targetConfiguration.xml No su
ch node (fields)

Can you provide me an XML which filters layer_uuid?

Thanks!

[Dror_Aharony]

Your xml looks good. It's supposed to work.
Did the other fields get filtered-out properly & only layer_uuid remained or all similar table fields remain?
Please open a TAC ticket.

[Daniel_]

If I didn't add <tableName>match_table</tableName> it gets filtered. If I add <tableName>match_table</tableName> cp_log_exporter didn't start.

I opened a (diamond) TAC request several month ago and also discussed it with professional service but IMHO the product is to complex (or the SKs are not good enough). Nobody knows how it works if you try to filter logs....

So I gave up and going to send it in the default configuration....

[Dror_Aharony]

I see.
so to be clear, Without the tableName most fields were filtered-out, but not all fields as you like. layer_uuid was still exported?
that's something, at least.
if log-exporter didn't start, it means you had some-sort of a syntax mistake there, but never-mind, go without it.

I'll try to push for more understanding in this area & clearer example in sk.
We're also planning to develop the filtering GUI for log-Exporter, but that will take some-time. No ETA yet.

[Daniel_]

Yes, without the tableName fields it filters most of the fields with the exception of: layer_uuid, match_id and parent_rule

A better SK with examples would be great. Thanks!

[asher]

HI

there is example for fieldmapping.xml file if i work the : <exportAllFields>false</exportAllFields> ?

means export only selected fields