Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
andy_currigan
Contributor

Check Point Log Export - filtering field

we want to export checkpoint logs to a syslog server using the cp log exporter, for privacy reasons we want to remove certain sensitive fields such as username.

According to the sk122323"The filtering feature allows to decide which logs will be exported based on values from the various fields on the raw log but not to filter an entire field, anyone has some experience? 

Is it possible?

tks

 

0 Kudos
12 Replies
Daniel_
Collaborator

I tried it and AFAIK you can just "tell" the system wich logs you would like to send. So you can't filter fields which is quite annoying. So if you are going to activate a new blade or upgrade your system (and fields change) you have to remember that you have to modify log exporter....

0 Kudos
Dror_Aharony
Employee
Employee

You can filter-out fields by field name, however you wish, but for multiple fields you'll need to edit it via CLI in the relevant configuration.xml file:

using the <exported>true/false attribute to determine whether you wish to filter-in only this field, or filter-out only this field

& via the general <exportAllFields>false/true depending whether you wish to start by filtering-out specific fields from all, or filter-in each field starting from nothing.

Hope that's clear.

See the log-Exporter sk122323's section below.

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

Field Mapping Configuration XML section

<exported></exported>

 [optional] You can filter out specific fields by using the 'exported' true/false tag in the mapping configuration file. Alternatively, if the 'exportAllFields' tag in the 'targetConfiguration.xml' file is set to false, only those fields which are listed in the mapping file will be exported.. 

 true \ false

 

andy_currigan
Contributor

 [optional] You can filter out specific fields by using the 'exported' true/false tag in the mapping configuration file. Alternatively, if the 'exportAllFields' tag in the 'targetConfiguration.xml' file is set to false, only those fields which are listed in the mapping file will be exported.. 

 

ok in this case we'll set the exportAllFields to false but where can I find the mapping file in order to set the fields we want to export?

thanks

0 Kudos
Dror_Aharony
Employee
Employee

in the relevant Field formatting mapping file. either in the general FieldsMapping.xml or the relevant format's fields mapping xml file.

Use & Edit (backup original) your chosen format's mapping file in the targetConfiguration.xml (format fields mapping files are under ./conf folder)

<mappingConfiguration>...

 

0 Kudos
Daniel_
Collaborator

Thanks for the hint.

So I tried it with targetConfiguration.xml

<mappingConfiguration>myfilter.xml</mappingConfiguration>
<exportAllFields>true</exportAllFields>

 

And myfilter.xml

<?xml version="1.0" encoding="utf-8"?>
<fields>
<field><origName>origin_sic_name</origName><exported>false</exported><required>false</required></field>
<field><origName>version</origName><exported>false</exported><required>false</required></field>
<field><origName>loguid</origName><exported>false</exported><required>false</required></field>
<field><origName>logid</origName><exported>false</exported><required>false</required></field>
<field><origName>flags</origName><exported>false</exported><required>false</required></field>
<field><origName>originsicname</origName><exported>false</exported><required>false</required></field>
<field><origName>__policy_id_tag</origName><exported>false</exported><required>false</required></field>
<field><origName>layer_uuid</origName><exported>false</exported><required>false</required></field>
<field><origName>match_id</origName><exported>false</exported><required>false</required></field>
<field><origName>parent_rule</origName><exported>false</exported><required>false</required></field>
<field><origName>service_id</origName><exported>false</exported><required>false</required></field>
<field><origName>sequencenum</origName><exported>false</exported><required>false</required></field>
<field><origName>nat_addtnl_rulenum</origName><exported>false</exported><required>false</required></field>
</fields>

It's better then my first try but as you can see I need a lot more filters. IMHO by default it filters some fields already.

BUT I can not remove some fields like "match_id", "parent_rule" and "layer_uuid" even I have configured them to exported false.

Any ideas? Running MDS on 80.30 with take 191

0 Kudos
Dror_Aharony
Employee
Employee

those are table fields, that need the right <table> enveloping it (similar to application fields in other format examples).

for these fields, Add the <match_table> tableName to envelop these fields.

<table> Some fields will appear in tables depending on the log format. This information can be found in the elg log - one entry for every new field. A field can appear in multiple tables, each distinct instance is considered as a new field.   

 

like this:

<table><tableName>match_table</tableName>

  <fields>

...

  </fields>
</table>



Full CEF example
cat ./conf/CefFieldsMapping.xml | grep match_table -A 30
<table><tableName>match_table</tableName>
  <fields>
<field><origName>appi_name</origName><dstName>cs6</dstName><dstLabel>cs6Label</dstLabel><dstLabelVal>Application Name</dstLabelVal></field>
<field><origName>matched_category</origName><dstName>cs5</dstName><dstLabel>cs5Label</dstLabel><dstLabelVal>Matched Category</dstLabelVal></field>
<field><origName>app_sig_id</origName><dstName>flexString1</dstName><dstLabel>flexString1Label</dstLabel><dstLabelVal>Application Signature ID</dstLabelVal></field>
<field><origName>category</origName><dstName>cs2</dstName><dstLabel>cs2Label</dstLabel><dstLabelVal>Category</dstLabelVal></field>
<field><origName>rule_name</origName><dstName>cs2</dstName><dstLabel>cs2Label</dstLabel><dstLabelVal>Rule Name</dstLabelVal></field>
  </fields>
</table>

0 Kudos
Daniel_
Collaborator

Thanks for the hint but I didn't got it 🙄

If I would like to filter layer_uuid: I took a look to sk144192 and see:

2020-09-24 12_54_30-Log Fields Description.png

So I tried it with you match_table:

 

<?xml version="1.0" encoding="utf-8"?>
<table>
        <tableName>match_table</tableName>
        <fields>
                <field><origName>origin_sic_name</origName><exported>false</exported><required>false</required></field>
                <field><origName>version</origName><exported>false</exported><required>false</required></field>
                <field><origName>loguid</origName><exported>false</exported><required>false</required></field>
                <field><origName>logid</origName><exported>false</exported><required>false</required></field>
                <field><origName>flags</origName><exported>false</exported><required>false</required></field>
                <field><origName>originsicname</origName><exported>false</exported><required>false</required></field>
                <field><origName>__policy_id_tag</origName><exported>false</exported><required>false</required></field>
                <field><origName>layer_uuid</origName><exported>false</exported><required>false</required></field>
                <field><origName>match_id</origName><exported>false</exported><required>false</required></field>
                <field><origName>parent_rule</origName><exported>false</exported><required>false</required></field>
                <field><origName>service_id</origName><exported>false</exported><required>false</required></field>
                <field><origName>sequencenum</origName><exported>false</exported><required>false</required></field>
                <field><origName>nat_addtnl_rulenum</origName><exported>false</exported><required>false</required></field>
        </fields>
</table>

 

 

But it didn't work (and I didn't understand it in detail):
ExportConfiguration::init failed to parse /opt/CPmds-R80.30/customers/<customer>/CPrt-R80.30/log_exporter/targets/<target>/targetConfiguration.xml No su
ch node (fields)

Can you provide me an XML which filters layer_uuid?

Thanks!

0 Kudos
Dror_Aharony
Employee
Employee

Your xml looks good. It's supposed to work.
Did the other fields get filtered-out properly & only layer_uuid remained or all similar table fields remain?
Please open a TAC ticket.

 

0 Kudos
Daniel_
Collaborator

If I didn't add <tableName>match_table</tableName> it gets filtered. If I add <tableName>match_table</tableName> cp_log_exporter didn't start.

I opened a (diamond) TAC request several month ago and also discussed it with professional service but IMHO the product is to complex (or the SKs are not good enough). Nobody knows how it works if you try to filter logs....

So I gave up and going to send it in the default configuration....

0 Kudos
Dror_Aharony
Employee
Employee

I see.
so to be clear, Without the tableName most fields were filtered-out, but not all fields as you like. layer_uuid was still exported?
that's something, at least.
if log-exporter didn't start, it means you had some-sort of a syntax mistake there, but never-mind, go without it.

I'll try to push for more understanding in this area & clearer example in sk.
We're also planning to develop the filtering GUI for log-Exporter, but that will take some-time. No ETA yet.

0 Kudos
Daniel_
Collaborator

Yes, without the tableName fields it filters most of the fields with the exception of: layer_uuid, match_id and parent_rule

A better SK with examples would be great. Thanks!

0 Kudos
asher
Contributor

HI

there is example for fieldmapping.xml file if i work the : <exportAllFields>false</exportAllFields> ?

means export only selected fields 

0 Kudos