How clipping logs field to syslog server used log exporter?

[Herschel_Liang]

requirement:

How clipping logs field to syslog server used log exporter? For example:

12-18-2019 12:38:35 Local0.Info 172.22.14.40 1
2019-12-18T04:38:33Z SMS CheckPoint 28309 - [action:"Accept"; flags:"411908";
ifdir:"inbound"; ifname:"eth1"; logid:"0";
loguid:"{0x5df9ad49,0x0,0x290e16ac,0xc0000001}"; origin:"172.22.14.41";
originsicname:"CN=GW,O=SMS..pekhdi"; sequencenum:"2"; time:"1576643913";
version:"5"; __policy_id_tag:"product=VPN-1 &
FireWall-1[db_tag={F8CCF5D5-A96C-FC47-89B5-DE562533C7A6};mgmt=SMS;date=1576562059;policy_name=Standard\]";
dst:"180.163.222.208"; log_delay:"1576643913"; layer_name:"Network";
layer_uuid:"c0264a80-1832-4fce-8a90-d0849dc4ba33"; match_id:"2";
parent_rule:"0"; rule_action:"Accept"; rule_name:"Cleanup rule";
rule_uid:"e5c72b7a-811f-4b4b-a6fe-2cf646e0b4c7"; product:"VPN-1 &
FireWall-1"; proto:"6"; s_port:"23031"; service:"80"; service_id:"http";
src:"172.22.14.56"; ]
----------------------------------->
12-18-2019 12:38:35 Local0.Info 172.22.14.40 1
2019-12-18T04:38:33Z SMS CheckPoint 28309 - [action:"Accept";
origin:"172.22.14.41"; dst:"180.163.222.208"; proto:"6"; s_port:"23031";
service:"80"; service_id:"http"; src:"172.22.14.56"; ]

The client just want to send useful fields to syslog server but not all fields.

sk122323

Resolver Parameters

Parameter	Description	Possible/Default Values
<mappingConfiguration></mappingConfiguration>	The XML file containing the log field mapping scheme. If left empty will use the default settings.	Default values are based on the 'format'.
<exportAllFields>true</exportAllFields>	When this field is set to 'true' all log fields will be sent regardless of whether or not they appear in the mapping scheme, , except for specifically black-listed fields in the relevant log format mapping file (<exported>false</exported>). When set to 'false' only those fields which appear in the relevant log format mapping file will be sent (with exported flag true: <exported>true</exported>)	true / false

If I set field as <exported>false</exported>, it seem that I can change export fields in mappingConfiguration, but I have no more information about grammar and syntax in that files(targetConfiguration.xml &FieldsMapping.xml). Can someone give me a good advice? THX!

[Dror_Aharony]

See the log-exporter sk122323 referring to the fields description sk144192 (https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...)

It states all fields raw names – that should help you to rather easily avoid exporting specific fields by clipping the exported logs to your syslog server.

[Herschel_Liang]

Off course, it will help but not change xml grammar and syntax.

[Dror_Aharony]

Not sure I understand what you mean regarding grammar/syntax.

Can you show an example or elaborate?

[Herschel_Liang]

just a example, such as:
if I change targetConfiguration.xml
<exportAllFields>false</exportAllFields>
and change FieldsMapping.xml
<fields>src</fields>
<fields>dst</fields>
<fields>proto</fields>
<fields>port</fields>
<fields>action</fields>
<fields>sxlate</fields>
I can implementation my requirement through exporting what I want to by changing xml files

[Dror_Aharony · Accepted solution]

if those are the only fields you'd like to export, then yea.

but you must include the exact syntax of the xml, best including: <exported>true</exported> for each field.

like this:

<field>
<exported>true</exported><origName>src</origName><dstName>src</dstName>
</field>

See full fieldMapping.xml example for guide usage

<?xml version="1.0" encoding="utf-8"?>
<fields>
<!-- field names/types of tables/fields can be found in fw.lf file -->
<field>
<exported>true</exported><!-- optional, field that is not exported won't be sent / by default it's true-->
<origName>time</origName><!-- field name in the log, can be different between joined and not joined log -->
<dstName>start</dstName><!-- the field will be exported with this name -->
<required>true</required><!-- optional, traffic without this field is dropped/default is false-->
</field>
<field>
<origName>src</origName>
<dstName>cef_src</dstName>
</field>
<field>
<origName>dst</origName>
<dstName>cef_dst</dstName>
</field>
<table>
<tableName>match_table</tableName><!-- name of the table/ field of the table is changed during join, see log_unification_scheme.C -->
<tableFormat></tableFormat><!-- optional, the format of table export, by default flat-->
<exported>true</exported><!-- optional, field that is not exported won't be sent / by default it's true-->
<required>false</required><!-- optional, traffic without this field is dropped/default is false-->
<fields>
<field>
<origName>layer_name</origName>
<dstName>cef_layer_name</dstName>
</field>
</fields>
</table>

[Herschel_Liang]

<?xml version="1.0" encoding="utf-8"?>
<fields>
<field>
<exported>true</exported>
<origName>time</origName>
<dstName>start</dstName>
<required>true</required>
</field>
<field>
<exported>true</exported>
<origName>src</dstName>
<dstName>src</required>
<required>true</required>
</field>
<field>
<exported>true</exported>
<dstName>dst</dstName>
<required>dst</required>
<required>true</required>
</field>
<field>
<exported>true</exported>
<dstName>proto</dstName>
<required>proto</required>
<required>true</required>
</field>
<field>
<exported>true</exported>
<dstName>protocol</dstName>
<required>protocol</required>
<required>true</required>
</field>
<field>
<exported>true</exported>
<dstName>xlatesrc</dstName>
<required>xlatesrc</required>
<required>true</required>
</field>
<field>
<exported>true</exported>
<dstName>xlatedst</dstName>
<required>xlatedst</required>
<required>true</required>
</field>
<table>
<tableName>match_table</tableName><!-- name of the table/ field of the table is changed during join, see log_unification_scheme.C -->
<tableFormat></tableFormat><!-- optional, the format of table export, by default flat-->
<exported>true</exported><!-- optional, field that is not exported won't be sent / by default it's true-->
<required>false</required><!-- optional, traffic without this field is dropped/default is false-->
<fields>
<field>
<origName>layer_name</origName>
<dstName>cef_layer_name</dstName>
</field>
</fields>
</table>
</fields>
Is that right? but it also fail
[Expert@SMS:0]# cp_log_export status
name: test1
status: Not running
last log read at: 19 Dec 16:35:40
debug file: /opt/CPrt-R80.30/log_exporter/targets/test1/log/log_indexer.elg
[Expert@SMS:0]# tail /opt/CPrt-R80.30/log_exporter/targets/test1/log/log_indexer.elg
[log_indexer 4251 4128230208]@SMS[23 Dec 18:33:25] SourceConfiguration::init: - Reading logs mode is raw
[log_indexer 4251 4128230208]@SMS[23 Dec 18:33:25] SourceConfiguration::init - read log files from directory: [/opt/CPsuite-R80.30/fw1/log]
[log_indexer 4251 4128230208]@SMS[23 Dec 18:33:25] ExportConfiguration::setLinkIP - export_link_ip is set to default
[log_indexer 4251 4128230208]@SMS[23 Dec 18:33:25] No default mapping file found.
[log_indexer 4251 4128230208]@SMS[23 Dec 18:33:25] The mapping configuration must be defined in case not all fields are exported!

[Herschel_Liang]

I have tried many times, but it seem failed in change xml(fieldsMapping.xml ).

Always errors:

[log_indexer 44884 4128398144]@SMS[23 Dec 22:27:01] No default mapping file found.

[log_indexer 44884 4128398144]@SMS[23 Dec 22:27:01] The mapping configuration must be defined in case not all fields are exported!

Could you please give me a detail example for this case? Many THX!

[Dror_Aharony · Accepted solution]

he only needed to configure the fieldsMapping.xml properly in the targetConfiguration.xml.

like this:

<mappingConfiguration><MappingFile_Path-relative-to-exporter-folder></<mappingConfiguration>

# Example:

<mappingConfiguration>fieldsMapping.xml</<mappingConfiguration>

[Herschel_Liang]

Meanwhile, I found that it can not defind export filedmapping about "information" like screenshots and  logs&monitor(new version) is no information fileds, quite ddifferent from old version. Is it expected or can it come true?