Log Exporter LEEF Field Mappings

[Bob_Bent]

Please use this discussion as a guide to understand how Check Point syslog Log Exporter maps Check Point logs to the LEEF format. This discussion is based upon R80.30 and may change in future versions.

LEEF fields have their own names such as cat, devTime, url, etc. Check Point fields such as src and dst that already match a LEEF field name do not need to be mapped from a Check Point to a LEEF name so are not covered in this discussion.

Note: in this discussion we refer to the raw Check Point field value. Check Point may translate the raw field name to show a different display name in the user interface like Tracker in R77.30 or SmartConsole in R80.x. 

LEEF Event Components, IBM Knowledge Center

The Log Event Extended Format (LEEF) is a customized event format for IBM Security QRadar. The LEEF format consists of the following components.

LEEF Header Mapping

The LEEF header is a required field and is composed of a pipe delimited (|) set of values that identifies Check Point events to QRadar.

Check Point fields are added to the header as specified in assign_order in the file $EXPORTERDIR/conf/LeefFormatDefinition.xml:

• first (use first added value - default)
• last (use last added value)
• join (join between values)
• init (set value once to header formatted string on init and do not generate per every log)

LEEF Format Header Definition Examples (note: a space is added between the “|” delimiter to make it easier to see the values).

• LEEF:Version | Vendor | Product | Version | Event ID |
• LEEF: 2.0 | Check Point | Log Update | 1.0 | Check Point Log |

LEEF Version: assign_order is set to init

LEEF: 2.0

Vendor: the assign_order is set to init

Check Point

Product: the assign_order is set to first

This default is Log Update, but may also be the value from the fields; product or productname.

Version: the assign_order is set to init

1.0

Event ID, the assign_order is set to init

The default is Check Point Log, but may also be the value from the fields protection_name, appi_name, action.

Event Attributes

The event attributes identify the payload information of the event in a set of key value pairs that provide detailed information about the security event. Each event attribute is separated by the tab delimiter character.

Examples

• key=value<tab>key=value<tab>key=value<tab>key=value<tab>
• src=192.0.2.0     dst=172.50.123.1    sev=5 cat=accept    srcPort=81     dstPort=21    usrName=joe.black

Predefined LEEF Event Attributes, IBM Knowledge Center

The LEEF format contains a number of predefined name=value pairs event attributes, which allow QRadar to categorize and display the event. Log Exporter uses these keys when possible.

Custom Event Keys, IBM Knowledge Center

Vendors and partners have the option to define their own custom event keys and include them in the payload of the LEEF format.

Check Point events that do not fall into one of the pre-defined LEEF event attributes are non-normalized, which means they are not displayed by default on the Log Activity tab of QRadar. To view custom attributes and non-normalized events on the Log Activity tab of QRadar, you must create a custom event property. Non-normalized event data is still part of your LEEF event, is searchable in QRadar, and is viewable in the event payload.

Mapping of Check Point Fields to Pre-defined LEEF Event Attributes

The below is the Log Exporter LEEF Field Mapping from R80.30 from $EXPORTERDIR/conf/LeefFieldsMapping.xml where origName is the Check Point raw field name and dstName is the LEEF attribute sorted by the LEEF dstName field name.

Callback Functions

In the name column are example uses of the callback functions where the value is replace_value. This function swaps values based on a key:value chart. We use this to map the Check Point severity (and other fields) to known LEEF values.

origName	dstName	name	key	value	tableName
attack_information	attackInformation
attack_name	attackName
status	attackStatus
business_impact	businessImpact
action	cat
industry_reference	cve
resource	destinationDnsDomain				resource_table
time	devTime
client_dst	dst
ipv6_dst	dst
received_bytes	dstBytes
server_inbound_bytes	dstBytes
client_inbound_bytes	dstBytes
mac_destination_address	dstMAC
d_port	dstPort
destination_port	dstPort
xlatedst	dstPostNAT
xlatedport	dstPostNATPort
recipient-recipients	emailRecipient
from	emailSender
sender	emailSender
subject	emailSubject
supress_logs	eventsCoalesced
extracted_files	extractedFiles
extracted_file_types	extractedFileTypes
extracted_hash	extractedHash
file_MD5	fileHash
file_id	fileID
file_id	fileId				file_table
file_name	filename
file_size	fileSize
filetype	fileType
file_name	fname				aggregated_file_table
file_name	fname				file_table
file_size	fsize				file_table
OU_group	identGrpName
source_machine_name	identHostName
malware_family	malware
malware_activity	malwareActivity
packet_capture	pcap
performance_impact	performanceImpact
phone_number	phoneNumber
policy_name	policy
policy	policyName
profile	profileName
protocol	proto
remediated_files	remediatedFiles
app_risk	sev	replace_value	default	0
app_risk	sev	replace_value	0	0
app_risk	sev	replace_value	1	2
app_risk	sev	replace_value	2	4
app_risk	sev	replace_value	3	6
app_risk	sev	replace_value	4	8
app_risk	sev	replace_value	5	10
severity	sev	replace_value	default	0
severity	sev	replace_value	0	0
severity	sev	replace_value	1	2
severity	sev	replace_value	2	5
severity	sev	replace_value	3	8
severity	sev	replace_value	4	10
app_risk	sev	replace_value	default	Unknown	match_table
app_risk	sev	replace_value	0	Unknown	match_table
app_risk	sev	replace_value	1	Low	match_table
app_risk	sev	replace_value	2	Low	match_table
app_risk	sev	replace_value	3	Medium	match_table
app_risk	sev	replace_value	4	High	match_table
app_risk	sev	replace_value	5	Very-High	match_table
app_risk	sev	replace_value	default	Unknown	primary_application
app_risk	sev	replace_value	0	Unknown	primary_application
app_risk	sev	replace_value	1	Low	primary_application
app_risk	sev	replace_value	2	Low	primary_application
app_risk	sev	replace_value	3	Medium	primary_application
app_risk	sev	replace_value	4	High	primary_application
app_risk	sev	replace_value	5	Very-High	primary_application
sha1	sha1
sha256	sha256
protection_name	signature
client_ip	src
ipv6_src	src
sent_bytes	srcBytes
server_outbound_bytes	srcBytes
client_outbound_bytes	srcBytes
imsi	srcMAC
mac_source_address	srcMAC
s_port	srcPort
xlatesrc	srcPostNAT
xlatesport	srcPostNATPort
proxy_source_ip	srcProxyIP
suspicious_events	suspiciousEvents
destination_dns_hostname	url
resource	url
orig_from	usrName
user	usrName

[Sven_Glock]

Hi Bob,

thanks for your nice post.

As I am facing exactly the problem you described.

Does this documentation also fit to R80.10?

If not what are the differences?

Thanks for your support.

Regards

Sven

[Bob_Bent]

Diffed an R80.10 LeefFieldsMapping.xml with the same file from R80.30. The major differences seem to be that the app_risk and severity callbacks numeric values in R80.10 have string values in R80.30, e.g. 0 replaced with a value of Low risk. Header field mapping may be handled differently. 

What problems are you running into?

[Lubomir]

Hello Bob,
we recently ran into a problem when forwarding logs to IBM QRadar. I know this is mainly IBM's issue, but according to step 4 of their troubleshooting guide (https://www.ibm.com/support/pages/node/876650), "product" needs to be mapped to "cat" (instead of action) to make it work with their parser. This workaround worked well with version R80.10, but one of our customers recently updated to R80.40 and since then we are unable to map product (or productname) to this field. It still works as part of header definition in LeefFormatDefinition.xml, so I believe it's still one of the available parameters, but it doesn't seem to work when processing LeefFieldsMapping.xml. Were there any changes that could affect this behavior? Is there any other field that contains product name information and could be used in LeefFieldsMapping.xml? I know that product is not listed in origName column in your table, but it worked anyway. I don't know why IBM ignores product name in the header, but changing mapping was way easier than writing own parser. Thank you in advance for any answer.

Best Regards,

Lubomir

[Yaakov_Ohayon]

Hi,

There are few known issues in the integration between log exporter and Qradar:

1. The Leef format, as log exporter forms, is not the exact format Qradar expects. the main issue is that CP field "product" is not mapped to "cat".

Solution: change LeefFormatDefinition.xml and LeefFieldsMapping.xml to adjust the header\mapping accordingly.

2. CP logs can have multiple blades reported in a single log, for instance, "Application Control" and "Content awareness". As we map the product field to "cat", Qradar fails to parse those logs and leaves them out of the structured data.

Solution: Log exporter will send only the first blade name in the product field.

3. In R80 and up, a layered rulebase was intruduced. Now logs are being reported with the match table, that details all the layers, rule names, rule numbers matched along the way. it causes logs to be exported with duplicate fields (with different values) as we "flatten" the table to single fields.

Solution: Log exporter will send only the first line from the match table.

All those solutions are currently in development and will be integrated into R80.X JHFs soon.

Thanks.

[Lubomir]

Hello Yaakov,

we had working workarounds for first two issues (on R80.10). The problem is, that the solution for the first issue is not working for us after upgrade to R80.40. After upgrade we have following mapping:

<?xml version="1.0" encoding="utf-8"?>
<fields>
<!-- Filter out fields -->
		<field><origName>flags</origName><exported>false</exported></field>
		<field><origName>__policy_id_tag</origName><exported>false</exported></field>
<!-- End of filter out -->
                <field><origName>product</origName><dstName>cat</dstName></field>
		<field><origName>time</origName><dstName>devTime</dstName></field>
		<field><origName>protocol</origName><dstName>proto</dstName></field>
...

but our logs still look like this (we restarted exporter):

LEEF:2.0|Check Point|VPN-1 & FireWall-1|1.0|Accept|devTime=1595317396	proto=HTTP	...

When we tried to map another field to cat, it worked (for example we mapped protocol and it showed cat=HTTP). But we were unable to map product. I thought you removed it because it wasn't used by default, but according to sk144192, it should be still available.

We solved second problem by adding following escape section(s), but if it will be done automatically in the future it would be great.

<char>
  <orig>Application Control URL Filtering</orig>
  <escaped>Application Control</escaped>
</char>	

Anyway, thank you for your response. In the end it looks like we use right solutions and according to CP documentation it should still work on R80.40. Now lets hope JHF will help.

Best Regards,

Lubomir

[Yaakov_Ohayon]

Hi,

The issue you describe is known to us and we already fixed it in R80.30 and R80.20 JHFs. 

This issue has no WA, please open a support ticket so we can provide the known fix, and I will make sure it will be integrated into R80.40 JHF also.

[Yaakov_Ohayon]

Hi,

Small update, the issue is fixed in R80.40 since take 38.

You can safely install and use it.

Kobi Ohayon

[Beja]

Hi Yaakov

Currently we are facing same problem with a QRADAR you said in point number 3. "duplicated fields due to layers.

Do you know how we could send only first line from match table?

Regards.

[Yaakov_Ohayon]

Hi @Beja ,

Currently you cannot, we are working to integrate a solution where you will get only the first line to all jumbo versions