The following is not a question but more of a possible solution for folks who are looking to perform exports of their log files and wish only to do so on specific fields.
I’ve provided both a manual and scripted method to extract this information from the raw logs. The “fields” are defined in $FWDIR/log/logexport_default.C – these samples are based on source/destination/service/protocol fields which were the only relevant information I needed. The export of a months worth of logs (4 log files on weekday and 3 log files on weekend) took roughly around 5 hours to extract the information from 112 log files. Please be aware this is resource intensive for the manager if you are doing a large amount of log files but a single 2 gig log files takes 3 to 4 minutes to be exported.
The following outlines how to perform a customized log export with only specific fields in the export.
In both the manual and automated process you will want to create the $FWDIR/conf/logexport.ini with the following information, where "field" is the actual field name taken from the logexport_default.C file:
[Fields_Info]
included_fields = field_1_name,field_2_name,field_3_name,field_4_name
excluded_fields = field_8_name,field_9_name,field_10_name,field_11_name
The only required line is the ‘included_fields’ - anything not in that list will be excluded by default. To explicitly list fields to exclude use the second line ‘excluded_fields’
Example:
[Fields_Info]
included_fields = date,orig,action,src,dst,service,proto
Note: there is a slight bug where the first field is not populated into the output - in my example I put in to include date, origin of log file, action, source, destination, service and protocol, however, in my output it will only show the last 6 entries, the date field will show up as
“,,” with no data - the number sequence is always included and not removable.
Manually run the ‘fwm logexport' against a specific file:
fwm logexport -d ,-n -p -m raw -i <log_file_name>.log -o /path/to/output/filename.txt
Starting... There are 6034707 log records in the file
File logexport.ini was opened successfully
Result will a comma delimited (this is specified by "-d ,") filename.txt with only the exported fields that were chosen
Script to automatically generate a csv file from the indicated log files based on date (assumption that the logexport.ini is already in place):
# Set Shell
# !/bin/bash
# Source Check Point variables file
source /opt/CPshrd-R80.20/tmp/.CPprofile.sh
# Specify log files to be read in
FILE="$FWDIR/log/<date>*.log”m #Example would be FILE=“$FWDIR/log/2019-04-01*.log to export all log files for April 1st
#Begin loop to export log files -n -p disable resolution of hostname and service ports
for f in $FILE ; do
echo "Converting File $FILE"
`time fwm logexport -n -p -m raw -i "$f" -o "$f.csv"`
done
$FWDIR/conf/logexport.ini #used to specify what fields to export
[Fields_Info]
included_fields = date,orig,action,src,dst,service,proto
date=include timestamp
orig=match gateway
action=include action for log entry
src=include source
dst=include destionation
service=include service
proto=include protocol type for the service
To cleanup the exported log file in vi for any non-relevant information:
:g/<search_pattern>/d
g=Global Search of entire file
<search_pattern>=match criteria for the line(s) you’d like to have removed
d=delete
Example:
Clear out all action of “drop” in the file so you only have “accept”: :g/drop/d
Remove all entries related to a specific origin (firewall): :g/<ip_of_gateway/d
All of the log fields are detailed in the $FWDIR/conf/logexport_default.C.
