Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Juan_Concepcion
Advisor

Customized 'fwm logexport'

The following is not a question but more of a possible solution for folks who are looking to perform exports of their log files and wish only to do so on specific fields. 

 I’ve provided both a manual and scripted method to extract this information from the raw logs.  The “fields” are defined in $FWDIR/log/logexport_default.C – these samples are based on source/destination/service/protocol fields which were the only relevant information I needed.  The export of a months worth of logs (4 log files on weekday and 3 log files on weekend) took roughly around 5 hours to extract the information from 112 log files.  Please be aware this is resource intensive for the manager if you are doing a large amount of log files but a single 2 gig log files takes 3 to 4 minutes to be exported.

The following outlines how to perform a customized log export with only specific fields in the export.

 

In both the manual and automated process you will want to create the $FWDIR/conf/logexport.ini with the following information, where "field" is the actual field name taken from the logexport_default.C file:

 

[Fields_Info]

included_fields = field_1_name,field_2_name,field_3_name,field_4_name

excluded_fields = field_8_name,field_9_name,field_10_name,field_11_name

 

The only required line is the ‘included_fields’ - anything not in that list will be excluded by default.  To explicitly list fields to exclude use the second line ‘excluded_fields’

 

Example:

 

[Fields_Info]

included_fields = date,orig,action,src,dst,service,proto

 

    Note: there is a slight bug where the first field is not populated into the output - in my example I put in to include date, origin of log file, action, source, destination, service and protocol, however, in my output it will only show the last 6 entries, the date field will show up as 

“,,” with no data - the number sequence is always included and not removable.

 

Manually run the ‘fwm logexport' against a specific file:

 

fwm logexport -d ,-n -p -m raw -i <log_file_name>.log -o /path/to/output/filename.txt 

Starting... There are 6034707 log records in the file

File logexport.ini was opened successfully

 

Result will a comma delimited (this is specified by "-d ,")  filename.txt with only the exported fields that were chosen

 

Script to automatically generate a csv file from the indicated log files based on date (assumption that the logexport.ini is already in place):

 

# Set Shell

# !/bin/bash

# Source Check Point variables file

source /opt/CPshrd-R80.20/tmp/.CPprofile.sh

 

# Specify log files to be read in

FILE="$FWDIR/log/<date>*.log”m #Example would be FILE=“$FWDIR/log/2019-04-01*.log to export all log files for April 1st

 

#Begin loop to export log files -n -p disable resolution of hostname and service ports

for f in $FILE ; do

        echo "Converting File $FILE"

        `time fwm logexport -n -p -m raw -i "$f" -o "$f.csv"`

done

 

$FWDIR/conf/logexport.ini  #used to specify what fields to export

[Fields_Info]

included_fields = date,orig,action,src,dst,service,proto

 

date=include timestamp

orig=match gateway

action=include action for log entry

src=include source

dst=include destionation

service=include service

proto=include protocol type for the service

 

To cleanup the exported log file in vi for any non-relevant information:

 

:g/<search_pattern>/d

 

g=Global Search of entire file

<search_pattern>=match criteria for the line(s) you’d like to have removed

d=delete

 

Example:

 

Clear out all action of “drop” in the file so you only have “accept”:  :g/drop/d

 

Remove all entries related to a specific origin (firewall):     :g/<ip_of_gateway/d

 

 

 All of the log fields are detailed in the $FWDIR/conf/logexport_default.C.

2 Replies
Beja
Contributor

Very useful. Thank you so much.

In this sample: fwm logexport -d ,-n -p -m raw -i <log_file_name>.log -o /path/to/output/filename.txt 

Just a typo, maybe. You have to add a space between ',' and '-n'.

and....

logexport.ini file you have to delete space later '='.

here:

"included_fields =date,time,orig,action,src,dst,service,proto"

 

 

🙂

0 Kudos
G_W_Albrecht
Champion
Champion

Why not point out sk118519: "fwm logexport" command where this is explained in detail ? Your script is a nice addition...

0 Kudos