Processing Logs Exported via 'fwm logexport -s'

[Bob_Zimmerman]

I recently needed to export and process raw log data outside of SmartLog. Thought I would share the most interesting bit in case anybody else needs to do this.

By default, 'fwm logexport' separates fields in the output with semicolons. Unfortunately, semicolons can occur within fields. Lots of simpler tools like 'cut' and 'awk' don't deal with quote delimited fields, which can hurt the ability to extract specific columns from the data.

To deal with this issue, 'fwm logexport' has the -s switch, which causes it to use the non-printing character with hex code 0xFF to separate fields. This gets a little weird to deal with, since how do you type a non-printing character to tell your tools to use it as the field separator?

printf to the rescue! It has the ability to emit a character from its octal code:

awk -v FS=$(printf "\377") '{print $2,$3}' <some file>.ffsv

Hex 0xFF corresponds to octal 377. That will print columns 2 and 3 (usually the date and time) from a log exported via 'fwm logexport -s'.

ASCII has a few non-printing characters specifically for separating stuff: the file separator (0x1C, octal 34), the group separator (0x1D, octal 35), the record separator (0x1E, octal 36), and the unit separator (0x1F, octal 37).

[the_rock]

Thanks for sharing @Bob_Zimmerman ...super interesting!

[PhoneBoy]

As long as I've been doing this, I never knew about the -s flag in fwm logexport.
Very clever trick!

[the_rock]

I seen it before, but never used it 🙂

[Expert@CP-MGMT:0]# fwm logexport -h

Usage:
fwm logexport [-d delimiter |-s] [-t table_delimiter] [-i filename] [-o filename] [-f|-e] [-x start_pos] [-y end_pos] [-z] [-n] [-p] [-a] [-u unification_scheme_file] [-m (initial|semi|raw)]
Where:
-d - Set the output delimiter. Default is ';'.
-s - Set the delimiter to be ASCII character #255.
-t - Set the output delimiter inside table field (table field would look like: (ROWx:COL0,ROWx:COL1,ROWx:COL2) etc. Default is ','.
-i - Input log file name. Default is the active log file, fw.log.
-o - Output file name. Default is printing to the screen.
-f - Only in case of active log file - Upon reaching end of file, wait for new records and export them as well.
-e - Same as -f flag, only start at end of file.
-x - Start exporting at the specified position.
-y - End exporting at the specified position.
-z - Continue exporting the next records, in case of an error. Default is to stop exporting.
-n - No IP resolving. Default is to resolve all IPs.
-p - No port resolving. Default is to resolve all ports.
-a - Export account records only. Default is export all records.
-z - Exit in case of wrong field expected value.
-u - Unification scheme file name. Default is log_unification_scheme.C.
-m - Unification mode: initial-order, semi-unified, or raw. Default is 'initial'.

[Expert@CP-MGMT:0]#

[Bob_Zimmerman]

I've been using the flag for a while, but I keep forgetting and rediscovering how to carve up the data once it's exported. 0xFF works with awk, but breaks some other tools. You can use the other non-printing separator characters with the -d switch. They have many of the same issues, but other tools like cut and tr handle them better. For the unit separator (meant to separate fields in tabular records), you have to do something like this:

[Expert@DallasSA]# fwm logexport -d "$(printf '\37')" -z -n -p -i fw.log >exported.ussv
Starting... There are 757 log records in the file

[Expert@DallasSA]# head -n 1 exported.ussv | cut -d "$(printf '\37')" -f 2
date

[Expert@DallasSA]# head -n 1 exported.ussv | tr "$(printf '\37')" ":" 
num:date:time:orig:type:action:alert:i/f_name:i/f_dir:product:LogId:ContextNum:origin_id:ContentVersion:hll_key:SequenceNum:log_sys_message:ProductFamily:description:status:version:comment:update_service:reason:Severity:failure_impact:TCP packet out of state:tcp_flags:src:dst:proto:service:s_port