topic Re: Syslog parsers in API / CLI Discussion

Syslog parsers

Hugo_vd_Kooij — Tue, 12 Sep 2017 08:25:47 GMT

Hi,

There is a tool to parse syslog files and create some sort of interpreter so the syslog events can get imported into something that makes sense in the logging.

Did someone write up any code they are willing to share?

Re: Syslog parsers

Irek_Romaniuk — Fri, 15 Sep 2017 22:18:56 GMT

Are you familiar with logstash ?

Re: Syslog parsers

Hugo_vd_Kooij — Mon, 18 Sep 2017 12:36:57 GMT

No. I decied to go with SK55020 and work through it the hard way.

But it looks to me that the generated code could be optimized. I currently allready did get 41 patterns to match just UDP and TCP logging from iptables.

There is a bunch of sample codes in R80.10 but they can't parse iptables output yet.

Re: Syslog parsers

Hugo_vd_Kooij — Mon, 18 Sep 2017 13:24:24 GMT

It seems to me a manual code would be much more efficient.

As I get a rather complex output with all sort of inefficient forms if I use the Eventia Log parsing Editor.

Take for example a log entry like this:

Sep 15 13:36:04 kernel: DROP IN=eth0 OUT= MAC=e0:3f:49:ee:63:00:a2:de:48:00:01:03:08:00 SRC=4.3.2.1 DST=1.2.3.4 LEN=64 TOS=0x00 PREC=0x00 TTL=46 ID=34134 DF PROTO=TCP SPT=55170 DPT=63189 SEQ=1695161045 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (02040548010303000101080A0D1E83CA0000000004020000) \012

In Italics the distinguished text. In Bold the fileds I want to extract.

I can think of a relative clean regex sample. But the nicest code would allow both inbound and outbound packets.

I might do the trick with looping the code over both options. As ther is either a string match after IN= or there is one after OUT=

Anyone got any additional documentation? I can reverse engineer this if neede and in fact have done it in part allready.

But any additional documentation will be appreciated.

Re: Syslog parsers

Hugo_vd_Kooij — Tue, 19 Sep 2017 12:07:20 GMT

I started with a WRT interpreter. The file name is WRT.C

# (C) 2017 Hugo van der Kooij
# STATUS : Experimental!
# Parsing Syslog from WRT
# Tested with:
# AsusWRT
(
:cmd_name (group_try)
:mode (try_until_success)
: (
:command (# Sep 19 12:31:24 kernel: DROP IN=eth0 OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00 SRC=192.0.2.1 DST=192.0.2.2 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=34252 PROTO=TCP SPT=10000 DPT=2001 SEQ=846184469 ACK=0 WINDOW=1024 RES=0x00 SYN URGP=0
:cmd_name (try)
:parse_from (last_position)
:regexp ("kernel: (DROP) (IN)=([a-zA-Z]+[0-9a-zA-Z_-]*) ")
:add_field (
:type (index)
:field_name (action)
:field_type (action)
:field_index (1)
:dict_name (WRT_action)
)
:add_field (
:type (index)
:field_name (ifdir)
:field_type (ifdir)
:field_index (2)
:dict_name (WRT_ifdir)
)
:add_field (
:type (index)
:field_name (ifname)
:field_type (ifname)
:field_index (3)
)
:on_success (
:command (try)
:parse_from (last_postion)
:regexp ("SRC=([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+) DST=([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)")
:add_field (
:type (index)
:field_name (src)
:field_type (ipaddr)
:field_index (1)
)
:add_field (
:type (index)
:field_name (dst)
:field_type (ipaddr)
:field_index (2)
)
:on_success (
:command (try)
:parse_from (last_position)
:regexp ("PROTO=(UDP|TCP) SPT=([0-9]+) DPT=([0-9]+)")
:add_field (
:type (index)
:field_name (proto)
:field_type (protocol)
:field_index (1)
:dict_name (WRT_protocol)
)
:add_field (
:type (index)
:field_name (s_port)
:field_type (port)
:field_index (2)
)
:add_field (
:type (index)
:field_name (service)
:field_type (port)
:field_index (3)
)
)
)
)
)
)

Along with it is the dictionary file WRT.ini

[WRT_action]
DROP = drop
[WRT_ifdir]
IN = 0
OUT = 1
[WRT_protocol]
TCP = 6
UDP = 17

Install with the command:

addParsingFile -p WRT.C -d WRT.ini

This should understand IPtables output as well as that is what WRT is in fact sending.

My first tests on R80.10 were .... not too positive. No errors but not much additional data in the logs files either.

Re: Syslog parsers

Hugo_vd_Kooij — Wed, 20 Sep 2017 12:08:06 GMT

Please find the files attached now. As the layout didn't make it.

Re: Syslog parsers

Ivo_Hrbacek — Thu, 16 Nov 2017 07:47:05 GMT

just for info, because it looks you spent some time with this, I have few cases opened for parsing syslog/win logs and it looks it does not work generally since there is some complex issue will be fixed in few months they said