Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Hugo_vd_Kooij
Advisor

Syslog parsers

Hi,

There is a tool to parse syslog files and create some sort of interpreter so the syslog events can get imported into something that makes sense in the logging.

Did someone write up any code they are willing to share?

<< We make miracles happen while you wait. The impossible jobs take just a wee bit longer. >>
0 Kudos
6 Replies
Irek_Romaniuk
Participant

Are you familiar with logstash ?

0 Kudos
Hugo_vd_Kooij
Advisor

No. I decied to go with SK55020 and work through it the hard way.

But it looks to me that the generated code could be optimized. I currently allready did get 41 patterns to match just UDP and TCP logging from iptables.

There is a bunch of sample codes in R80.10 but they can't parse iptables output yet.

<< We make miracles happen while you wait. The impossible jobs take just a wee bit longer. >>
0 Kudos
Hugo_vd_Kooij
Advisor

It seems to me a manual code would be much more efficient.

As I get a rather complex output with all sort of inefficient forms if I use the Eventia Log parsing Editor.

Take for example a log entry like this:

Sep 15 13:36:04 kernel: DROP IN=eth0 OUT= MAC=e0:3f:49:ee:63:00:a2:de:48:00:01:03:08:00 SRC=4.3.2.1 DST=1.2.3.4 LEN=64 TOS=0x00 PREC=0x00 TTL=46 ID=34134 DF PROTO=TCP SPT=55170 DPT=63189 SEQ=1695161045 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (02040548010303000101080A0D1E83CA0000000004020000) \012

In Italics the distinguished text. In Bold the fileds I want to extract.

I can think of a relative clean regex sample. But the nicest code would allow both inbound and outbound packets.

I might do the trick with looping the code over both options. As ther is either a string match after IN= or there is one after OUT=

Anyone got any additional documentation? I can reverse engineer this if neede and in fact have done it in part allready.

But any additional documentation will be appreciated.

<< We make miracles happen while you wait. The impossible jobs take just a wee bit longer. >>
0 Kudos
Hugo_vd_Kooij
Advisor

I started with a WRT interpreter. The file name is WRT.C

# (C) 2017 Hugo van der Kooij
# STATUS : Experimental!
# Parsing Syslog from WRT
# Tested with:
# AsusWRT

(
:cmd_name (group_try)
:mode (try_until_success)
: (
:command (# Sep 19 12:31:24 kernel: DROP IN=eth0 OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00 SRC=192.0.2.1 DST=192.0.2.2 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=34252 PROTO=TCP SPT=10000 DPT=2001 SEQ=846184469 ACK=0 WINDOW=1024 RES=0x00 SYN URGP=0
:cmd_name (try)
:parse_from (last_position)
:regexp ("kernel: (DROP) (IN)=([a-zA-Z]+[0-9a-zA-Z_-]*) ")
:add_field (
:type (index)
:field_name (action)
:field_type (action)
:field_index (1)
:dict_name (WRT_action)
)
:add_field (
:type (index)
:field_name (ifdir)
:field_type (ifdir)
:field_index (2)
:dict_name (WRT_ifdir)
)
:add_field (
:type (index)
:field_name (ifname)
:field_type (ifname)
:field_index (3)
)
:on_success (
:command (try)
:parse_from (last_postion)
:regexp ("SRC=([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+) DST=([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)")
:add_field (
:type (index)
:field_name (src)
:field_type (ipaddr)
:field_index (1)
)
:add_field (
:type (index)
:field_name (dst)
:field_type (ipaddr)
:field_index (2)
)

:on_success (
:command (try)
:parse_from (last_position)
:regexp ("PROTO=(UDP|TCP) SPT=([0-9]+) DPT=([0-9]+)")
:add_field (
:type (index)
:field_name (proto)
:field_type (protocol)
:field_index (1)
:dict_name (WRT_protocol)
)
:add_field (
:type (index)
:field_name (s_port)
:field_type (port)
:field_index (2)
)
:add_field (
:type (index)
:field_name (service)
:field_type (port)
:field_index (3)
)
)
)
)
)
)

Along with it is the dictionary file WRT.ini

[WRT_action]
DROP = drop

[WRT_ifdir]
IN = 0
OUT = 1

[WRT_protocol]
TCP = 6
UDP = 17

Install with the command:

 addParsingFile -p WRT.C -d WRT.ini

This should understand IPtables output as well as that is what WRT is in fact sending.

My first tests on R80.10 were .... not too positive. No errors but not much additional data in the logs files either.

<< We make miracles happen while you wait. The impossible jobs take just a wee bit longer. >>
0 Kudos
Hugo_vd_Kooij
Advisor

Please find the files attached now. As the layout didn't make it.

<< We make miracles happen while you wait. The impossible jobs take just a wee bit longer. >>
0 Kudos
Ivo_Hrbacek
Contributor
Contributor

just for info, because it looks you spent some time with this, I have few cases opened for parsing syslog/win logs and it looks it does not work generally since there is some complex issue  will be fixed in few months they said Smiley Happy

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events