- CheckMates
- :
- Products
- :
- Developers
- :
- API / CLI Discussion
- :
- Syslog parsers
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Are you a member of CheckMates?
×- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Syslog parsers
Hi,
There is a tool to parse syslog files and create some sort of interpreter so the syslog events can get imported into something that makes sense in the logging.
Did someone write up any code they are willing to share?
- Labels:
-
General
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Are you familiar with logstash ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No. I decied to go with SK55020 and work through it the hard way.
But it looks to me that the generated code could be optimized. I currently allready did get 41 patterns to match just UDP and TCP logging from iptables.
There is a bunch of sample codes in R80.10 but they can't parse iptables output yet.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It seems to me a manual code would be much more efficient.
As I get a rather complex output with all sort of inefficient forms if I use the Eventia Log parsing Editor.
Take for example a log entry like this:
Sep 15 13:36:04 kernel: DROP IN=eth0 OUT= MAC=e0:3f:49:ee:63:00:a2:de:48:00:01:03:08:00 SRC=4.3.2.1 DST=1.2.3.4 LEN=64 TOS=0x00 PREC=0x00 TTL=46 ID=34134 DF PROTO=TCP SPT=55170 DPT=63189 SEQ=1695161045 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (02040548010303000101080A0D1E83CA0000000004020000) \012
In Italics the distinguished text. In Bold the fileds I want to extract.
I can think of a relative clean regex sample. But the nicest code would allow both inbound and outbound packets.
I might do the trick with looping the code over both options. As ther is either a string match after IN= or there is one after OUT=
Anyone got any additional documentation? I can reverse engineer this if neede and in fact have done it in part allready.
But any additional documentation will be appreciated.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I started with a WRT interpreter. The file name is WRT.C
# (C) 2017 Hugo van der Kooij
# STATUS : Experimental!
# Parsing Syslog from WRT
# Tested with:
# AsusWRT(
:cmd_name (group_try)
:mode (try_until_success)
: (
:command (# Sep 19 12:31:24 kernel: DROP IN=eth0 OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00 SRC=192.0.2.1 DST=192.0.2.2 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=34252 PROTO=TCP SPT=10000 DPT=2001 SEQ=846184469 ACK=0 WINDOW=1024 RES=0x00 SYN URGP=0
:cmd_name (try)
:parse_from (last_position)
:regexp ("kernel: (DROP) (IN)=([a-zA-Z]+[0-9a-zA-Z_-]*) ")
:add_field (
:type (index)
:field_name (action)
:field_type (action)
:field_index (1)
:dict_name (WRT_action)
)
:add_field (
:type (index)
:field_name (ifdir)
:field_type (ifdir)
:field_index (2)
:dict_name (WRT_ifdir)
)
:add_field (
:type (index)
:field_name (ifname)
:field_type (ifname)
:field_index (3)
)
:on_success (
:command (try)
:parse_from (last_postion)
:regexp ("SRC=([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+) DST=([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)")
:add_field (
:type (index)
:field_name (src)
:field_type (ipaddr)
:field_index (1)
)
:add_field (
:type (index)
:field_name (dst)
:field_type (ipaddr)
:field_index (2)
):on_success (
:command (try)
:parse_from (last_position)
:regexp ("PROTO=(UDP|TCP) SPT=([0-9]+) DPT=([0-9]+)")
:add_field (
:type (index)
:field_name (proto)
:field_type (protocol)
:field_index (1)
:dict_name (WRT_protocol)
)
:add_field (
:type (index)
:field_name (s_port)
:field_type (port)
:field_index (2)
)
:add_field (
:type (index)
:field_name (service)
:field_type (port)
:field_index (3)
)
)
)
)
)
)
Along with it is the dictionary file WRT.ini
[WRT_action]
DROP = drop[WRT_ifdir]
IN = 0
OUT = 1[WRT_protocol]
TCP = 6
UDP = 17
Install with the command:
addParsingFile -p WRT.C -d WRT.ini
This should understand IPtables output as well as that is what WRT is in fact sending.
My first tests on R80.10 were .... not too positive. No errors but not much additional data in the logs files either.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
just for info, because it looks you spent some time with this, I have few cases opened for parsing syslog/win logs and it looks it does not work generally since there is some complex issue will be fixed in few months they said
