Is there anything to tweak to make searches in the logs faster?

[George_Ellis]

Background:  We have 2 6000XL boxes as MLMs.  Each has >45% free space (33TB total).  We have a 5150 MDM.  We have 13 domains.  The last time I checked, we log >150 GB a day. Edit - Version is 81.10 JHF 95.

What I do:  I get to do the PCI/Swift compliance audits and remediations, as well as rule tuning.  This frequently includes rules that are too broad and broad rules often shadow rules that were designed for the workflow.  Example:  Src:10. Dst:Any Port:TCP/1024-5000. (MS AD from 2012 srvr iirc)

Method on isolating the traffic:  Search: <30 day> <rule uid> 

From that point, it then is process of elimination.  "<30 day> <rule uid> not port:1088"  "<30 day> <rule uid> not port:(1088 or 3200)"
Slowly identifying the business traffic that is sneaking through the rule port by port or src/dst by src/dst.  And then we may switch to "All Time" (those once a month or qtr jobs...)

It takes more than an hour for an All Time search.  Any thoughts on making it faster?

PS - Search from the MDM or the specific domain on the MLM run about the same.

PPS - Per sk144192, rule_uid is indexed.

[PhoneBoy]

If you’re not running at least R81.10, upgrade.
Check to see if the drives are using xfs (instead of ext4).
You might also see better performance by doing a fresh install from R81.20 since it uses a fixed fdisk to ensure disk partitions fall on cylinder boundaries.

[George_Ellis]

81.10 #95 with xfs /var/log.  It would have been 81.20, but the upgrade got an unknown error during the weekend.  Sigh

[the_rock]

What was the hiccup with R81.20 upgrade? Mind sharing?

Andy

[George_Ellis]

No idea yet.  Almost completed and then said "Unknown Error" and reverted.

[the_rock]

Does not get more generic than that error, for sure. 

[the_rock]

I always found that in R81.20, log search works way better than before.

Andy

[Amir_Senn]

Hi,

The advise here are how to improve query time but from my POV I think there might be some better ways to accomplish what you're trying to do.

I would suggest a few things to consider:

1) Hit Count for rules.

2) Try to export all the logs (or daily and add them up if you have a lot of traffic). It's easier to manipulate CSV file in Excel. (See 1.png)

3) If you have SmartView, with a little handling you can create some tables that show you rules + information (See 2.png). Handling - SmartView only index non-connection, see there's a need to add session type for every rule.

[George_Ellis]

The Excel export can have some benefit.  But then I then had to submit a case because search field rule_uid: is not working correctly.

[the_rock]

That sort of search query works fine for me in R81.20 lab.

Andy

[Amir_Senn]

Which version are you using and what problem are you experiencing?

Perhaps you can use rule name/rule number to some extent instead

[George_Ellis]

Duh, left that off.  81.10 #95.  Rule number will have too many collisions, then force origin: into it too.  I just created a SR because I discovered that rule_uid: is not working either in SmartConsole or SmartView.  I was trying to use rule_uid: in SmartView to help with the csv export.

Also, if I remember correctly, rule number will always return that result, so it the specific rule gets displaced by an add/move/remove above it, your result will vary.

[Amir_Senn]

True, a rule number could be changed but you can cross reference it with rule name if all your rules have names.

IDK what exactly are the issues with rule_uid, I have R81.10 GA with no JHF and it looks like it works.

I used definitions in 1.PNG and got rule_uid in csv (2.PNG). Try to replicate?

[the_rock]

You are 100% right, it also works in R81.10 as well.

Andy