Installing Event Policy EXTREMELY slow on SmartEvent Smart-1 600-S appliance

[Joe_Kanaszka]

Hey guys.  Small Network.  (2) 9100 Security Gateways - Cluster XL.  SMS is a VM.  Smart Event is logging server and running on a Smart-1 600-S appliance.

SGs are running R81.20 Take 118

SMS and Smart Event are running R81.20 Take 119

Whenever I need to change my Event Policy and install it - I might as well go grab a cup of coffee because it literally takes a few minutes.

Why is this?  What could be causing this slowness?

[the_rock]

Hey brother,

You are strictly referring to applying SE policy from smart event console itself, right?

[Joe_Kanaszka]

Hey Andy - yes sir.

[the_rock]

Man, no need to call me sir, I just turned 46, haha lol.

Anywho, mind sharing cpu/memory usage from top command? Also, how much RAM on this smart-1? Since its all in one box (mgmt + se), that would usually warrant for at least 32 GB of ram.

[Joe_Kanaszka]

Thanks again Andy.  My SMS is a separate VM.  Looks like this Smart Event box has 16GB Mem / 6 CPUs

Installing event policy does spike two of the 6 CPUs

[the_rock]

If its separate VM, that sounds like good amount of resources, specially 6 CPUs and 16 GB of ram. Mine is 4 CPUs and 8 GB of ram in the lab and its fine. Did you try evrestart or quick reboot to see if it helps?

[the_rock]

Btw, just to be 100% sure, I assume this is policy you meant?

 

[Joe_Kanaszka]

Yes

[the_rock]

Hey Joe,

Just a thought, but since really doing evstop; evstart would not cause any issues, I would really give that a go and see if its any faster. If it is, just monitor for 24 hours and then compare.

[Don_Paterson]

In the lower right hand side of the SmartEvent gui there is a status link. 

How does the status and sizing look?

Do you have any more 'complex' event definitions?

I would also check the output of top or htop and/or cpview. You can do that before and during to see resources utilisation. 

The two cpse.. processed are the SME server and SME Correlation Unit

Cpview can show log receive rates, in the Advanced > Management area (if I remember that path correctly)  

The Correlation Unit may be busy analysing a lot of log volume while also doing the policy installation, and that'swhy it is slow. . 

If you rebooted already you could still try evstop and evstart.

You could try the command CPLogInvestigator to get more visibility of log volumes. 

I've always found SME policy installations took minutes.

[Joe_Kanaszka]

Good morning and thank you Don!.  Status looks good - all green.  Scale is according to recommendation.   Actually i just installed Event Policy again and it did not take that long...maybe two minutes.

In my case, the SME is both the Correlation unit and the log server.  Any issue there?

[Don_Paterson]

You're welcome. 

That sounds good, and maybe even normal for SmartEvent. 

No issues with the setup. It's a good deployment option and having log and event management separate from the SMS is ideal. 

The CU does all the heavy lifting, trawling through incoming logs and keeping Event Candidates in memory until a threshold is reached and then it notifies the event server of the new event. That's for events/incidents  that depend on multiple logs, and dependant on the event policy. 

A single IPS log is all that is needed for a new event of that type. 

[the_rock]

But even 2 mins, seems a bit long, should not take longer than a minute, in my experience.

[Don_Paterson]

You are right - It may be, but will depend on what's going on in the box.

Time to talk tools... Maybe recommending hcp -r all is a good idea, to see what Health Check Point reports.

I would also run Doctor Log

/opt/CPrt-R82/scripts/doctor-log.sh

I just installed policy in my lab, which is one VM (R82 T44), running both SMS and SME and has 8 cores (i9 - 14900 - far more powerful than a Smart-1 600S) and 16GB RAM, and fast NVMe storage.

I enabled about 20 new event definitions before I installed and it took about 90 seconds.

Then I enabled 10 or so more and installed again and that took 70 seconds.

Since FWM is involved and tends to use only one core that could be a bottleneck.

FWM is effectively single-threaded for policy compilation; only one core is used for the heavy work.

@Timothy_Hall  Will surely have a few words to say about FWM being involved 😉 

[the_rock]

I thought maybe has to do with java process consuming high memory, but since we dont have the facts about it yet, hence I asked for output of top command.

[Timothy_Hall]

As @Don_Paterson said, process fwm is heavily involved with installing the event policy and tuning SmartEvent.  This is why the legacy SmartEvent Policy Editor GUI is separate from the main SmartConsole GUI: only fwm can handle it.  This is also why some other legacy functions remain in the old-school SmartDashboard GUI: fwm dependency.

Unfortunately, fwm is very old and single-threaded, so the speed of operations like installing the event policy is largely constrained by the processing power of a single core.  I wasn't able to determine which processor is used in the 600-S, but its replacement, the 700, uses "Intel(R) Celeron(R) CPU J4105 @ 1.50GHz," so I would expect the 600-S to use something similar.  As you can see, they do not put high-power processors such as Xeon in their lower-end Smart-1 models.

Thankfully with each new release, responsibilities are taken away from fwm and added into cpm or other new processes.  Quite a bit of this happened for R82, and full policy installation times saw definite improvement as fwm was moved further out of the loop.  Haven't looked at R82.10 yet, but I would expect this process of getting away from fwm to continue.

[the_rock]

I personally found fwm is totally fine in R82, never had any problems in the lab or with any clients.