Log_indexer Performance Issues because of core limitation

[Us4r]

Hello,

we have a big logging issue because we get logfiles of about 100 checkpoint appliances on our management server.

During work hours we have a delay of about 1h before we can see the indexed log files.

As I verified I think the issue occurs becaus log_indexer service does only use 2 CPU Cores and both cores are completly busy!

We are using R80.30 and I can see on the management node that the configuration of the log indexer service is placed at /opt/CPrt-R80.30/log_indexer

Any chance to change the number of cores there?

Thanks and Regards

Florian

[Magnus-Holmberg]

Are you running dedicated log servers or everything on the same box?

[Us4r]

Yes everything is running on the same box. It's one openserver with following hardware specs: Dell R640, 2xXEON Gold 6244 3,6Ghz CPUs, 128GB RAM, SSD Drives

[Wolfgang]

@Us4r 

with such an environment I suggest to use separate logserver and for performance reason you can use more then one and let the gateways send there logs to different logserver. Meaning 50 gateways are send to logserver A and the other gateways send to logserver B.

And for your hardware, have a look at the Power Management settings in the BIOS. They should be set to "maximum Performance" following DELLs knowledgebase

Best Practices in Power Management 

Wolfgang

[HeikoAnkenbrand · Accepted solution]

Hi @Us4r,

You can adjust the following parameters in the file "log_indexer_settings.conf", then you should have more cores available. Afterwards a reboot of the SMS is necessary.

I don't know if there is Check Point support for this. But I would only change this parameter via a TAC case.

PS:
But I think you should change the design and maybe use another log server.

[Magnus-Holmberg]

@HeikoAnkenbrand
This setting is per CMA right? because in MDS atleast i see multiple threads of logindexer, 10+

[Wolfgang]

@Magnus-Holmberg 

yes, these setting is per CMA.

But as we wrote switching to a separate logserver will be better for all operations (like policy install, working in SmartConsole, log search  etc.)

Wolfgang

[Magnus-Holmberg]

for sure, we uses 3 dedicated log servers to some MDS 🙂

[HeikoAnkenbrand]

Hi @Magnus-Holmberg, @Wolfgang,

In my test environment the file is only visible once on the MDS.

Maybe, you can also set the parameter in the customer file. I'm not sure about this!
In this case you should use the following file: log_indexer_custom_settings.conf

You should check it out.

But like I said, I wouldn't do it without TAC.

[Us4r]

Hi @HeikoAnkenbrand
we also opened a TAC case for the Issue and an engeneer tried to change the core settings in the mentioned file but this hadn't any affect on the number of started processes.

It stays on two processes. The engeneer also tried to change following settings in the file /opt/CPSmartLog-R80.30/conf/smartlog_settings.conf:
:num_pre_index_threads (4)
:num_index_threads (8)

[Dror_Aharony]

Hi All,

the SmartLog settings are no longer relevant on R80.x, only the Indexer settings in:

$INDEXERDIR/...

-What is your Mgmt server's HW specs ?

-Strong Mgmt who is the actual LS too? Best separate to a dedicated Log-Server as suggested.

-What is your avg log-rate/sec?

   # Check by: cpstat mg -f log_server & cpstat mg -f indexer

-Also send: cat $MDS_FWDIR/conf/serverSettings.props

or best collect full logs by running:

SmartEventCollectLogs

[Us4r]

Hello @Dror_Aharony 

- our hardware specs: Dell R640, 2xXEON Gold 6244 3,6Ghz CPUs, 128GB RAM, SSD Drives

- We have the blades "Network Policy Management", "Logging & Status" and "Provisioning" on the same HW. So yes it's the same server.

-  Regarding the Log rate check the attached output "log-rate.JPG"

Regards

Florian

[Dror_Aharony]

Hi Florian,

the 4,700 logs/sec rate is from a specific CMA/DLS?
what is the incoming log-rate from other CMAs? the total rate to the MDS?

Is this the main & the problematic one, that keeps getting backlogged-up up to 1 hour of delay.

This is complex, it won't probably be fixed by trying to adjust the indexer threads.

It may, but it might require some tweaking and  testing & there's no guarantees for improvement.
Your server is currently configured with eight (8) Pre-indexing & Index threads for each CMA.

you could try to reduce it to 4 (and then even 2) on the MDS level $INDEXERDIR via the .conf file, as was written above.

assuming you have many CMAs, which may interefere with each other.  

but it's a guess. it may not help.

Your logs are probably either extremely heavy (high ratio of Threat logs) and/or some other underlying issue that requires a deeper TAC investigation, cause there's no apparent reason that your strong server wouldn't hold a ~5K logs/sec load for online log-indexing without any delay/backlog.

Run: SmartEventCollectLogs & send me the output for another attempt to help.

Also: with such an environment I can suggest adding a separate log-server/MLM.

Best open a TAC ticket.

[Us4r]

Hello,

we found a solution for that now 🙂

After Upgrade of the Management - Server to Take 214 everything looks fine and is working as expected.