increase vCPU on Check Point Security Management due to high CPU usage (SmartLog)

[cyberluke365]

Hello,

I have a Check Point Security Management Server (R81.20 + Take 53) on virtual environment (VMware) with the following specifications:

• 4x1 vCPU
• 32 GB of RAM
• 700 GB of disk (SCSI Controller - LSI Logic Parallel)
• E1000 Virtual Network Adapter

I have observed that periodically the smartlog_server process shows high CPU usage. This occurs when a third-party application accesses the SmartLog via API to retrieve logs (/opt/CPSmartLog-R81.20/log/smartlog_server.elg😞

Questions:

1. Do you suggest increasing the CPU, or is there something I can do to optimize the smartlog_server process usage?
2. Does increasing the vCPU (from 4x1 to 8x1) require the purchase of additional licenses? I don't think so, but a confirmation would be appreciated.
3. Do you also suggest changing the Virtual Network Adapter from E1000 to VMXNET3?
4. Regarding the disk:
   cat /sys/block/<DISK_DEVICE>/device/queue_depth = 32
   cat /sys/block/<DISK_DEVICE>/queue/nr_requests = 128
   According to sk104848 - Best Practices - Performance Optimization of Security Management Server installed on VMwa... the nr_requests should be set to twice the size of the queue_depth. In my case it is four times the queue_depth: do you suggest leaving as it is or decrease to twice ? 
5. Any other thoughts (except for what is already written in sk104848 - Best Practices - Performance Optimization of Security Management Server installed on VMwa...)?

Thank you.

[PhoneBoy]

Gateways require licenses for core usage, management does not.
Changing to vmxnet3 will definitely help performance overall.
I'll defer to others on the other questions.

[Tomer_Noy]

I also suggest to check what the 3rd party component is querying and what is the purpose for it fetching logs. Is it some internal script, or a 3rd party product?

If the 3rd party is continuously querying SmartLog to fetch a large quantity of logs (possibly all logs), then it's not an efficient method. It would make sense to try and switch it to using log exporter that will stream the data as its being ingested, instead of running the query engine to fetch it. If you need partial data, log exporter can also support some filters.

[Hugo_vd_Kooij]

There are some basic questions I have:

- Is adding CPU's expensive? (If you pay $100/year per CPU then investigating is more expensive and adding 4 core more is the fast solution.

• How many gateways do send their logs?
• How much log data do you get per day?
• How long is your retention time?
• How long is your index kept?
• How many qeuries do you expect to handle at the same time?

All of these have impact on your sizing.

[cyberluke365]

Hello guys,

I opened a TAC for this issue. After running a support-provided script (doctor-log.sh) that collects and analyzes system data, it became clear that, based on the recorded workload, the server required an increase in computational resources. After upgrading the CPU and RAM, the issue was resolved, and the server now responds more efficiently.

Thank you for your support.

[Amir_Senn]

1. For the general case - I think that the best way to decide if you need to increase resources is to see if current settings satisfies your operations.

Thumb rules for logging:

a. Can log server handle all the logs sent from all the gateways - if gateways have local logging due to log server resources it's a good sign to increase. In general I also recommend to activate log forwarding if not defined already.

b. If the speed of response satisfies you.

c. If you want to increase log retention (storage)

3. Not only do I recommend to do so, E1000 will be blocked on future versions. The reason is that is EOL from security updates so from security POV it's important.