topic Re: solr memory usage in Firewall and Security Management

solr memory usage

Colin_Campbell1 — Thu, 17 Dec 2020 00:36:26 GMT

Hi,

I have an R80.40 (JHF Take 48) HA pair of MDSes and a single MLS which takes all of the logs. About three weeks ago, the MLS started showing up as disconnected in SmartConsole. SOLR was crashing with "out of memory" error. The MLS had 64GB of RAM and was using a small amount of swap. After trying a number of things with TAC, none of which helped, I ended up adding another 64GB of RAM. After two days, it's started using swap.

The weird thing is, SOLR is showing as using a ridiculous amount of virtual memory. Here's a snapshot from "top"

top - 10:31:15 up 2 days, 20:46, 1 user, load average: 0.25, 0.25, 0.29
Tasks: 354 total, 1 running, 353 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.2 sy, 2.1 ni, 97.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 13151273+total, 5328456 free, 28719140 used, 97465136 buff/cache
KiB Swap: 66107468 total, 66095180 free, 12288 used. 99952048 avail Mem

PID PPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND
10043 9755 admin 39 19 2.789t 0.027t 6.674g S 28.2 22.4 1323:47 8 /opt/CPshrd-R80.40/jre_64/bin/java -D_solr=TRUE -Xdump:director+

How can it purport to be using 2.8TB of VM?

# uname -a
Linux m-va-qld-ir-mdls01 3.10.0-957.21.3cpx86_64 #1 SMP Tue May 19 22:57:34 IDT 2020 x86_64 x86_64 x86_64 GNU/Linux

Any hints on what more information I can provide?

Colin

Re: solr memory usage

PhoneBoy — Thu, 17 Dec 2020 01:57:13 GMT

Sounds like a memory leak maybe, but that’s just based on the symptoms.
Can you PM me the relevant SR?

Re: solr memory usage

Colin_Campbell1 — Thu, 17 Dec 2020 02:28:40 GMT

If I hit "reply" is that a PM or does everyone see it?

Re: solr memory usage

Maarten_Sjouw — Thu, 17 Dec 2020 06:48:52 GMT

In the top right click on the envelope and there create a new message to Phoneboy.

Hitting Reply here shows the message to everyone.

Re: solr memory usage

Colin_Campbell1 — Mon, 21 Dec 2020 02:57:41 GMT

Hi,

Thanks

Colin

Re: solr memory usage

PhoneBoy — Wed, 23 Dec 2020 01:36:06 GMT

Has TAC pointed you at this?
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk168616

Re: solr memory usage

Colin_Campbell1 — Wed, 23 Dec 2020 02:18:02 GMT

No, TAC didn't. I can confirm the symptoms

- missing

loadOnStartup=false
transient=true

and the error message occurring.

Thanks will hit up TAC to re-open the case and get their thoughts

Re: solr memory usage

Dror_Aharony — Sun, 27 Dec 2020 08:58:00 GMT

No real need to wait for TAC then, no?
Simply update to latest JHF (take 67 & above, like latest take 89), as issue is most probably fixed there.
or is that not an option for you now?

Please update here & TAC afterwards...

Re: solr memory usage

Colin_Campbell1 — Mon, 18 Jan 2021 23:59:23 GMT

Hi,

Sorry for the delayed response - been on leave. When I got back from holidays I found more logging-related problems (smartview and log indexing processes stopped on all domains). Worked with TAC who basically came up with "install JHF Take 89". I'm doing that tonight.

Colin