solr memory usage

[Colin_Campbell1]

Hi,

I have an R80.40 (JHF Take 48) HA pair of MDSes and a single MLS which takes all of the logs. About three weeks ago, the MLS started showing up as disconnected in SmartConsole. SOLR was crashing with "out of memory" error. The MLS had 64GB of RAM and was using a small amount of swap. After trying a number of things with TAC, none of which helped, I ended up adding another 64GB of RAM. After two days, it's started using swap.

The weird thing is, SOLR is showing as using a ridiculous amount of virtual memory. Here's a snapshot from "top"

top - 10:31:15 up 2 days, 20:46, 1 user, load average: 0.25, 0.25, 0.29
Tasks: 354 total, 1 running, 353 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.2 sy, 2.1 ni, 97.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 13151273+total, 5328456 free, 28719140 used, 97465136 buff/cache
KiB Swap: 66107468 total, 66095180 free, 12288 used. 99952048 avail Mem

PID PPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND
10043 9755 admin 39 19 2.789t 0.027t 6.674g S 28.2 22.4 1323:47 8 /opt/CPshrd-R80.40/jre_64/bin/java -D_solr=TRUE -Xdump:director+

How can it purport to be using 2.8TB of VM?

# uname -a
Linux m-va-qld-ir-mdls01 3.10.0-957.21.3cpx86_64 #1 SMP Tue May 19 22:57:34 IDT 2020 x86_64 x86_64 x86_64 GNU/Linux

Any hints on what more information I can provide?

Colin

[PhoneBoy]

Sounds like a memory leak maybe, but that’s just based on the symptoms.
Can you PM me the relevant SR?

[Colin_Campbell1]

If I hit "reply" is that a PM or does everyone see it?

[Maarten_Sjouw]

In the top right click on the envelope and there create a new message to Phoneboy.

Hitting Reply here shows the message to everyone.

[Colin_Campbell1]

Hi,

Thanks

Colin

[PhoneBoy]

Has TAC pointed you at this?
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut... 

[Colin_Campbell1]

No, TAC didn't.  I can confirm the symptoms 

- missing

loadOnStartup=false
transient=true

and the error message occurring.

Thanks will hit up TAC to re-open the case and get their thoughts

[Dror_Aharony]

No real need to wait for TAC then, no?
Simply update to latest JHF (take 67 & above, like latest take 89), as issue is most probably fixed there.
or is that not an option for you now?

Please update here & TAC afterwards...

[Colin_Campbell1]

Hi,

Sorry for the delayed response - been on leave.  When I got back from holidays I found more logging-related problems (smartview and log indexing processes stopped on all domains). Worked with TAC who basically came up with "install JHF Take 89". I'm doing that tonight.

Colin