topic Re: Skyline requires too much memory in OpenTelemetry/Skyline

Skyline requires too much memory

Daniel1107 — Mon, 18 Dec 2023 12:36:59 GMT

Hello all,

we run Skyline now for one week with 50 Gatewys and it's working fine.

But for all Gateways, the memory requires about 100Gb per day, and i ask me if there is a solution to reduce this amount of memory? Or can we save the data on a S3 Storage? If so, you can I configure it in Prometheus?

We would like to keep the data up to 30 days ago, so for this we would have 3tb and that's too much.

Any advice would be appreciated.

Thanks

Re: Skyline requires too much memory

Elad_Chomsky — Mon, 18 Dec 2023 16:30:47 GMT

Hi @Daniel1107 , if we are talking about memory consumption on the GW itself then please open a support ticket to CheckPoint, and we will take a look into the subject, if this is on the Prometheus, then I can advise you to try the following:

1) Try to filter only the specific metrics required for your dashboard ( See the sk for the required commands to do so )

2) Scale out your Prometheus, using multiple destinations - ( Or use a solution with better scale up support like VictoriaMetrics, which works the same way as Prometheus )

3) Try to optimize your Prometheus instance ( https://prometheus.io/docs/introduction/faq/ )

Re: Skyline requires too much memory

Dorian — Fri, 26 Jan 2024 02:31:55 GMT

Hi Daniel,

funny enough we had the exact same issue you are describing when we setup Skyline a few months back (59 appliances so I know where you're coming from). I tried playing with the compression, retention flags and anything really that remotely could have something to do with it without any success. At the end of the day what resolved it for us was the implementation of TLS between GWs & Prometheus.. That itself of course most likely didn't resolve it but reissuing the payload command & config attached to that must have given it the bump it needed. Storage went down from what would have been 2TB to 30GB for a month worth of data.

Below is a useful link I found when dealing with this issue (I used the GUI in Prometheus to query it rather than in Grafana)
https://faun.pub/how-to-calculate-the-storage-space-required-for-prometheus-server-24ce4cacc30d

Hope this helps

Re: Skyline requires too much memory

Alexander_Wilke — Fri, 26 Jan 2024 14:46:41 GMT

I do not understand the problem:

Are you talking about memory usage on the gateway or on the prometheus?

Are you talking about disk usage on the prometheus?

Did you configure any other metrics than default?

Re: Skyline requires too much memory

Dorian — Sun, 28 Jan 2024 20:40:52 GMT

Hi @Alexander_Wilke - sorry should of been more specific. I was referring to disk usage on the Prometheus server. This however is not a problem anymore in our deployment. I was just trying to help Daniel by sharing what we experienced. This was at a stage were no other metrics were being configured apart from the default Skyline ones.