topic Re: Skyline: How to reset skyline config? in OpenTelemetry/Skyline

Skyline: How to reset skyline config?

Toolmaker — Thu, 25 Jan 2024 15:04:43 GMT

After getting Skyline to send data to our Prometheus, I got overconfident and started feeding different json files to

sklnctl export --set "$( cat file.json)".

(along sk178566)

Situation now:

- No new data appears in Prometheus

- Runing sklnctl with the original json file does NOT restore the original (working) behaviour

- "/opt/CPviewExporter/otlp_cpview.log" shows lines of

2024/01/25 15:24:23 max retry time elapsed: rpc error: code = Unavailable desc = connection error:
desc = "transport: Error while dialing dial unix /opt/CPotelcol/grpc_otlp.sock: connect: no such file or directory"

- Running "/opt/CPotelcol/GetOTDynamicConfig.sh | jq ."    gives some strange data, different from what it was before:

     "exporters": {
         "prometheusremotewrite": {
                      "headers": {
                               "authorization": "Basic ${env:PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN}"

                               (^^^^^^^^^^^^ This had been the base64-endcoded user:pass before)

     "service": {
       "pipelines": {
          "metrics/skyline": {
              "exporters": [
                 "prometheusremotewrite",
                 "prometheusremotewrite",
                 "prometheusremotewrite",
                "prometheusremotewrite"
               ],
             "processors": [
                 "batch",
                 "filter/skyline"
               ],
             "receivers": [
                  "otlp"
            ]

"exporters" and "processors" previously contained only one value each.

To me, it looks like sklnctl somehow accumulates entries instead of replacing them.

--> Is there a way to completely reset the skyline configuration so I can restart from scratch?

We are running R81.10 Take 130.

Re: Skyline: How to reset skyline config?

Elad_Chomsky — Thu, 25 Jan 2024 15:34:12 GMT

Hi @Toolmaker ,

The multiple exporters is a known cosmetic issue, we are working to fix it, should not impact Skyline.

Try to run /opt/CPotelcol/CPotelcolCli.sh set_dynamic_config "$(cat /opt/CPotelcol/config.json)", and then re-run the script.

Re: Skyline: How to reset skyline config?

Toolmaker — Fri, 26 Jan 2024 14:27:47 GMT

Hi @Elad_Chomsky,

many thanks for the reply.

/opt/CPotelcol/CPotelcolCli.sh set_dynamic_config "$(cat /opt/CPotelcol/config.json)"

indeed removes the multiple "exporters" values.

Still, no data arrives at prometheus.

What finally did work was using the "old" method via REST.py:

/opt/CPotelcol/REST.py --set_open_telemetry "$(cat payload.json)"

Printed a warning about better using sklnctl, but now prometheus gets fed again.

Some thoughts - not sure if these are related to the issue:

(1) Running
/opt/CPotelcol/GetOTDynamicConfig.sh | jq .exporters.prometheusremotewrite.headers
after REST.py gives

"Authorization": "Basic dXNlcjp0b3BzZWNyZXQK",

(not the real user/pass), while after "sklnctl export" ... it gives

"authorization": "Basic ${env:PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN}"

(2) /var/log/otelcol.env:

After several runs of sklnctl export --set, this file contained many identical lines

PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK
PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK
...

Running REST.py once reduced this to two lines

PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK
prometheus_remote_write_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK

Re: Skyline: How to reset skyline config?

Elad_Chomsky — Fri, 26 Jan 2024 16:01:56 GMT

Hi @Toolmaker ,

HTTP headers are case-insensitive, so this is not an issue, however, we are still curios to understand what happened, please reach out to me on private at eladch@checkpoint.com, so we can collect logs and analyze this issue.

Re: Skyline: How to reset skyline config?

asj — Tue, 28 May 2024 15:34:36 GMT

Hi @Elad_Chomsky ,

I have the same issue, I've fed the configuration with several payload.json files in order to remedy mistakes. Running the CPotelcolCli.sh with set_dynamic_config does not work for me when trying to reset the configuration. When I rerun the script another set of certificate credentials are just added to the configuration.

Any help is highly appreciated.

Re: Skyline: How to reset skyline config?

Elad_Chomsky — Wed, 29 May 2024 07:42:29 GMT

Hi @asj ,

Please open a support ticket for CheckPoint, so we can try to assist you directly.

Re: Skyline: How to reset skyline config?

Alexander_Wilke — Thu, 30 May 2024 18:47:34 GMT

Hello @asj

you can check the current config running

sklnctl --show_open_telemetry

Check it carefully, in my case I use username and password and if I run the sklnctl command to import the config then the username and password entries are empty. Connection will not work to prometheus.

If I use the old REST.py command it works.

I am using autoupdater package Take 93 and all the latest skyline fixes but I had this issue at the beginning of the year, I discussed this and showed this issue in a remote session with our dimond engineer and together with @Elad_Chomsky but this issue still exists and is not fixed.

Re: Skyline: How to reset skyline config?

asj — Tue, 04 Jun 2024 12:14:16 GMT

Hi @Alexander_Wilke

I left the appliance alone since my last post and ran through the process of onboarding the device to Skyline again today with a colleague. Now the device is posting using REST.py. We're unsure whether the device just needed time to clear the configuration or if an automatic update to Skyline components might have fixed the issue.

Re: Skyline: How to reset skyline config?

scordy — Thu, 28 Nov 2024 21:28:21 GMT

Did you ever come up with a solution to this problem? I've just run into what appears to be an identical scenario - sklnctl is accumulating certificate entries rather than replacing / updating. End result is that it no longer connects to Skyline. 😞

Re: Skyline: How to reset skyline config?

Alexander_Wilke — Fri, 29 Nov 2024 07:09:37 GMT

Hello,
no unfortunately support never provided me such a command. The answer was that it will be fixed in newer versions of skyline (can not confirm this happend /happens automatically) or this should not hurt or is just a cosmetic issue. command shows wrong results.

In earlier versions of skyline CPviewExporter, CPotelExporter, CPotelCollector components I always had to use the REST.py command to import the config.json file. With the latest versions of these packages in version Take 40, Take 50, Take 129 the "sklnctl" command worked for me.

In addition the behaviour of CPotelcolCollector changed. In earlier versions it was sending traffic through mplne interface when MDPS was enabled and the otelcol process was added to "add mdps task list process". Since the update of the latest skyline components this is NOT honored anymore and data is sent through dplane instead.

Re: Skyline: How to reset skyline config?

Vincent_Bacher — Fri, 29 Nov 2024 09:23:18 GMT

Skyline components should be maintained by autoupdater.
I usually check the status of the components using autoupdatercli

autoupdatercli show json | jq -r '.products[] | select(."product-name"=="diagnostics") | ."product-components"[] | select(."component-name" == "CPotelcol" or ."component-name" == "CPOtlpAgent" or ."component-name" == "CPviewExporter")'

Re: Skyline: How to reset skyline config?

Elad_Chomsky — Sun, 01 Dec 2024 08:58:14 GMT

Hello all,

In the latest Skyline versions ( CPotelcol 129 ), we have added the option to run sklnctl with a "rebase" operation, this should allow you to completely reset the configuration , before applying the new one.

Also, by default all the Skyline components are now running on the d-plane, if further assistance on MDPS is required, please contact the CheckPoint support, and we will try to assist you.

Re: Skyline: How to reset skyline config?

Nik_Bloemers — Thu, 05 Dec 2024 10:05:23 GMT

Hi Elad,

Can you explain how to do the rebase operation?

Re: Skyline: How to reset skyline config?

scordy — Thu, 05 Dec 2024 20:30:35 GMT

Disregard this - missed a step in recreating the certificates!

2nded to Nik_Bloemers request for more information about the "rebase" operation. I assume this would clear/reset all of the certificates in /var/log/otlp_certs/ ?

Re: Skyline: How to reset skyline config?

Alexander_Wilke — Fri, 06 Dec 2024 09:15:44 GMT

hello,

please provide the exact commands to do this or provide the link to the documentation which describes it, please.

Re: Skyline: How to reset skyline config?

Elad_Chomsky — Sat, 14 Dec 2024 12:36:38 GMT

Hi All,

1) To use the rebase operation - simply switch the word "add" on the payload to "rebase", the relevant flow will activate following that. Notice this is not working with the GAiA API, only the sklnctl executable as of now.

2) And yes - it will hard reset everything.