Skyline: How to reset skyline config?

[Toolmaker]

After getting Skyline to send data to our Prometheus, I got overconfident and started feeding different json files to

            sklnctl export --set "$( cat file.json)".

(along sk178566)

Situation now:

- No new data appears in Prometheus

- Runing sklnctl with the original json file does NOT restore the original (working) behaviour

- "/opt/CPviewExporter/otlp_cpview.log"  shows lines of

              2024/01/25 15:24:23 max retry time elapsed: rpc error: code = Unavailable desc = connection error:
             desc = "transport: Error while dialing dial unix /opt/CPotelcol/grpc_otlp.sock: connect: no such file or directory"

- Running  "/opt/CPotelcol/GetOTDynamicConfig.sh | jq ."    gives some strange data, different from what it was before:

     "exporters": {
         "prometheusremotewrite": {
                      "headers": {
                               "authorization": "Basic ${env:PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN}"

                               (^^^^^^^^^^^^  This had been the base64-endcoded user:pass before)

     "service": {
       "pipelines": {
          "metrics/skyline": {
              "exporters": [
                 "prometheusremotewrite",
                 "prometheusremotewrite",
                 "prometheusremotewrite",
                "prometheusremotewrite"
               ],
             "processors": [
                 "batch",
                 "filter/skyline"
               ],
             "receivers": [
                  "otlp"
            ]

   "exporters" and "processors" previously contained only one value each.

To me, it looks like sklnctl somehow accumulates entries instead of replacing them.

--> Is there a way to completely reset the skyline configuration so I can restart from scratch?

We are running R81.10 Take 130.

[Elad_Chomsky]

Hi @Toolmaker ,

The multiple exporters is a known cosmetic issue, we are working to fix it, should not impact Skyline.

Try to run /opt/CPotelcol/CPotelcolCli.sh set_dynamic_config "$(cat /opt/CPotelcol/config.json)", and then re-run the script.

[Toolmaker]

Hi @Elad_Chomsky,

many thanks for the reply.

                    /opt/CPotelcol/CPotelcolCli.sh set_dynamic_config "$(cat /opt/CPotelcol/config.json)"

indeed removes the multiple "exporters" values.

Still, no data arrives at prometheus.

What finally did work was using the "old" method via REST.py:

                        /opt/CPotelcol/REST.py --set_open_telemetry "$(cat payload.json)"

Printed a warning about better using sklnctl, but now prometheus gets fed again.

Some thoughts - not sure if these are related to the issue:

(1)  Running
             /opt/CPotelcol/GetOTDynamicConfig.sh | jq .exporters.prometheusremotewrite.headers  
after REST.py gives       

         "Authorization": "Basic dXNlcjp0b3BzZWNyZXQK",

(not the real user/pass), while after "sklnctl export" ... it gives

       "authorization": "Basic ${env:PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN}"


(2) /var/log/otelcol.env:

After several runs of sklnctl export --set, this file contained many identical lines

PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK
PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK
...

Running REST.py once reduced this to two lines

PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK
prometheus_remote_write_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK

[Elad_Chomsky]

Hi @Toolmaker , 

HTTP headers are case-insensitive, so this is not an issue, however, we are still curios to understand what happened, please reach out to me on private at eladch@checkpoint.com, so we can collect logs and analyze this issue.

[asj]

Hi @Elad_Chomsky ,

I have the same issue, I've fed the configuration with several payload.json files in order to remedy mistakes. Running the CPotelcolCli.sh with set_dynamic_config does not work for me when trying to reset the configuration. When I rerun the script another set of certificate credentials are just added to the configuration. 

Any help is highly appreciated. 

[Elad_Chomsky]

Hi @asj ,

Please open a support ticket for CheckPoint, so we can try to assist you directly. 

[Alexander_Wilke]

Hello @asj 

you can check the current config running

sklnctl --show_open_telemetry

Check it carefully, in my case I use username and password and if I run the sklnctl command to import the config then the username and password entries are empty. Connection will not work to prometheus.

If I use the old REST.py command it works.

I am using autoupdater package Take 93 and all the latest skyline fixes but I had this issue at the beginning of the year, I discussed this and showed this issue in a remote session with our dimond engineer and together with @Elad_Chomsky  but this issue still exists and is not fixed.

[asj]

Hi @Alexander_Wilke 

I left the appliance alone since my last post and ran through the process of onboarding the device to Skyline again today with a colleague. Now the device is posting using REST.py. We're unsure whether the device just needed time to clear the configuration or if an automatic update to Skyline components might have fixed the issue. 

[scordy]

Did you ever come up with a solution to this problem? I've just run into what appears to be an identical scenario - sklnctl is accumulating certificate entries rather than replacing / updating. End result is that it no longer connects to Skyline. 😞

[Alexander_Wilke]

Hello,
no unfortunately support never provided me such a command. The answer was that it will be fixed in newer versions of skyline (can not confirm this happend /happens automatically)  or this should not hurt or is just a cosmetic issue. command shows wrong results.

In earlier versions of skyline CPviewExporter, CPotelExporter, CPotelCollector components I always had to use the REST.py command to import the config.json file. With the latest versions of these packages in version Take 40, Take 50, Take 129 the "sklnctl" command worked for me.

In addition the behaviour of CPotelcolCollector changed. In earlier versions it was sending traffic through mplne interface when MDPS was enabled and the otelcol process was added to "add mdps task list process". Since the update of the latest skyline components this is NOT honored anymore and data is sent through dplane instead.

[Vincent_Bacher]

Skyline components should be maintained by autoupdater.
I usually check the status of the components using autoupdatercli

autoupdatercli show json | jq -r '.products[] | select(."product-name"=="diagnostics") | ."product-components"[] | select(."component-name" == "CPotelcol" or ."component-name" == "CPOtlpAgent" or ."component-name" == "CPviewExporter")'

[Elad_Chomsky]

Hello all,

In the latest Skyline versions ( CPotelcol 129 ), we have added the option to run sklnctl with a "rebase" operation, this should allow you to completely reset the configuration , before applying the new one. 

Also, by default all the Skyline components are now running on the d-plane, if further assistance on MDPS is required, please contact the CheckPoint support, and we will try to assist you. 

[Nik_Bloemers]

Hi Elad,

Can you explain how to do the rebase operation?

[Alexander_Wilke]

hello,

please provide the exact commands to do this or provide the link to the documentation which describes it, please.

[scordy]

Disregard this - missed a step in recreating the certificates!

2nded to Nik_Bloemers request for more information about the "rebase" operation. I assume this would clear/reset all of the certificates in /var/log/otlp_certs/ ?

[Elad_Chomsky]

Hi All,

1) To use the rebase operation - simply switch the word "add" on the payload to "rebase", the relevant flow will activate following that. Notice this is not working with the GAiA API, only the sklnctl executable as of now.

2) And yes - it will hard reset everything.