Skyline: How to reset skyline config?

[Toolmaker]

After getting Skyline to send data to our Prometheus, I got overconfident and started feeding different json files to

            sklnctl export --set "$( cat file.json)".

(along sk178566)

Situation now:

- No new data appears in Prometheus

- Runing sklnctl with the original json file does NOT restore the original (working) behaviour

- "/opt/CPviewExporter/otlp_cpview.log"  shows lines of

              2024/01/25 15:24:23 max retry time elapsed: rpc error: code = Unavailable desc = connection error:
             desc = "transport: Error while dialing dial unix /opt/CPotelcol/grpc_otlp.sock: connect: no such file or directory"

- Running  "/opt/CPotelcol/GetOTDynamicConfig.sh | jq ."    gives some strange data, different from what it was before:

     "exporters": {
         "prometheusremotewrite": {
                      "headers": {
                               "authorization": "Basic ${env:PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN}"

                               (^^^^^^^^^^^^  This had been the base64-endcoded user:pass before)

     "service": {
       "pipelines": {
          "metrics/skyline": {
              "exporters": [
                 "prometheusremotewrite",
                 "prometheusremotewrite",
                 "prometheusremotewrite",
                "prometheusremotewrite"
               ],
             "processors": [
                 "batch",
                 "filter/skyline"
               ],
             "receivers": [
                  "otlp"
            ]

   "exporters" and "processors" previously contained only one value each.

To me, it looks like sklnctl somehow accumulates entries instead of replacing them.

--> Is there a way to completely reset the skyline configuration so I can restart from scratch?

We are running R81.10 Take 130.

[Elad_Chomsky]

Hi @Toolmaker ,

The multiple exporters is a known cosmetic issue, we are working to fix it, should not impact Skyline.

Try to run /opt/CPotelcol/CPotelcolCli.sh set_dynamic_config "$(cat /opt/CPotelcol/config.json)", and then re-run the script.

[Toolmaker]

Hi @Elad_Chomsky,

many thanks for the reply.

                    /opt/CPotelcol/CPotelcolCli.sh set_dynamic_config "$(cat /opt/CPotelcol/config.json)"

indeed removes the multiple "exporters" values.

Still, no data arrives at prometheus.

What finally did work was using the "old" method via REST.py:

                        /opt/CPotelcol/REST.py --set_open_telemetry "$(cat payload.json)"

Printed a warning about better using sklnctl, but now prometheus gets fed again.

Some thoughts - not sure if these are related to the issue:

(1)  Running
             /opt/CPotelcol/GetOTDynamicConfig.sh | jq .exporters.prometheusremotewrite.headers  
after REST.py gives       

         "Authorization": "Basic dXNlcjp0b3BzZWNyZXQK",

(not the real user/pass), while after "sklnctl export" ... it gives

       "authorization": "Basic ${env:PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN}"


(2) /var/log/otelcol.env:

After several runs of sklnctl export --set, this file contained many identical lines

PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK
PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK
...

Running REST.py once reduced this to two lines

PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK
prometheus_remote_write_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK

[Elad_Chomsky]

Hi @Toolmaker , 

HTTP headers are case-insensitive, so this is not an issue, however, we are still curios to understand what happened, please reach out to me on private at eladch@checkpoint.com, so we can collect logs and analyze this issue.

[asj]

Hi @Elad_Chomsky ,

I have the same issue, I've fed the configuration with several payload.json files in order to remedy mistakes. Running the CPotelcolCli.sh with set_dynamic_config does not work for me when trying to reset the configuration. When I rerun the script another set of certificate credentials are just added to the configuration. 

Any help is highly appreciated. 

[Elad_Chomsky]

Hi @asj ,

Please open a support ticket for CheckPoint, so we can try to assist you directly. 

[Alexander_Wilke]

Hello @asj 

you can check the current config running

sklnctl --show_open_telemetry

Check it carefully, in my case I use username and password and if I run the sklnctl command to import the config then the username and password entries are empty. Connection will not work to prometheus.

If I use the old REST.py command it works.

I am using autoupdater package Take 93 and all the latest skyline fixes but I had this issue at the beginning of the year, I discussed this and showed this issue in a remote session with our dimond engineer and together with @Elad_Chomsky  but this issue still exists and is not fixed.

[asj]

Hi @Alexander_Wilke 

I left the appliance alone since my last post and ran through the process of onboarding the device to Skyline again today with a colleague. Now the device is posting using REST.py. We're unsure whether the device just needed time to clear the configuration or if an automatic update to Skyline components might have fixed the issue.