Skyline: How to reset skyline config?

[Toolmaker]

After getting Skyline to send data to our Prometheus, I got overconfident and started feeding different json files to

            sklnctl export --set "$( cat file.json)".

(along sk178566)

Situation now:

- No new data appears in Prometheus

- Runing sklnctl with the original json file does NOT restore the original (working) behaviour

- "/opt/CPviewExporter/otlp_cpview.log"  shows lines of

              2024/01/25 15:24:23 max retry time elapsed: rpc error: code = Unavailable desc = connection error:
             desc = "transport: Error while dialing dial unix /opt/CPotelcol/grpc_otlp.sock: connect: no such file or directory"

- Running  "/opt/CPotelcol/GetOTDynamicConfig.sh | jq ."    gives some strange data, different from what it was before:

     "exporters": {
         "prometheusremotewrite": {
                      "headers": {
                               "authorization": "Basic ${env:PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN}"

                               (^^^^^^^^^^^^  This had been the base64-endcoded user:pass before)

     "service": {
       "pipelines": {
          "metrics/skyline": {
              "exporters": [
                 "prometheusremotewrite",
                 "prometheusremotewrite",
                 "prometheusremotewrite",
                "prometheusremotewrite"
               ],
             "processors": [
                 "batch",
                 "filter/skyline"
               ],
             "receivers": [
                  "otlp"
            ]

   "exporters" and "processors" previously contained only one value each.

To me, it looks like sklnctl somehow accumulates entries instead of replacing them.

--> Is there a way to completely reset the skyline configuration so I can restart from scratch?

We are running R81.10 Take 130.

[Elad_Chomsky]

Hi @Toolmaker ,

The multiple exporters is a known cosmetic issue, we are working to fix it, should not impact Skyline.

Try to run /opt/CPotelcol/CPotelcolCli.sh set_dynamic_config "$(cat /opt/CPotelcol/config.json)", and then re-run the script.

[Toolmaker]

Hi @Elad_Chomsky,

many thanks for the reply.

                    /opt/CPotelcol/CPotelcolCli.sh set_dynamic_config "$(cat /opt/CPotelcol/config.json)"

indeed removes the multiple "exporters" values.

Still, no data arrives at prometheus.

What finally did work was using the "old" method via REST.py:

                        /opt/CPotelcol/REST.py --set_open_telemetry "$(cat payload.json)"

Printed a warning about better using sklnctl, but now prometheus gets fed again.

Some thoughts - not sure if these are related to the issue:

(1)  Running
             /opt/CPotelcol/GetOTDynamicConfig.sh | jq .exporters.prometheusremotewrite.headers  
after REST.py gives       

         "Authorization": "Basic dXNlcjp0b3BzZWNyZXQK",

(not the real user/pass), while after "sklnctl export" ... it gives

       "authorization": "Basic ${env:PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN}"


(2) /var/log/otelcol.env:

After several runs of sklnctl export --set, this file contained many identical lines

PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK
PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK
...

Running REST.py once reduced this to two lines

PROMETHEUSREMOTEWRITE_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK
prometheus_remote_write_SKYLINE_BASIC_TOKEN=dXNlcjp0b3BzZWNyZXQK

[Elad_Chomsky]

Hi @Toolmaker , 

HTTP headers are case-insensitive, so this is not an issue, however, we are still curios to understand what happened, please reach out to me on private at eladch@checkpoint.com, so we can collect logs and analyze this issue.