Skyline and Azure managed prometheus

[Peter_Gastinger]

Hi, 

I am currently playing around with Skyline and Azure managed Prometheus. At the moment, my setup looks following:

Azure NVA (R81.10, latest HFA) -> Prometheus in Container Instance -> Managed Prometheus. 

I would like to get rid of the Container Instance. I tried some settings:

{"enabled": true,
"export-targets": {"add": [
{
"client-auth": {
"oauth2": {
"client_id": "<client_id>",
"client_secret": "<client_secret>",
"token_url": "https://login.microsoftonline.com/<tenant>/oauth2/token",
"scopes": ["https://prometheus.monitor.azure.com"]
}
},
"enabled": true,
"type": "prometheus-remote-write",
"url": "https://<instance>.westeurope-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-1566dca15e8d4766af99f66fbdaeda18/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24"
}
]}
}

 Not sure if this syntax is right, the configuration example only has a basic authentication example. 

According to the logs, this doesn't seem to work. There is a token missing:

2024-03-26T15:36:16.532Z error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 401 Unauthorized; err = %!w(<nil>): {\"Error\":{\"Code\":\"MissingToken\",\"Message\":\"Authentication token required.\"}}", "dropped_items": 416}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47

Has anyone ever played around with that and managed to solve it? 

oauth2 seems to be supported (e.g. opentelemetry-collector-contrib/extension/oauth2clientauthextension at main · open-telemetry/opentel...), but I need a bit of configuration advice here. 

Thank you!

[Elad_Chomsky]

Hi @Peter_Gastinger ,

For the correct format of the payload, refer to the Skyline Administration Guide .

[Peter_Gastinger]

Hi, 

Thanks for your response! Unfortunately, there is nothing oauth2 specific. 

I don't have a static token, but I generated a access_token and tried the example, but it doesn't work. I was using the mlmproxy to check the request and I don't see any authentication related part in the header:

Host: testhost
User-Agent: opentelemetry-collector-binary---check-point-modifications/CPotelcol_0.82.0
Content-Length: 15439
Content-Encoding: snappy
Content-Type: application/x-protobuf
X-Prometheus-Remote-Write-Vers 0.1.0
ion:
Accept-Encoding: gzip
[cannot decode] Couldn't parse: falling back to Raw

This is the example I was using:

{
  "enabled": true,
  "export-targets": {
    "add": [
      {
        "client-auth": {
          "token": {
            "custom-header": {
              "key": "Authorization",
              "value": "Bearer {{ token }}"
            }
          }
        },
        "enabled": true,
        "type": "prometheus-remote-write",
        "url": "https://azuresomething.westeurope-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-1566dca15e8d4766af99f66fbdaeda18/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24"
      }
    ]
  }
}

Any idea?

regards, 

Peter

[Elad_Chomsky · Accepted solution]

Hi @Peter_Gastinger ,

We don't support oauth2, only the base sigv4 authentication from AWS, for any enhancements, please open an RFE - and we will see if we can push it as part of the roadmap.