topic Re: Skyline and Azure managed prometheus in OpenTelemetry/Skyline

Skyline and Azure managed prometheus

Peter_Gastinger — Tue, 26 Mar 2024 15:37:46 GMT

Hi,

I am currently playing around with Skyline and Azure managed Prometheus. At the moment, my setup looks following:

Azure NVA (R81.10, latest HFA) -> Prometheus in Container Instance -> Managed Prometheus.

I would like to get rid of the Container Instance. I tried some settings:

{"enabled": true, "export-targets": {"add": [ { "client-auth": { "oauth2": { "client_id": "<client_id>", "client_secret": "<client_secret>", "token_url": "https://login.microsoftonline.com/<tenant>/oauth2/token", "scopes": ["https://prometheus.monitor.azure.com"] } }, "enabled": true, "type": "prometheus-remote-write", "url": "https://<instance>.westeurope-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-1566dca15e8d4766af99f66fbdaeda18/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24" } ]} }

Not sure if this syntax is right, the configuration example only has a basic authentication example.

According to the logs, this doesn't seem to work. There is a token missing:

2024-03-26T15:36:16.532Z error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 401 Unauthorized; err = %!w(<nil>): {\"Error\":{\"Code\":\"MissingToken\",\"Message\":\"Authentication token required.\"}}", "dropped_items": 416} go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391 go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125 go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1 go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195 go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1 go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47

Has anyone ever played around with that and managed to solve it?

oauth2 seems to be supported (e.g. opentelemetry-collector-contrib/extension/oauth2clientauthextension at main · open-telemetry/opentelemetry-collector-contrib (github.com)), but I need a bit of configuration advice here.

Thank you!

Re: Skyline and Azure managed prometheus

Elad_Chomsky — Fri, 03 May 2024 19:06:49 GMT

Hi @Peter_Gastinger ,

For the correct format of the payload, refer to the Skyline Administration Guide .

Re: Skyline and Azure managed prometheus

Peter_Gastinger — Thu, 28 Mar 2024 09:23:16 GMT

Hi,

Thanks for your response! Unfortunately, there is nothing oauth2 specific.

I don't have a static token, but I generated a access_token and tried the example, but it doesn't work. I was using the mlmproxy to check the request and I don't see any authentication related part in the header:

Host: testhost User-Agent: opentelemetry-collector-binary---check-point-modifications/CPotelcol_0.82.0 Content-Length: 15439 Content-Encoding: snappy Content-Type: application/x-protobuf X-Prometheus-Remote-Write-Vers 0.1.0 ion: Accept-Encoding: gzip [cannot decode] Couldn't parse: falling back to Raw

This is the example I was using:

{ "enabled": true, "export-targets": { "add": [ { "client-auth": { "token": { "custom-header": { "key": "Authorization", "value": "Bearer {{ token }}" } } }, "enabled": true, "type": "prometheus-remote-write", "url": "https://azuresomething.westeurope-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-1566dca15e8d4766af99f66fbdaeda18/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24" } ] } }

Any idea?

regards,

Peter

Re: Skyline and Azure managed prometheus

Elad_Chomsky — Thu, 28 Mar 2024 09:29:26 GMT

Hi @Peter_Gastinger ,

We don't support oauth2, only the base sigv4 authentication from AWS, for any enhancements, please open an RFE - and we will see if we can push it as part of the roadmap.