topic Re: 401 http code from perometheus to skyline telemetry in OpenTelemetry/Skyline

401 http code from perometheus to skyline telemetry

aminiy — Fri, 05 Apr 2024 12:33:49 GMT

I have setup skyline exporter via command

but I have nothing in prometheus. When learned tcp dump, got the skyline sending two different type of requests an one of them contain correct basic auth header and other with wrong basic header. How I can correct this?

https://gist.github.com/rajabiy/145da85a4766ee9461812c97c04ef94c

Solution for me to use client-auth.token.custom-header for auth

"client-auth": {
"token": {
"custom-header": {
"key": "Authorization",
"value": "Basic tockenwithpadding="
}
}
}

Re: 401 http code from perometheus to skyline telemetry

Elad_Chomsky — Thu, 28 Mar 2024 10:28:42 GMT

Hi @aminiy ,

Please review /opt/CPotelcol/otelcol.log, and try to inspect the errors. If this is not helping, please open a support ticket to CheckPoint, so we can try to assist you directly.

Re: 401 http code from perometheus to skyline telemetry

aminiy — Fri, 29 Mar 2024 09:55:35 GMT

Hello @Elad_Chomsky
I have this content

2024-03-29T13:15:15.756+0500 error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"http://prom:9090/api/v1/write\": dial tcp prom:9090: connect: connection refused", "dropped_items": 1026}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47
2024-03-29T13:15:30.581+0500 error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"http://prom:9090/api/v1/write\": dial tcp prom:9090: connect: connection refused", "dropped_items": 1026}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47
2024-03-29T13:15:45.605+0500 error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"http://prom:9090/api/v1/write\": dial tcp prom:9090: connect: connection refused", "dropped_items": 1025}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47
2024-03-29T13:16:00.652+0500 error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: remote write returned HTTP status 503 Service Unavailable; err = %!w(<nil>): Service Unavailable", "dropped_items": 1025}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47

Re: 401 http code from perometheus to skyline telemetry

Elad_Chomsky — Fri, 29 Mar 2024 16:07:48 GMT

Hi @aminiy ,

It looks like something is blocking the connection, try to see if a proxy or anything is blocking the connection (like the policy in place ). Try to use CURL or other tools to simulate a call to this API from your gateway.

Re: 401 http code from perometheus to skyline telemetry

aminiy — Fri, 05 Apr 2024 12:34:50 GMT

Hello Elad, the are difference in Equal sign in basic auth header skyline sends without it, I have captured https traffic and decode it this is result.

POST /api/v1/write HTTP/1.0 Host: prom:9000 Connection: close Content-Length: 36209 User-Agent: opentelemetry-collector-binary---check-point-modifications/CPotelcol_0.82.0 Authorization: Basic Y3ayb206UGFzcadvcmQxMjM Content-Encoding: snappy Content-Type: application/x-protobuf X-Prometheus-Remote-Write-Version: 0.1.0 Accept-Encoding: gzip HTTP/1.0 401 Unauthorized Content-Type: text/plain; charset=utf-8 Www-Authenticate: Basic X-Content-Type-Options: nosniff Date: Fri, 05 Apr 2024 08:24:22 GMT Content-Length: 13 Unauthorized GET /favicon.ico HTTP/1.0 Host: prom:9000 Connection: close User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0 Accept: image/avif,image/webp,*/* Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate, br Authorization: Basic Y3ayb206UGFzcadvcmQxMjM= Referer: https://prom:9090/graph?g0.expr=&g0.tab=0&g0.display_mode=lines&g0.show_exemplars=1&g0.range_input=2h Sec-Fetch-Dest: image Sec-Fetch-Mode: no-cors Sec-Fetch-Site: same-origin HTTP/1.0 200 OK Accept-Ranges: bytes Content-Length: 15086 Content-Type: image/x-icon Date: Fri, 05 Apr 2024 08:24:06 GMT ............ .h...6... .... .........00.... ..%..F...(....... ..... ............................................I...........................D............................................Po..Qq......................................................]z..Rq..Rq..`}.............................................................................................J................Tr..Wu..Wu..Wu..Wu..Sr.....................@................................................................................4X..0U..+Q..+Q..+Q..+Q..2W..3X......................................Nn..*Q..,R..,R..*P..h...........................................5Z..+R..,R..,R..*P..Ll..........................................1W..+R..,R..,R..*Q..Df..........................................Gh..-S..+Q..8\..1V..b~.........................G................z...d...2W..Ts..Ll.........................>............................Ce..............................................................^{..........................

Solution for me to use client-auth.token.custom-header for auth

"client-auth": {
"token": {
"custom-header": {
"key": "Authorization",
"value": "Basic tockenwithpadding="
}
}
}