401 http code from perometheus to skyline telemetry

[aminiy]

I have setup skyline exporter via command

but I have nothing in prometheus. When learned tcp dump, got the skyline sending two different type of requests an one of them contain correct basic auth header and other with wrong basic header. How I can correct this?

https://gist.github.com/rajabiy/145da85a4766ee9461812c97c04ef94c 

Solution for me to use client-auth.token.custom-header for auth

"client-auth": {
"token": {
"custom-header": {
"key": "Authorization",
"value": "Basic tockenwithpadding="
}
}
}

[Elad_Chomsky]

Hi @aminiy ,

Please review /opt/CPotelcol/otelcol.log, and try to inspect the errors. If this is not helping, please open a support ticket to CheckPoint, so we can try to assist you directly.

[aminiy]

Hello @Elad_Chomsky 
I have this content

2024-03-29T13:15:15.756+0500 error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"http://prom:9090/api/v1/write\": dial tcp prom:9090: connect: connection refused", "dropped_items": 1026}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47
2024-03-29T13:15:30.581+0500 error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"http://prom:9090/api/v1/write\": dial tcp prom:9090: connect: connection refused", "dropped_items": 1026}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47
2024-03-29T13:15:45.605+0500 error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"http://prom:9090/api/v1/write\": dial tcp prom:9090: connect: connection refused", "dropped_items": 1025}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47
2024-03-29T13:16:00.652+0500 error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: remote write returned HTTP status 503 Service Unavailable; err = %!w(<nil>): Service Unavailable", "dropped_items": 1025}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47

[Elad_Chomsky]

Hi @aminiy ,

It looks like something is blocking the connection, try to see if a proxy or anything is blocking the connection (like the policy in place ). Try to use CURL or other tools to simulate a call to this API from your gateway. 

[aminiy · Accepted solution]

Hello Elad, the are difference in Equal sign in basic auth header skyline sends without it, I have captured  https traffic and decode it this is result.

POST /api/v1/write HTTP/1.0
Host: prom:9000
Connection: close
Content-Length: 36209
User-Agent: opentelemetry-collector-binary---check-point-modifications/CPotelcol_0.82.0
Authorization: Basic Y3ayb206UGFzcadvcmQxMjM
Content-Encoding: snappy
Content-Type: application/x-protobuf
X-Prometheus-Remote-Write-Version: 0.1.0
Accept-Encoding: gzip

HTTP/1.0 401 Unauthorized
Content-Type: text/plain; charset=utf-8
Www-Authenticate: Basic
X-Content-Type-Options: nosniff
Date: Fri, 05 Apr 2024 08:24:22 GMT
Content-Length: 13

Unauthorized


GET /favicon.ico HTTP/1.0
Host: prom:9000
Connection: close
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0
Accept: image/avif,image/webp,*/*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Authorization: Basic Y3ayb206UGFzcadvcmQxMjM=
Referer: https://prom:9090/graph?g0.expr=&g0.tab=0&g0.display_mode=lines&g0.show_exemplars=1&g0.range_input=2h
Sec-Fetch-Dest: image
Sec-Fetch-Mode: no-cors
Sec-Fetch-Site: same-origin

HTTP/1.0 200 OK
Accept-Ranges: bytes
Content-Length: 15086
Content-Type: image/x-icon
Date: Fri, 05 Apr 2024 08:24:06 GMT

............ .h...6...  .... .........00.... ..%..F...(....... ..... ............................................I...........................D............................................Po..Qq......................................................]z..Rq..Rq..`}.............................................................................................J................Tr..Wu..Wu..Wu..Wu..Sr.....................@................................................................................4X..0U..+Q..+Q..+Q..+Q..2W..3X......................................Nn..*Q..,R..,R..*P..h...........................................5Z..+R..,R..,R..*P..Ll..........................................1W..+R..,R..,R..*Q..Df..........................................Gh..-S..+Q..8\..1V..b~.........................G................z...d...2W..Ts..Ll.........................>............................Ce..............................................................^{..........................

Solution for me to use client-auth.token.custom-header for auth

"client-auth": {
"token": {
"custom-header": {
"key": "Authorization",
"value": "Basic tockenwithpadding="
}
}
}