topic Re: Error when sending metrics to Promotheus server in OpenTelemetry/Skyline

Error when sending metrics to Promotheus server

Hasslight1 — Mon, 10 Mar 2025 11:03:14 GMT

In a Maestro environment, I often have a node in a security group that no longer sends metrics to Prometheus. Here is the error message:

2025-03-10T08:22:52.012+0100 error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"https://*:443/api/v1/write\": context deadline exceeded", "dropped_items": 750}

Someone can help me?

Re: Error when sending metrics to Promotheus server

cdav — Wed, 23 Apr 2025 07:39:02 GMT

From my experience this has always been a connectivity/firewall issue.

Re: Error when sending metrics to Promotheus server

Vincent_Bacher — Wed, 23 Apr 2025 07:47:17 GMT

Same here. My first step is usually connectivity test to the server configured for open telemetry and then to check firewall logs for drops.

Re: Error when sending metrics to Promotheus server

cdav — Wed, 23 Apr 2025 08:13:06 GMT

@Hasslight1 if you do a tcpdump on the gateway streaming telemetry or Skyline server do you see completed TCP handshake?

Re: Error when sending metrics to Promotheus server

Alexander_Wilke — Sat, 07 Jun 2025 22:37:33 GMT

Hello,

I have this issue, too from time to time. sometimes only one SGM, sometimes multiple SGMs. In general the issue is only with Maestro or 64k. To fix this I run this:

## somehow resets the skyline config gexec -b all -c '/opt/CPotelcol/CPotelcolCli.sh set_dynamic_config "$(cat /opt/CPotelcol/config.json)"' sleep 3 ### add the new config gexec -b all -c 'sklnctl export --set "$(cat /home/admin/skyline_payload.json)"' sleep 3 ### CPview API restart g_all cpview -s off; sleep 3; g_all cpview -s on g_all cpview -a off; sleep 3; g_all cpview -a on ## Restart processes g_all /opt/CPotelcol/CPotelcolCli.sh stop; sleep 3; g_all /opt/CPotelcol/CPotelcolCli.sh start g_all /opt/CPotlpAgent/CPotlpagentCli.sh stop; sleep 3; g_all /opt/CPotlpAgent/CPotlpagentCli.sh start g_all /opt/CPviewExporter/CPviewExporterCli.sh stop; sleep 3; g_all /opt/CPviewExporter/CPviewExporterCli.sh start