Error when sending metrics to Promotheus server

[Hasslight1]

In a Maestro environment, I often have a node in a security group that no longer sends metrics to Prometheus. Here is the error message:

2025-03-10T08:22:52.012+0100    error   exporterhelper/queued_retry.go:391      Exporting failed. The error is not retryable. Dropping data.   {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"https://*:443/api/v1/write\": context deadline exceeded", "dropped_items": 750}

Someone can help me?

[cdav]

From my experience this has always been a connectivity/firewall issue.

[Vincent_Bacher]

Same here. My first step is usually connectivity test to the server configured for open telemetry and then to check firewall logs for drops.

[cdav]

@Hasslight1 if you do a tcpdump on the gateway streaming telemetry or Skyline server do you see completed TCP handshake? 

[Alexander_Wilke]

Hello,

I have this issue, too from time to time. sometimes only one SGM, sometimes multiple SGMs. In general the issue is only with Maestro or 64k. To fix this I run this:

## somehow resets the skyline config
gexec -b all -c '/opt/CPotelcol/CPotelcolCli.sh set_dynamic_config "$(cat /opt/CPotelcol/config.json)"'
sleep 3

### add the new config
gexec -b all -c 'sklnctl export --set "$(cat /home/admin/skyline_payload.json)"'
sleep 3

### CPview API restart
g_all cpview -s off;  sleep 3; g_all cpview -s on
g_all cpview -a off;  sleep 3; g_all cpview -a on

## Restart processes
g_all /opt/CPotelcol/CPotelcolCli.sh stop; sleep 3; g_all /opt/CPotelcol/CPotelcolCli.sh start
g_all /opt/CPotlpAgent/CPotlpagentCli.sh stop; sleep 3; g_all /opt/CPotlpAgent/CPotlpagentCli.sh start
g_all /opt/CPviewExporter/CPviewExporterCli.sh stop; sleep 3; g_all /opt/CPviewExporter/CPviewExporterCli.sh start