I appear to be having the same issue. Everything is setup as es expected including the remote write option when running prometheus.
[Expert@AGFWHS01-Temp:0]# tail otelcol.log
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47
2024-12-04T23:14:58.311Z error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"
http://10.128.251.44:9090/api/v1/write\": context deadline exceeded", "dropped_items": 2802}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47
My open telemetry configuration for a non tls encrypted prometheus server running on an ec2 in AWS.
sklnctl --show_open_telemetry
{"enabled":true,"export-targets":[{"client-auth":{"basic":{"password":"N/A","username":"N/A"},"token":{"custom-header":{"key":"N/A","value":"N/A"},"header-bearer-token":"N/A"}},"enabled":true,"server-auth":{"ca-public-key":{"type":"Default","value":"N/A"}},"type":"prometheusremotewrite","url":"http://10.128.251.44:9090/api/v1/write","name":"prometheusremotewrite"}]}
