Writing to Prometheus Issue for all gateways/managers but one

cdav — Fri, 06 Dec 2024 16:04:02 GMT

I have been trying to configure my CheckPoint estate to stream open telemetry to a prometheus/grafana instance. I have followed all relevent documentation/videos yet only one appliance out of a few is working.

For example i have a pair of managers both configured the same but only one has logs successfully written to prometheus. The one that doesnt work shows the following in the otelcol.log:

The same can be applied to some gateways that i have also tried this on.

Has anyone else encountered this issue?

2024-12-06T15:11:04.302Z error exporterhelper/queued_retry.go:391 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post \"http://10.128.251.44:9090/api/v1/write\": context deadline exceeded", "dropped_items": 269}

go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send

go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:391

go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send

go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/metrics.go:125

go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1

go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/queued_retry.go:195

go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1

go.opentelemetry.io/collector/exporter@v0.82.0/exporterhelper/internal/bounded_memory_queue.go:47

Re: Writing to Prometheus Issue for all gateways/managers but one

cdav — Sun, 08 Dec 2024 21:31:36 GMT

Seems I can get both managers and event box to work but i can not get any of my 5 clusters to work. All return the above issue in otelcol.log.

Can anyone suggest some log files worth looking in on either cluster members or prometheus. Looking at the resources on prometheus server there is no trace of high utilisation across disk/network/cpu/memory.

Re: Writing to Prometheus Issue for all gateways/managers but one

cdav — Wed, 05 Feb 2025 19:24:19 GMT

this was resolved - firewall issue. Was not allowing VIP address through firewall.

topic Writing to Prometheus Issue for all gateways/managers but one in OpenTelemetry/Skyline

Writing to Prometheus Issue for all gateways/managers but one

Re: Writing to Prometheus Issue for all gateways/managers but one

Re: Writing to Prometheus Issue for all gateways/managers but one