how to ingest skyline data into splunk

[Markus_Malits]

Dear Fellow Checkmates,

Since I was missing an Splunk integration in sk178566, I put together some details for you.

Find below a quick Guide to add Skyline Metrics into Splunk Metric Index on a single instance Splunk server.

ClusterXL Sample Metrics

Prerequisites for a single instance Splunk installation:

• Install Splunk (Follow Instructions here)
• Install Prometheus App for Splunk https://splunkbase.splunk.com/app/4077
• create  an metric type index to use e.g. skyline_prometheus_metric

Configuration of Splunk as receiver:

Create Input to listen for Otel data:
Create inputs.conf inside /opt/splunk/etc/apps/modinput_prometheus/local/inputs.conf with following settings:

[prometheusrw]
port = 8098
maxClients = 10
disabled = 0

[prometheusrw://skyline_via_prometheus_metric]
bearerToken = ABC123
index = skyline_prometheus_metric
sourcetype = prometheus:metric
whitelist = *

Now restart splunk, and check for port 8098 listening.

Check Point Skyline Configuration to send metrics:

Create config.yml with the following content, it will be used in the next step:

{
    "enabled": true,
    "export-targets": {"add": [
        {
            "client-auth": {
              "token": {
               "header-bearer-token": "ABC123"
          }
            },
            "enabled": true,
            "type": "prometheus-remote-write",
            "url": http://splunk.lab.local:8098
        }
    ]}
}

Finaly configure CPotelcol:

/opt/CPotelcol/REST.py --set_open_telemetry “$(cat config.yml)”

On Splunk UI to see the metrics preview:

| mpreview index=skyline_prometheus_metric

To see latest ClusterXL details for example:

| mstats latest(cluster_xl_members_state) as state WHERE index="skyline_prometheus_metric" by host_name,name,id

For details about splunk metric search language refer to:
https://docs.splunk.com/Documentation/Splunk/9.0.4/Metrics/Search

Final Hint: To get proper mapping of clusterxl description, I extracted the lookup from prometheus dashboards provided by Check Point. 

Enjoy, and if there are any questions let me know.

[the_rock]

Wow, thats impressive work @Markus_Malits , thank you very much for sharing! 👍💪

[the_rock]

Its always nice to see put true professional effort to help others, its so valuable.

Vielen Dank @Markus_Malits