how to ingest skyline data into splunk

[Markus_Malits]

Dear Fellow Checkmates,

Since I was missing an Splunk integration in sk178566, I put together some details for you.

Find below a quick Guide to add Skyline Metrics into Splunk Metric Index on a single instance Splunk server.

ClusterXL Sample Metrics

Prerequisites for a single instance Splunk installation:

• Install Splunk (Follow Instructions here)
• Install Prometheus App for Splunk https://splunkbase.splunk.com/app/4077
• create  an metric type index to use e.g. skyline_prometheus_metric

Configuration of Splunk as receiver:

Create Input to listen for Otel data:
Create inputs.conf inside /opt/splunk/etc/apps/modinput_prometheus/local/inputs.conf with following settings:

[prometheusrw]
port = 8098
maxClients = 10
disabled = 0

[prometheusrw://skyline_via_prometheus_metric]
bearerToken = ABC123
index = skyline_prometheus_metric
sourcetype = prometheus:metric
whitelist = *

Now restart splunk, and check for port 8098 listening.

Check Point Skyline Configuration to send metrics:

Create config.yml with the following content, it will be used in the next step:

{
    "enabled": true,
    "export-targets": {"add": [
        {
            "client-auth": {
              "token": {
               "header-bearer-token": "ABC123"
          }
            },
            "enabled": true,
            "type": "prometheus-remote-write",
            "url": http://splunk.lab.local:8098
        }
    ]}
}

Finaly configure CPotelcol:

/opt/CPotelcol/REST.py --set_open_telemetry “$(cat config.yml)”

On Splunk UI to see the metrics preview:

| mpreview index=skyline_prometheus_metric

To see latest ClusterXL details for example:

| mstats latest(cluster_xl_members_state) as state WHERE index="skyline_prometheus_metric" by host_name,name,id

For details about splunk metric search language refer to:
https://docs.splunk.com/Documentation/Splunk/9.0.4/Metrics/Search

Final Hint: To get proper mapping of clusterxl description, I extracted the lookup from prometheus dashboards provided by Check Point. 

Enjoy, and if there are any questions let me know.

[the_rock]

Wow, thats impressive work @Markus_Malits , thank you very much for sharing! 👍💪

[the_rock]

Its always nice to see put true professional effort to help others, its so valuable.

Vielen Dank @Markus_Malits 

[Network007]

Hi @Markus_Malits - could you please assist me configuring the above solution:

I'm encountering an issue with ingesting data from a Prometheus remote_write_agent into Splunk Enterprise – this solution utilises the ‘Prometheus Metrics for Splunk and is within a Test Environment.

Details:
Splunk Version: Splunk Enterprise 9.2 (Trial License)
Operating System: Ubuntu 22.04
Splunk Application: Prometheus Metrics for Splunk (Latest Version 1.0.1)

Configuration and troubleshooting completed:

1) Splunk Enterprise installed
2) Installed latest compatible version of Prometheus App for Splunk https://splunkbase.splunk.com/app/4077
3) Created a metric type index to use: 'prometheus' - ensured it was enabled

4) Configured inputs.conf (/opt/splunk/etc/apps/modinput_prometheus/local/inputs.conf) to meet the configuration parameters within the inputs.spec.conf (/opt/splunk/etc/apps/modinput_prometheus/README/inputs.conf.spec)

I copied this chunk from the examples in the modinput_prometheus/default/inputs.conf file to ensure no syntax errors. 

5) Confirmed the port 8098 State had changed to Listen

6) Created the config.yml and according to formatting structure above 

However, there appeared to be a syntax error (potentially the ]} causing this error to be returned:

7) Configured and Ran CPotelcol - ran command /opt/CPotelcol/REST.py --set_open_telemetry “$(cat /opt/CPotelcol/config.yml)” - Result: TypeError: <lambda> ( ) missing 1 required positional argument 'val'

I am not able to change the config.yml file to match the JSON payload for Splunk - connection without TLS in Skyline Configuration on Check Point Servers that run Gaia OS - Other Monitoring Tools

I have also tried to isolate the issue with the config.yml by investigating the /opt/CPotecol/REST.py but have been unsuccessful.

Could you please advise on what needs to be fixed to be able to configure the config.yml file correctly, thank you very much.

[Elad_Chomsky]

Hi @Network007 , 

Please review the admin guide, with then new deployment method ( skyline configuration tool ) - you should be able to interact with Splunk directly. 

https://sc1.checkpoint.com/documents/Appliances/Skyline/Content/Topics-AG/Introduction.htm

[Markus_Malits]

Thank you Elad for pointing Network007 in the right direction.
Using PrometheusWrite is still possible, but HEC Input is more convinient, and does not require the addon.

cheers
Markus