Log exporter & Splunk TLS

[cowboyx2]

I have recently switched from OPSEC/LEA to log exporter to send log to my SEIM, which is Splunk. I need to encrypt those syslog files per security requirements. I found the document stating how to do this, sk122323, and it contains a section called TLS Configuration. The instructions show how to generate a number of certificate files. The issue I am am having is that I'm not overly familiar with certificates. I now have all these files that I have generated on a Linux server (the instructions say not to use a Checkpoint appliance) but I'm not sure what to do with them:

1675 Aug 9 09:39 ca.key
1281 Aug 9 09:41 ca.pem
17 Aug 9 09:44 ca.srl
1164 Aug 9 09:43 cp_client.crt
985 Aug 9 09:42 cp_client.csr
1679 Aug 9 09:41 cp_client.key
2421 Aug 9 09:43 cp_client.p12
1164 Aug 9 09:44 server.crt
985 Aug 9 09:44 server.csr
1675 Aug 9 09:43 server.key

I wish the sk provided more detail. What files go where? What do I do with all of these files?

I am running R80.30.

Any assistance would be appreciated.

[the_rock]

Hey @cowboyx2 ,

I asked one of my colleagues about this, since he and I did this for customer few months back, so he will email me the steps and I will update you as soon as I get them.

Cheers!

[the_rock · Accepted solution]

Ok, had a quick chat with my colleague. So, in a nutshell, here is what you need to do. Below certs you have to transfer to CP device (say any dir...can be /var/log/certificates) and then run command from sk:

cp_log_export add name <Name> [domain-server <Name or IP address of Domain Server>] target-server <HostName or IP address of Target Server> target-port <Port on Target Server> protocol {udp | tcp} format {syslog | cef | splunk | logrhythm | leef | rsa | json | generic} [Optional Arguments]

Certs to copy to CP:

1675 Aug 9 09:39 ca.key
1281 Aug 9 09:41 ca.pem
17 Aug 9 09:44 ca.srl
1164 Aug 9 09:43 cp_client.crt
985 Aug 9 09:42 cp_client.csr
1679 Aug 9 09:41 cp_client.key
2421 Aug 9 09:43 cp_client.p12

All the certs you go on Splunk can stay there, but you may need to talk to their support as far as to what commands to run on their end. Also, important on CP side, table with optional parameters in the sk, you need to have encrypted there as well.

Let me know if you get stuck, I can reach out to my colleague again and help you out. Sorry brother, certs are not my stronger side, so apologies if this does not make full sense : (

[cowboyx2]

Thanks for sending me the "missing steps" 😁 I appreciate you checking with your colleague. I will give it a shot and see how it goes.

[cowboyx2]

Just one more question. Do I need to put the files titled server.crt, server.csr & server.key on the Splunk server?

[the_rock]

Yes sir! Its because technically, in this case, CP would be client and your splunk would be the server.

[cowboyx2]

One more question. What is the syntax for adding the cert? According to the sk, I need to add the following optional arguments:

encrypted (Using TLS (SSL) encryption for exporting the logs)

ca-cert (Full path to the CA PEM certificate file)
Relevant only when the value of 'encrypted' is 'true'

client-cert (Full path to the client P12 certificate file)
(Relevant only when the value of 'encrypted' is 'true')

client-secret (The challenge phrase that was used to create the client P12 certificate)
(Relevant only when the value of 'encrypted' is 'true')

I'm thinking maybe the following:
cp_log_export add name splunk_exporter target-server 10.10.10.10 target-port 18184 protocol tcp format splunk read-mode semi-unified encrypted ca-cert var/log/certificates client-cert var/log/certificates client-secret abc123

Do I need a challenge phrase?
Also, when I created the p12 cert, I was asked for an export password. I made it the same as the challenge phrase? Do I even need an export password?
Too many options for someone who is not familiar with this 😞

[the_rock]

Dont worry man, questions are free, no charge ;). Anyway, I believe you got syntax right and Im pretty sure you do need challenge phase as well. If it fails, let me know and I can check with my colleague tomorrow.

[bob111]

Hey I see that this is an old post, but did it work for you? I am currently getting unclear log messages on the server side - there is no tls termination happening.

[cowboyx2]

I never did get around to trying the solution.

[the_rock]

You would need to also add something like below:

cp_log_export set name actualname encrypted true

[crescentwire]

Hey, @bob111 , I'm guessing you're seeing this sort of gibberish in your Splunk dashboard:

I have this working in a lab environment, so let me explain how I did it:

Splunk server

Assuming Splunk is installed under /opt/splunk, you'll want to create a new directory under /opt/splunk/etc/apps:

cd /opt/splunk/etc/apps

mkdir -p sms-tls/{certs,local}

 (I'm using a specific "app" called "sms-tls", but you can choose any name you like.)

Now, depending on what you're using for your signed certificates (external CA, internal CA, etc.), you'll want to issue a new certificate signed by that CA. I'm using an internal CA (OpenSSL on Linux), so here are the steps for that:

openssl genrsa -out server-splunk-key.pem 2048

openssl req -new -key server-splunk-key.pem -out server-splunk-req.pem

openssl x509 -req -in server-splunk-req.pem -CA cacert.pem -CAkey private/cakey.pem -CAcreateserial -out server-splunk-cert.pem -days 365 -sha256

Let me know if you'd like a description of what these commands do, but we're simply creating a new private key called server-splunk-key.pem, generating a CSR and signing it with our private key, and then asking the CA to generate a new certificate, signed with its private key. The final certificate we'll use on Splunk is server-splunk-cert.pem.

Next, we want to concatenate the generated certificate and the generated private key into one single PEM file, like so:

cat server-splunk-cert.pem server-splunk-key.pem >> server-splunk-cert-combined.pem

Copy the following files to /opt/splunk/etc/apps/sms-tls/certs:

• server-splunk-cert-combined.pem
• cacert.pem (your CA's certificate, likely generated when you established your internal CA; an external CA will provide this to you separately after satisfying your CSR)

Next, navigate to the sms-tls directory with cd /opt/splunk/etc/apps/sms-tls/local and create a new file called inputs.conf. Using vi or nano, paste the following contents inside:

# NOTE: I'm restricting TLS versions to 1.1 and 1.2; 1.0 is not supported thanks to many vulnerabilities over the years
[SSL]
serverCert = /opt/splunk/etc/apps/sms-tls/certs/server-splunk-cert-combined.pem
sslVersions = tls,-tls1.0
requireClientCert = true

# NOTE: I'm using TCP port 6514, but you can choose any you like
[tcp-ssl:6514]

Notice the serverCert value is the new combined cert file we created earlier. Make sure this value matches the directory structure and filename exactly!

Next, create another new file called server.conf in the same directory (local). Using vi or nano, paste the following contents inside:

[sslConfig]
serverCert = /opt/splunk/etc/apps/sms-tls/certs/server-splunk-cert-combined.pem
sslRootCAPath = /opt/splunk/etc/apps/sms-tls/certs/cacert.pem
sslVerifyServerCert = true

Finally, your directory structure on Splunk should resemble this:

splunk@splunk-test:/opt/splunk/etc/apps/sms-tls$ tree ./
./
├── certs
│   ├── cacert.pem
│   ├── server-splunk-cert-combined.pem
├── local
    ├── inputs.conf
    └── server.conf

2 directories, 4 files

Restart your Splunk service and proceed to configuring the SMS!

# NOTE: May need to prefix with 'sudo'
/opt/splunk/bin/splunk restart 

SMS server

You probably have this config correct, but I'll include it anyway. Create a new cp_log_export instance using the destination IP address and TCP port you intend to use on your Splunk indexer:

cp_log_export add name splunk target-server 10.5.1.191 target-port 6514 protocol tcp format splunk

Though you will be prompted, do not restart the Log Exporter service yet; proceed to configure TLS.

mkdir $EXPORTERDIR/targets/splunk/tls

Copy the CA's certificate and the generated certificate for the SMS (in PKCS12/P12) format to this new tls directory. If you're not sure how generate a P12 certificate, you can use these steps (assuming you have access to the SMS's private key and generated certificate in PEM format):

# NOTE: Use 'cpopenssl' if on the SMS itself; otherwise, use 'openssl' 
cpopenssl pkcs12 -inkey client-sms-key.pem -in client-sms-cert.pem -export -out client-sms-cert.p12

Next, edit the XML configuration file:

vi $EXPORTERDIR/targets/splunk/targetConfiguration.xml

Between the <transport> tags, we'll need to configure TLS. Look for this section towards the top:

<transport>
    <security></security>
    <!--clear/tls-->
    <!-- the following section is relevant only if <security> is tls -->
    <pem_ca_file></pem_ca_file>
    <p12_certificate_file></p12_certificate_file>
    <client_certificate_challenge_phrase></client_certificate_challenge_phrase>
</transport>

And insert the following values between the various tags:

<transport>
    <security>tls</security><!--clear/tls-->
    <!-- the following section is relevant only if <security> is tls -->
    <pem_ca_file>/opt/CPrt-R81.20/log_exporter/targets/splunk/tls/cacert.pem</pem_ca_file>
    <p12_certificate_file>/opt/CPrt-R81.20/log_exporter/targets/splunk/tls/client-sms-cert.p12</p12_certificate_file>
    <client_certificate_challenge_phrase>changeme</client_certificate_challenge_phrase>
</transport>

Obviously, don't use "changeme" as your decryption password, but you'll want to type whatever password you used to decrypt the private key contained within the P12 file. 

Save changes to the file and quit using :x or :wq!.

Finally, restart the cp_log_export instance to begin sending logs to Splunk:

cp_log_export restart name splunk

Back on Splunk, validate log delivery and indexing with this query:

index="main" <SMS-IP-Address>

You can also monitor the cp_log_export instance's status with:

tail -f $EXPORTERDIR/targets/syslog-ng/log/log_indexer.elg

Let me know if you still run into trouble or if this is working for you. Hopefully it's helpful!

[bob111]

@crescentwire I just saw you reply, thank you very much! I'll try it out and let you know.

Thanks!

[steve_ll]

Hey, I know it's been a while but would that configuration work if I do that in a splunk agent and not on the actual splunk server?

[crescentwire]

Hey @steve_ll , that’s a good question. Do you mean an agent running on an Linux/Windows server? If so I believe you need to utilize an Splunk forwarder if you don’t plan to use the main Splunk indexer (like I showed in the steps above). You might be able to get an agent-only system working, but I haven’t tested that, and I’m not sure that’s a supported config from Splunk’s perspective. But please prove me wrong if you know otherwise!

Let me know if I can help in any other ways!

[steve_ll]

Yes, I meant an agent running on a linux server. I already have a splunk forwarder on it and I managed to send logs from my sms server to the linux server with the splunk forwarder without tls (with udp) and it worked , and for a while now I am struggling to find the right configuration for the splunk forwarder to do it using tls.

I tried your solution for doing it on the splunk server and I see that splunk is not listening on the port. I am using a splunk forwarder because I don't have an option to work on the actual splunk server or on the main index.

[crescentwire]

Got it--my steps should still work, then. It's just a matter of configuring the Splunk forwarder with the proper certificate settings, listening ports, and so on.

You're saying the Splunk forwarder isn't listening on the port you're defining, correct? Silly question, but you've restarted Splunk services after defining the new app/service with the desired port, and have confirmed no errors in Splunk's startup log? The

/opt/splunk/bin/splunk restart

 command output should indicate any trouble. 

[steve_ll]

the service is restarted in a different way on the Splunk forwarder, but I restarted it successfully, which is what I don't understand because if there were issues with the configuration then it would probably fail.

[the_rock]

Not sure if below links may help verify things on Splunk side:

https://docs.splunk.com/Documentation/Splunk/9.0.4/Troubleshooting/Enabledebuglogging

https://docs.splunk.com/Documentation/ES/7.1.0/Admin/Logfiles

[crescentwire]

Got it. I'd be happy to take a look at your config/code if you're comfortable pasting it here (with sections redacted). Let me know!

[steve_ll]

Of the inputs.conf and server.conf? I can but I it is just your configuration 

[the_rock]

Im 99% sure you need to do it on the actual server.