Hey, @bob111 , I'm guessing you're seeing this sort of gibberish in your Splunk dashboard:
I have this working in a lab environment, so let me explain how I did it:
Splunk server
Assuming Splunk is installed under /opt/splunk, you'll want to create a new directory under /opt/splunk/etc/apps:
cd /opt/splunk/etc/apps
mkdir -p sms-tls/{certs,local}
(I'm using a specific "app" called "sms-tls", but you can choose any name you like.)
Now, depending on what you're using for your signed certificates (external CA, internal CA, etc.), you'll want to issue a new certificate signed by that CA. I'm using an internal CA (OpenSSL on Linux), so here are the steps for that:
openssl genrsa -out server-splunk-key.pem 2048
openssl req -new -key server-splunk-key.pem -out server-splunk-req.pem
openssl x509 -req -in server-splunk-req.pem -CA cacert.pem -CAkey private/cakey.pem -CAcreateserial -out server-splunk-cert.pem -days 365 -sha256
Let me know if you'd like a description of what these commands do, but we're simply creating a new private key called server-splunk-key.pem, generating a CSR and signing it with our private key, and then asking the CA to generate a new certificate, signed with its private key. The final certificate we'll use on Splunk is server-splunk-cert.pem.
Next, we want to concatenate the generated certificate and the generated private key into one single PEM file, like so:
cat server-splunk-cert.pem server-splunk-key.pem >> server-splunk-cert-combined.pem
Copy the following files to /opt/splunk/etc/apps/sms-tls/certs:
Next, navigate to the sms-tls directory with cd /opt/splunk/etc/apps/sms-tls/local and create a new file called inputs.conf. Using vi or nano, paste the following contents inside:
# NOTE: I'm restricting TLS versions to 1.1 and 1.2; 1.0 is not supported thanks to many vulnerabilities over the years
[SSL]
serverCert = /opt/splunk/etc/apps/sms-tls/certs/server-splunk-cert-combined.pem
sslVersions = tls,-tls1.0
requireClientCert = true
# NOTE: I'm using TCP port 6514, but you can choose any you like
[tcp-ssl:6514]
Notice the serverCert value is the new combined cert file we created earlier. Make sure this value matches the directory structure and filename exactly!
Next, create another new file called server.conf in the same directory (local). Using vi or nano, paste the following contents inside:
[sslConfig]
serverCert = /opt/splunk/etc/apps/sms-tls/certs/server-splunk-cert-combined.pem
sslRootCAPath = /opt/splunk/etc/apps/sms-tls/certs/cacert.pem
sslVerifyServerCert = true
Finally, your directory structure on Splunk should resemble this:
splunk@splunk-test:/opt/splunk/etc/apps/sms-tls$ tree ./
./
├── certs
│ ├── cacert.pem
│ ├── server-splunk-cert-combined.pem
├── local
├── inputs.conf
└── server.conf
2 directories, 4 files
Restart your Splunk service and proceed to configuring the SMS!
# NOTE: May need to prefix with 'sudo'
/opt/splunk/bin/splunk restart
SMS server
You probably have this config correct, but I'll include it anyway. Create a new cp_log_export instance using the destination IP address and TCP port you intend to use on your Splunk indexer:
cp_log_export add name splunk target-server 10.5.1.191 target-port 6514 protocol tcp format splunk
Though you will be prompted, do not restart the Log Exporter service yet; proceed to configure TLS.
mkdir $EXPORTERDIR/targets/splunk/tls
Copy the CA's certificate and the generated certificate for the SMS (in PKCS12/P12) format to this new tls directory. If you're not sure how generate a P12 certificate, you can use these steps (assuming you have access to the SMS's private key and generated certificate in PEM format):
# NOTE: Use 'cpopenssl' if on the SMS itself; otherwise, use 'openssl'
cpopenssl pkcs12 -inkey client-sms-key.pem -in client-sms-cert.pem -export -out client-sms-cert.p12
Next, edit the XML configuration file:
vi $EXPORTERDIR/targets/splunk/targetConfiguration.xml
Between the <transport> tags, we'll need to configure TLS. Look for this section towards the top:
<transport>
<security></security>
<!--clear/tls-->
<!-- the following section is relevant only if <security> is tls -->
<pem_ca_file></pem_ca_file>
<p12_certificate_file></p12_certificate_file>
<client_certificate_challenge_phrase></client_certificate_challenge_phrase>
</transport>
And insert the following values between the various tags:
<transport>
<security>tls</security><!--clear/tls-->
<!-- the following section is relevant only if <security> is tls -->
<pem_ca_file>/opt/CPrt-R81.20/log_exporter/targets/splunk/tls/cacert.pem</pem_ca_file>
<p12_certificate_file>/opt/CPrt-R81.20/log_exporter/targets/splunk/tls/client-sms-cert.p12</p12_certificate_file>
<client_certificate_challenge_phrase>changeme</client_certificate_challenge_phrase>
</transport>
Obviously, don't use "changeme" as your decryption password, but you'll want to type whatever password you used to decrypt the private key contained within the P12 file.
Save changes to the file and quit using :x or :wq!.
Finally, restart the cp_log_export instance to begin sending logs to Splunk:
cp_log_export restart name splunk
Back on Splunk, validate log delivery and indexing with this query:
index="main" <SMS-IP-Address>
You can also monitor the cp_log_export instance's status with:
tail -f $EXPORTERDIR/targets/syslog-ng/log/log_indexer.elg
Let me know if you still run into trouble or if this is working for you. Hopefully it's helpful!
