R81 - External IOC feeds

[Rui_Gomes_PT]

Hi,

Trying to add an external IOC feed in R81

I get an error regarding the ssl certificate. Is there a way to import de CA cert?

Thanks!

[Tal_Paz-Fridman]

Use an actual Custom Intelligence Feed site and it will work. Refer to sk132193 "What is the "Custom Intelligence Feeds" feature?"

[Rui_Gomes_PT]

Hi,

I already know that sk. We are using a internal site to add our ioc. It works if I follow the sk132193, but on the smart console (in R81) I get an error

[Tal_Paz-Fridman]

@Youssef_Obeidal can you look at this - failure validating a certificate from an internal site. 

[Thomas_Eichelbu]

Hello, 

well use this:
"For HTTPS remote feeds, if the certificate update process failed, you can skip the certificate verification. Run: export EXT_IOC_NO_SSL_VALIDATION=1 on the Security Gateway."

choose https for transport:
ioc_feeds add --feed_name XXXYYYZZZ--transport https --resource "https://XXXYYYZZZ" --format [value:1,type:ip]

this should help!

[Mstay]

Team

I will make a brief summary about this issue and the results of the case with the TAC.

Smart Console External IOC Feeds works properly if the GWs are in R81 and above. After long sessions with the TAC, labs, Escalation Team, that was the conclusion. Maybe somebody had luck with different versions, but we could not.  We had 4 different environments with SMS in R81.10 and GWS R80.40

It is clear in documentation the SMS must be in R81 and higher (Smart Console Feature), but not the GWs

From SK this part is confuse

Installation

The feature is integrated in version R80.30 and above.

Note: To import external Custom Intelligence Feeds using SmartConsole in versions R81 and higher, refer to: Threat Prevention R81 Administration Guide > Configuring Advanced Threat Prevention Settings > Configuring Threat Indicators > Importing External Custom Intelligence Feeds > Importing External Custom Intelligence Feeds in SmartConsole.

In some way they must to include the Smart console feature ¨ works properly¨ in GWs with R81 and higher. Was suggested to the TAC to edit the sk132193 and add some captures, Logs queries for verifications as is posted in CHECKMATES threads.

We tested the CLI way and works perfect in the versions they mentioned, but not the Smart console External IOC feeds.

We also realized in all the environment we tested this file could not be found when you troubleshoot

$FWDIR/log/ext_ioc_push.elg

I think with all the tests we made,  there is a lot of information from the case we had to edit the SK and help the community.

Cheers

[Scottc98]

During any of these testings with using an internal server with either a internal or self signed cert, are we saying that the only way to deploy these now is via the GW CLI; along with the cert exception listed in SK132193?

• • For HTTPS remote feeds, if the certificate update process failed, you can skip the certificate verification. Run: export EXT_IOC_NO_SSL_VALIDATION=1 on the Security Gateway(does not survive particular SSH session).
    • To add Environmental variable permanently on gateway,

# sed -i -e '$aEXT_IOC_NO_SSL_VALIDATION=1' $CPDIR/tmp/.CPprofile.sh

# sed -i -e '$asetenv EXT_IOC_NO_SSL_VALIDATION "1"' $CPDIR/tmp/.CPprofile.csh

What I find odd here even if this issue is still persistent with the GW not updating post deployment is that the 'test connectivity' of the external IOC feed via smartconsole doesn't seem to work.

I have one internal server that I can access with credentials via web browser (with providing the untrusted cert error first) and can access the feed with no issues.

When testing via Smartconsole (which the source of the 'test connectivity' should come from my same PC i tested web on), i first get this popup that states " URL resource's certificate is from an untrusted source' with 3 options to choose:   "Approve certificate", "Show certificate" or "Reject Certificate".      

When I select "Approve certificate", i then get this error "Unauthorized" Access is denied due to invalid credentials (401)"

I can't seem to understand why this validation would fail here simply on the test from smartconsole regardless of the possible GW issues once its deployed.

R81.10 with latest smartconsole (Management on JHF 87)

[Scottc98]

Followup question:   When someone clicks on the "approve certificate" in this process, where is that exemption stored?  Is that within our smartconsole app or on the management server itself?  

I ask as I was able to get a proper certificate installed on our source but I continue to get these '401' errors about credentials; which i am 100% certain is not the issue.

I wonder if there is some cache here that I can clear out so to attempts against the new cert (i.e Act like I never attempted before).  

[RemieDK]

Hi Scottc98,

Did you found a solution for the 401 (auth) error in SmartConsole? We have the same issue with an External IOC with authentication. We tried from complex to simple passwords but it does not seem to work. (The username is received correctly on the server side)

In a browser the authentication works, and even through the ioc_feeds CLI command, but that is hard(er) to manage, and we like to keep the CLI settings to an absolute minimum.

Thanks!

[Scottc98 · Accepted solution]

@RemieDK 

Oh....i've had my fun with TAC on this one 🙂

Here is everything I have learned to date:

1.  Deploying IOC feeds with authentication via SmartConsole is NOT supported unless your management server is on R81.20 (Issue ID PMTR-73043).
1. From what I am told via TAC, R&D is not backporting this despite knowing this is an issue and instead telling folks to upgrade management to R81.20 instead  (i.e. since original release).
1. Which….is really annoying since authentication has been listed as an option in R81 and up but has never worked until R81.20.
1. Just remove it as an option in smartconsole if backporting is not going to occur…..or at least a nice warning message until it is 😉
2. After upgrading our management server to R81.20 this weekend (Take 10), I found some more issues :
1. It looks like the 'test connectivity" option now is only allowed on R81.20 gateways 😞  
1. It displays a warning that I have not attempted on my R81.10 gateway
2. Test connectivity on non-authenticated feeds in R81.10 management worked fine so it seems like this is now removed overall if you upgrade management to R81.20.
2. While still able to deploy with a threat policy install, I can confirm that the auth issue is gone......BUT.....i now have a format issue  😠
1. (New TAC case opened now)  - my format is very simple IP list type like the first IP feed example in SK132193)
3. During our initial testing, we also found a bug that wasn't documented regarding the username used for any authentication
1. Only lowercase usernames are allowed today
1. Note:  R&D did write up a port fix for this for R81.10 Take 95 and I was told it will be wrapped into a future JHF
1. That being said, we just changed our username and moved on.....but just a note for others whom might get the SAME auth error message in smartconsole......it could be your username used  😉
4. Checkpoint is VERY 'by the book' on SSL certificates.   Just cause it can chain up in a browser or other systems doesn't mean it going to work here.
1. Had a lot of back and forth with PKI team with internal certs missing some attributes.
2. And no complains here on on the checkpoint end…..just lessons learned on internal cert requirements.....but would be soooooooo great if there was a CLI or Smartconsole option to bypass cert check (See notes at end of thread)

So....yeah.....still working with TAC now so we can avoid the CLI based deployments.   

Somethings that would have made CLI based deployment 'manageable' IMO:

NOTE:  References here are within the "$FWDIR/conf/ioc_feeder.conf"

1. Allow for deployment using an "obfuscated_password" option
1. Since it’s the same feed being deployed over and over to gateways, it would be nice to add to one site manually (i.e using your 'ioc_feeds add" structure and then manually entering the password afterwards)…..then use the hash here to deploy to the other 100+ FWs. 
1. This option would have been workable for us since you could just run it from a repository script in smartconsole.
2. Add a "no_ssl_validation" option in the CLI add/modify.   
1. This field is mentioned as a workaround to bypass certificate validation (i.e. that internal CA use case issue)
2. While you can VI edit the .conf file post deployment, it is overwritten if you do any 'modify' or remove/add to the feed 😞
1. Not to mention a bigger pain to deploy if you have to edit this every time……
3. This option would have saved me weeks on the internal CA issue I was facing.  
4. The option is there……just add it please on a future request 😉

[AntE]

Thank you so much for the excellent write up!

I'm struggling with similar issues while attempting to deploy and external IOC feed, with authentication, via SmartConsole, in our environment (R81.10 on our management and our gateways). I have an SR open with TAC and am wondering if we get the same response about the SmartConsole feed with authentication not being supported until R81.20... 🤔

[AntE]

For what it's worth, I just got a response from TAC confirming that IOC feeds with authentication, via SmartConsole, are not supported prior to R81.20. Sounds like they confirmed what you described in your post. They did mention that we could use the CLI to set the authentication for the SmartConsole IOC:

ioc_feeds add --feed_name NAMEOFFEED --transport https --resource <URL OF FEED> --user_name USERNAME

The other alternative they suggested was to upgrade our management servers to R81.20, which also seems to follow what you posted about (I assume include the potential caveats with testing a non-R81.20 gateway, etc.).

[Scottc98]

@AntE    That seems to align with what I was told.   Deployment of an authenticated feed via 81.20 management seems to work the best verses the CLI method; especially for a large environment.   As you can see with your 'ioc_feeds add' statement, you would then be prompted for the password right afterwards to enter.   Now...imaging doing that on 50+ clusters (100+ gateways) 😞

After playing around in my lab more, I do wish they had the "obfuscated_password" option via CLI.    When it came to deployment planning, you could at least control the deployment via CLI in batches.    I haven't see where you can put in an 'exception' or 'install on' option for the IOC feeds.    Its seems like once you configure it, it gets installed on any thread profile where you have indicators activated in your profile and have the appropriate blades enabled (i.e. Anti-virus/Anti-bot).    While l like the ease of deployment out to prod with just a threat policy install, it would be nice to have more controls here on 'where' you deploy them 

(Note to anyone:  if there is some documentation I am missing on that non-cli ability, kindly point me in the right direction ;))

The ability to 'test' the feed is what I found to be off here between R81.20 Management and non-R81.20 gateways.  I believe that is different from R81.10 'testing' before.   The way I understood the 'test connection' on R81.10 management is that it just tested the credentials/connection from your smartconsole source.    On R81.20, I think they reach out to the gateway you select and actually run a test feed setup to validate not only the credentials/connection to the server but also the content based on your filter feed settings.    If my assumption is correct, I am thinking that Checkpoint has that R81.20 warning simply cause its the only version they can assure will work.  My hope is that it will be allowed on previous versions via some JHF or if its allowed on R81.10 out of the box already, maybe fix that language in the warning ;))

 Oh....and TAC did fix my feed filtering issues.     Evidently, you have to use 'space' as the delimiter on the feed example I mentioned.    REALLY wish that there was something noted in some SK or in the R81.20 Threat guides that breaks those options down 🤔    There is clearly differences in the smartconsole options between R81.10 & R81.20 and can't seem to find docs that support those change options for customers to reference .  

Lastly, I did a quick review of the latest jumbo release for R81.10 (Take 110).   Don't see the username fix included yet (i.e. lowercase only limitation) so i assume that is going to be in another release. 

[AntE]

Ah! Thank you so much for the clarification! You have definitely helped save me a lot of work (and potentially a few headaches, lol)! 😄

You are correct regarding the IOC "Test Connectivity" button in the SmartConsole for R81.10 - it does seem to just test the provided credentials from the SmartConsole machine. I am glad to hear that the testing functionality seem much more robust in R81.20. 

I would also second everything that you've mentioned in terms of additions/improvements to the IOC feed features and documentation. An "obfuscate password" would greatly help with large scale deployments, having more granular controls on where the IOC feeds are deployed would also be useful in larger environments, and more clarification/information in the documentation would be helpful for anyone looking to deploy IOC feeds in their environment.

[Matlu]

Hello, @Tal_Paz-Fridman 
Is it mandatory for the “Indicators” that one explicitly configures, the source server, to work with “csv”?
Or can they work with txt?
I have one SRV with IP 192.168.136.132 where I have two files, one for IPs and another for domains, but both are in TXT format.
Can I create an indicator for each of my files in their original format (TXT), or is it mandatory to convert them to CSV format?
This information is not clear to me.

[PhoneBoy]

ioc_feed files have to be a specific format, as I recall.
If you want to use a flat file format, use a Network Feed object instead, which can be used with the Access or Threat Prevention policy.