topic Re: Custom Intelligence Feeds in CIDR format via SmartConsole in General Topics

Custom Intelligence Feeds in CIDR format via SmartConsole

CheckMate-R77 — Mon, 13 Feb 2023 08:16:34 GMT

Dear Mates,

R81.10 Take 87

1. Since Custom Intelligence Feeds via SmartConsole allows only "IP Address" or "IP Range" types, is it possible to apply Spamhaus DROP list which is in CIDR format? In ioc_feeder.elg I get: "Feed format problem. Feed format not supported" - for both IP types.

2. TOR_Exit_Nodes (https://secureupdates.checkpoint.com/IP-list/TOR.txt)

When I "Test Connectivity" it is OK, but in ioc_feeder.elg I get: "Feed format problem. Empty feed file".

Any ideas?

Regards

Re: Custom Intelligence Feeds in CIDR format via SmartConsole

PhoneBoy — Mon, 13 Feb 2023 17:27:58 GMT

Custom Intelligence Feeds files have a very specific format that is required.
It’s documented here: https://support.checkpoint.com/results/sk/sk132193
If the file is not in that format, it won’t work.

Network Feed objects (available in R81.20) should solve both issues.

Re: Custom Intelligence Feeds in CIDR format via SmartConsole

CheckMate-R77 — Tue, 14 Feb 2023 08:35:05 GMT

Thank You PhoneBoy.

1. In sk132193 there is following CLI example for Spamhaus and CIDR format:

Original CSV structure is a list of IP addresses in CIDR format and comment lines are marked as ';'

ioc_feeds add --feed_name ip_cidr_list_with_delimiter --transport https --resource "https://www.spamhaus.org/drop/edrop.txt" --format [value:1,type:ip] --delimiter ";" --comment ";"

When I test it in gateway CLI (I added only "--test true" option to original example) I get following error:

[Expert@gw1:0]# export EXT_IOC_NO_SSL_VALIDATION=1
[Expert@gw1:0]# ioc_feeds add --feed_name ip_cidr_list_with_delimiter --transport https --resource "https://www.spamhaus.org/drop/edrop.txt" --format [value:1,type:ip] --delimiter ";" --comment ";" --test true

Modifying feed ip_cidr_list_with_delimiter
start add
Feed ip_cidr_list_with_delimiter will add on

Feed Name: ip_cidr_list_with_delimiter
Feed is Active
File will be fetched via HTTPS
Resource: https://www.spamhaus.org/drop/edrop.txt
Action: Prevent
Feed is cli managed
Feed type: custom_csv

Fetching active feeds
Something went wrong
Something went wrong
Signatures load failed

The same error when I try to test it with http transport and even in case of local file downloaded by curl_cli - still doesn't work.

2. There is last point in Known Limitations section of sk132193: "Before 81.20, there is limit of number of observables , See sk171988."

Maybe I have exceeded the limit, but where can I find sk171988? I wonder if there is any mechanism to check and eliminate duplicated IOCs (IPs for example) by few external (and maybe overlapping) feeds?

3. I have also noticed differences in notation in sk132193

--format [value:#1 orvalue:1

and

--comment [#] or --comment "#"

It seems both forms are equal?

sk132193 was last modified on 2023-02-07 and seems ... not so actual?

Re: Custom Intelligence Feeds in CIDR format via SmartConsole

PhoneBoy — Tue, 14 Feb 2023 15:14:23 GMT

Hm...if it's an explicitly listed example that doesn't work, might be worth a TAC case.
Regardless, you might see if there are other messages in $FWDIR/log/load_sigs.elg that explain what's happening.
Don't know that there's a way to deduplicate things.

The SK mentioned is internal, but it provides some details on the limits that exist prior to R81.20.
R81.20 has new infrastructure for IOC Feeds and Network Feeds that supports ~2 million observables.