What is the maximum IOC feed range?

[Pachango]

Hello all,

We're getting some error that have gone by unnoticed for possibly a long time in the IOC_Feeder.elg:

observable[1044] ::span_indicator_range: [ERROR] range is to wide, addr count is: 4096, skip 206.209.192.0-206.209.207.255

This is just one of 60 lines in that log file that fails for that same error ranging in size from 4000 to 1000000 hosts. All the other lines work just fine and are being parsed without a problem.

So my question is what is the maximum range that can be given? And why is it maxed either way?

sk132193 gives no information about the max amount of hosts.

Kind regards

B

[PhoneBoy]

There are some limits to the number of observables we support in releases prior to R81.20.
Unfortunately, it’s difficult to provide an exact number due to the number of other things that leverage the same infrastructure.
I suspect this “limit” on ranges is related to this limitation.

In R81.20, we’ve changed the mechanism so it uses different infrastructure that is faster to boot.
We can also state a definitive limit: 2 million observables.

[Pachango]

Thank you for the information! However there doesn't seem to be an issue with the amount of observables but with the amount of hosts in an observable. Or am i wrong in my understanding?

[the_rock]

I believe you are correct.

[Pachango]

I guess I'll have to open a TAC case with RND support than probably to get this sorted out? Seems weird that the amount of hosts would be capped as other CP products have experience with handling large ranges with no problem... 

[the_rock]

I think thats probably your best bet, see if they can provide something official for you.

[Pachango · Accepted solution]

I have reached out to TAC and there is a limit to 1024 IP's per observable (which our lists greatly exceed) and a limit to 1M characters per feed.

"After an internal discussion with R&D, we discovered a different limitation that has to do with the range error you observed.
The limit for the IP range is 1024 IP's. So you will need to put up to 1024 IP's per range and split it into different lines.
Other than that, the IOC feed follows the standard limitation of 1M characters per feed."

[the_rock]

Good job, thanks for sharing 👍

[PhoneBoy]

To get the precise reason these limits are being encountered, your best best is to work with TAC.
My guess (and it's only that) is that the range of IPs is being turned into something bigger that would exceed the limits of the underlying infrastructure.

[Henrik_Noerr1]

Hey PhoneBoy,

I can't read from your answer if the range issue is fixed in r81.20.

And besides, will the range fix be available in r81.10

Regards,

Henrik 

[PhoneBoy]

Since I didn't know for certain, I created an IOC file that looks like:

; Test file
206.209.192.0-206.209.207.255 ; test
172.16.0.0-172.31.255.255 ; test2

I then imported it into R81.20 with: ioc_feeds add --feed_name test_ip_feed --transport local_file --resource "/home/admin/test.csv" --format [value:1,type:ip] --delimiter ";" --comment ";"

The import succeeded, telling me: 1052672 IPv4 addresses loaded.
That leads me to believe this issue either doesn't exist in R81.20, or the limit is much higher. 

The likely reason for this limit in R81.10 and earlier is due to the underlying infrastructure used.
We've addressed this in R81.20 with new infrastructure that is far more scalable and performant to the task at hand.
Generally, new infrastructure isn't backported to earlier releases.

[the_rock]

@PhoneBoy 

Just tested on my R81.20 mgmt server and all I get is below:

Feed Name: test_ip_feed
Feed is Active
File is locally on the Gateway
Path: /home/admin/ioc.csv
Action: Prevent
Feed is cli managed

Fetching active feeds

Update summary
##############

[Blason_R]

Why cant you user fwaccel dos deny -l feature? Its far better and faster than to block using ioc_feeds

[the_rock]

Interesting point. @Blason_R ...so just wondering. Say you navigate to that dir $FWDIR/conf/deny_list and create a file called blocked_ips.txt and literally input single line in it (just a s tupid example)

say line containing below:

205.50.0.0-205.78.0.255

Are you saying by executing command fwaccel dos deny -l or -L, it would load the range )ranges indicated) and you simply need to push policy to the fw and thats it? Or no policy push needed?

Andy

[Blason_R]

There is no need to install the policy at all since the IPs get blocked at sxl level; just import the text file and you are done. With little bit of automation and bash scripting you can block millions of IP addresses at one go without even pushing or creating objects.

I guess ioc_feeds block the IP addresses much lower in a chain while fwaccel dos deny knocks the packet off in first 3 chain

[PhoneBoy]

The problem with using this command directly is you can't just "import a file" to change the rules.
Besides, at least in R81.20, it's using the same mechanism as fwaccel dos.

See below:

[Expert@R8120S:0]# fwaccel dos stats get

Firewall Instances in Aggregate:
    Memory Usage:                      0
    Total Active Connections:  (FW connection limiting inactive)
    New Connections/Second:    (FW connection limiting inactive)
    Number of Elements in Tables:
        Penalty Box Violating IPs:                     0
        Rate Limit Source Only Tracks:                 0
        Rate Limit Source and Service Tracks:          0
        Rate Limit Dest Only Tracks:                   0
        Rate Limit Dest and Service Tracks:            0

SecureXL:
    Memory Usage:                      0
    Packets/Second:                    (rate limiting inactive)
    Bytes/Second:                      (rate limiting inactive)
    Reasons Packets Dropped:
        IP Fragment:          0
        IP Option:            0
        Penalty Box:          0
        Deny List:            0
        Rate Limit:           0
    Number of Elements in Tables:
        Penalty Box:                                   0
        Non-Empty Deny Lists:                          1
        Deny List IPs:                           1052672
        Rate Limit Matches:                            0
        Rate Limit Source Only Tracks:                 0
        Rate Limit Source and Service Tracks:          0
        Rate Limit Dest Only Tracks:                   0
        Rate Limit Dest and Service Tracks:            0

[Blason_R]

You can import the file with -l option

[the_rock]

Thanks! Will give it a go.

[PhoneBoy]

Not sure how it handles when that file changes, though (does it remove the entries when they no longer show in the file, etc).
In any case, ioc_feeds actually will manage that.

[PhoneBoy]

That command is meant to be executed on the Security Gateway (not the management).

[the_rock]

I also did do it on the gateway, but got exact same result.

[Blason_R]

Its pretty much possible if that is used to cprid_util command from mgmt. This is the way I am doing. 

Even ioc_feeds needs to be executed from gateway. 

[the_rock]

I actually did ioc_feeds add commands from the below sk and output was the same, so Im guessing its probably due to the fact I dont have av/ab blades enabled and its VM, but not sure. I dont have access to actual physical CP appliance to test this theory. Does not show me number of entries, which would be nice to see 🙂

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

Known feeds examples (using the Custom CSV feature)

Description	URL	Command Line
Alienvault IP Reputation	http://reputation.alienvault.com/reputation.data	ioc_feeds add --feed_name reputation --transport http --resource "http://reputation.alienvault.com/reputation.data" --format [type:ip,value:#1,comment:#4] --delimiter "#"
Domains	https://www.botvrij.eu/data/ioclist.hostname.raw	ioc_feeds add --feed_name domains --transport https --resource "https://www.botvrij.eu/data/ioclist.hostname.raw" --format [type:domain,value:#1]
IPs	https://sslbl.abuse.ch/blacklist/sslipblacklist.csv	ioc_feeds add --feed_name ips --transport https --resource "https://sslbl.abuse.ch/blacklist/sslipblacklist.csv" --format [type:ip,value:#2] --comment [#] --delimiter ","
Talos IP Blacklist	http://www.talosintelligence.com/documents/ip-blacklist	ioc_feeds add --feed_name ip_blacklist --transport https --resource "https://www.talosintelligence.com/documents/ip-blacklist" --format [type:ip,value:#1]
Spam List	http://www.ipspamlist.com/public_feeds.csv	ioc_feeds add --feed_name spam_list --transport https --resource "https://www.ipspamlist.com/public_feeds.csv" --format [type:ip,value:#3,comment:#4] --comment ["#", "first_seen"] --delimiter ","
Cybercrime hash list	http://cybercrime-tracker.net/ccamlist.php	ioc_feeds add --feed_name hash_list --transport http --resource "http://cybercrime-tracker.net/ccamlist.php" --format [type:sha1,value:#1]

[Expert@quantum-firewall:0]# ioc_feeds add --feed_name reputation --transport http --resource "http://reputation.alienvault.com/reputation.data" --format [type:ip,value:#1,comment:#4] --delimiter "#"
start add
HTTP url feed transport is insecure and not recommended. Please consider using only url feeds with HTTPS transport.
Default value for active is: true
Default value for feed_action is: prevent
Feed reputation will add on

Feed Name: reputation
Feed is Active
File will be fetched via HTTP
Resource: http://reputation.alienvault.com/reputation.data
Action: Prevent
Feed is cli managed

Fetching active feeds

Update summary
##############

[Expert@quantum-firewall:0]# ioc_feeds add --feed_name domains --transport https --resource "https://www.botvrij.eu/data/ioclist.hostname.raw" --format [type:domain,value:#1]
start add
Default value for active is: true
Default value for feed_action is: prevent
Feed domains will add on

Feed Name: domains
Feed is Active
File will be fetched via HTTPS
Resource: https://www.botvrij.eu/data/ioclist.hostname.raw
Action: Prevent
Feed is cli managed

'proxy'
SHA256 Fingerprint=F7:02:33:19:BB:93:D4:83:88:21:42:03:9B:11:62:7F:4C:88:DB:17:0B:84:66:B2:E5:90:CB:D2:B5:8C:80:AE

Do you trust the server www.botvrij.eu public certificate? [y/n]: y
Fetching active feeds

Update summary
##############

[Expert@quantum-firewall:0]# ioc_feeds add --feed_name ips --transport https --resource "https://sslbl.abuse.ch/blacklist/sslipblacklist.csv" --format [type:ip,value:#2] --comment [#] --delimiter ","
start add
Default value for active is: true
Default value for feed_action is: prevent
Feed ips will add on

Feed Name: ips
Feed is Active
File will be fetched via HTTPS
Resource: https://sslbl.abuse.ch/blacklist/sslipblacklist.csv
Action: Prevent
Feed is cli managed

'proxy'
SHA256 Fingerprint=16:8F:8D:D7:CD:C1:1D:AF:CB:85:54:79:20:09:42:29:29:2C:AA:BA:13:9E:34:AC:4E:20:EE:CE:4B:0E:9E:50

Do you trust the server sslbl.abuse.ch public certificate? [y/n]: y
Fetching active feeds

Update summary
##############

[Expert@quantum-firewall:0]# ioc_feeds add --feed_name ip_blacklist --transport https --resource "https://www.talosintelligence.com/documents/ip-blacklist" --format [type:ip,value:#1]
start add
Default value for active is: true
Default value for feed_action is: prevent
Feed ip_blacklist will add on

Feed Name: ip_blacklist
Feed is Active
File will be fetched via HTTPS
Resource: https://www.talosintelligence.com/documents/ip-blacklist
Action: Prevent
Feed is cli managed

'proxy'
SHA256 Fingerprint=64:BF:71:2E:6F:DA:8D:6A:37:24:8F:44:57:91:38:2E:E8:14:A3:E3:4E:32:18:9C:B5:B3:DE:83:80:D4:C9:2B

Do you trust the server www.talosintelligence.com public certificate? [y/n]: y
Fetching active feeds

Update summary
##############

[Expert@quantum-firewall:0]# ioc_feeds add --feed_name spam_list --transport https --resource "https://www.ipspamlist.com/public_feeds.csv" --format [type:ip,value:#3,comment:#4] --comment ["#", "first_seen"] --delimiter ","
start add
Default value for active is: true
Default value for feed_action is: prevent
Feed spam_list will add on

Feed Name: spam_list
Feed is Active
File will be fetched via HTTPS
Resource: https://www.ipspamlist.com/public_feeds.csv
Action: Prevent
Feed is cli managed

'proxy'
SHA256 Fingerprint=80:43:D6:EC:5E:8F:A6:E6:00:E2:A4:E0:55:96:9D:16:43:89:35:A9:11:B7:5D:4C:17:65:9B:DD:36:79:9B:2B

Do you trust the server www.ipspamlist.com public certificate? [y/n]: y
Fetching active feeds

Update summary
##############

[Expert@quantum-firewall:0]# ioc_feeds add --feed_name hash_list --transport http --resource "http://cybercrime-tracker.net/ccamlist.php" --format [type:sha1,value:#1]
start add
HTTP url feed transport is insecure and not recommended. Please consider using only url feeds with HTTPS transport.
Default value for active is: true
Default value for feed_action is: prevent
Feed hash_list will add on

Feed Name: hash_list
Feed is Active
File will be fetched via HTTP
Resource: http://cybercrime-tracker.net/ccamlist.php
Action: Prevent
Feed is cli managed

Fetching active feeds

Update summary
##############

[PhoneBoy]

ioc_feeds requires AB/AV blades to be enabled.
It works fine in a VM (that's how I tested it).

[the_rock]

So yours is on VM but with av/ab blades enabled?

Andy

[PhoneBoy]

Yes

[the_rock]

Got it, now it makes sense...tx for confirming! 👍

[SubZer0]

Is the 2 million limit referring to the number of characters or the number of lines?

[PhoneBoy]

In R81.10 and earlier, IoCs use a piece of infrastructure that has a hard character limit, which limited the number of IoCs you could import. 
In R81.20, the infrastructure used doesn’t have that limitation.
We’ve tested at least 2 million IOCs (IPs/Domains/Hashes) and I assume the limits are primarily based on memory.