The IP quantity limit of a NetworkFeed

[TONYFU]

Hello,
the documentation states that a network feed:

1. A Security Gateway supports up to 500 network feed objects. Each object can hold up to 50,000 IP addresses. There is no limitation on the number of domains per object.

2. A Security Gateway supports a total of 5,000 objects of these types: Dynamic objects, Updatable objects, Generic Data Center objects, and Network Feed objects. A Security Gateway supports a total of 350,000 IP addresses and 12,500 domains across all of these object types combined.

In my Lab , I can see approximately 140,000 IP entries in the object "IPSUM1".
[Expert@CPSG:0]# dynamic_objects -efo IPSUM1 | wc -l
140842
[Expert@CPSG:0]# dynamic_objects -efo IPSUM1 | tail
range 140830 : 223.255.153.194 223.255.153.194
range 140831 : 223.255.163.249 223.255.163.249
range 140832 : 223.255.177.204 223.255.177.204
range 140833 : 223.255.183.10 223.255.183.10
range 140834 : 223.255.183.18 223.255.183.18


My original understanding was that seeing 140,000 entries was because of rule 2, but due to rule 1, only 50,000 IP could be processed.
However, practical testing seems to show that it is not actually the case.

I applied the network feed object to a policy and tested whether the firewall could block the 60,000th IP.
However, the firewall was still able to block it correctly.
[Expert@CPSG:0]# dynamic_objects -efo IPSUM1 | grep 60000
range 60000 : 103.86.1.22 103.86.1.22

So what exactly does the 50,000 IP in rule1 mentioned here refer to?
Thank you!

[PhoneBoy · Accepted solution]

How are the IPs listed in this file, individually, or in ranges?
Because I think the 50,000 refers to the number of items that refer directly to IPs or networks (by IP/mask or range).
The number of IPs can be much larger, I believe.

[the_rock · Accepted solution]

Interesting you ask, because when I spoke to TAC about it few months ago, guy told me would check internally and advised that documentation was actually wrong, that there was no limit. I believe that is 100% true, since I tested feed with 15M entries, no issues.

I was hoping documentation would be corrected, but not yet.

Best,

Andy

[TONYFU]

Thank you!

[the_rock]

Glad we can help!

[the_rock]

For the context, Fortinet feeds are limited to 32000 entries, so really noticeable difference.

Andy

[C0rwin]

What about domains limit for network feed?

BR

Daniel

[PhoneBoy]

The various limits for Network Feeds are in the documentation: https://sc1.checkpoint.com/documents/R82/WebAdminGuides/EN/CP_R82_SecurityManagement_AdminGuide/Cont...
For domains specifically, it states: "There is no limitation on the number of domains per object."

[C0rwin]

But there is also statement -  A Security Gateway supports a total of 350,000 IP Addresses and 12,500 domains across all of these object types combined.

So is it possible and supported to use network feed with domains up to the mentioned limit? Any experience with that.

I'm asking about it - because I found on cluster R82 JHF Take_60 huge issue with network feed containing 150k domains - cpu spikes, failovers, lack of responsivness during policy install or feed refresh time.

BR

Daniel

[PhoneBoy]

You're right, maybe we can fix that contradiction. @Sergei_Shir 

Sounds like you're exceeding the limit of 12,500 domains by a factor of 10, which might explain the behavior.
TAC might be able to suggest an adjustment you an make to support that many. 

[C0rwin]

TAC already suggested and we moved public feed to internal one - divided it to many network feeds up to 10k, but it didn't solve the issue. Every update of single network feed with 10k domains is also causing cpu spikes, failovers, cpu soft lockup etc.

TAC suggested RFE but it seems that we need to move from network feed to external ioc feed in TP policy to make it work without impact.

What's a pitty that network feed has such huge limitation.

BR

Daniel.

[PhoneBoy]

The limits are "total per gateway" not "per feed" which explains why splitting the feeds up didn't work.

[C0rwin]

So it seems to be really poor mechanism for resolving dns in background written for network feed. I did similar test in demopoint and results are the same:

- network feed with about 150k domains = performance issues, rad errors in fwk.elg etc.

- external ioc feed with the same number of 150k domains working fine, even if by default there is 5 min refresh for feed

BR 

Daniel

[Ronit_Segal]

Please clarify, what is the required correction in the admin guide?

[PhoneBoy]

Under the "Use Cases" section, the second and third bullet points conflict with each other.
The second bullet point says "no limit" where the third bullet point clearly states there is (not in the object, but at the gateway level).
This is where the confusion lies.
I would remove the statement "There is no limitation on the number of domains per object" to make it more clear.