Generic Data Center limitations

[Jonathan]

Hi,

I'm looking into the Generic Data Center object as a solution for automatically blocking IPs, but I can't find enough information on it's limitations and best practices. We using R81.10 with JHF take 66

Specifically -

- is there any hard limit on number of IPs/ranges/networks in a JSON file or per object in a JSON file?

- Is there any performance impact on having to much IPs in a json file, and if so what's the best practice?

- Is 50,000 or more IPs a realistic number for this mechanism?

- Is there any performance impact on setting the interval to 10 seconds instead of the default 60 seconds?

- Is there a way to add an exclusion in the JSON file or just by adding another rule above it?

- How do I generate a UID and make sure it's not already in use in the DB?

Thanks

[Chris_Atkinson]

The following limitations are outlined in the SK linked below.

Limitations  

• Policy Installation failed when maximum number dynamic objects was reached (The default value is 5000): refer to sk167576.
• We recommend that you avoid more than 30,000 changes per update as it will affect user experience.

Source: sk167210: Generic Data Center feature

Note: R81.20 (sk173903) introduced two relevant enhancements as follows

- Increased scalability of ioc_feeds (2 million and more observables*)

- Introduced new "network feed" objects.

[Jonathan]

Hi Chris,

I believe  sk167576 refers to overall dynamic objects in the policy, not numbers of IP addresses inside the JSON.

Also, the 30,000 recommendation refers to the changes between JSON sample intervals, not total number of IP address.

Regarding the IOC_feeds - does the Generic Data Center Object considered an IOC_Feed?

[Sorin_Gogean]

Hey,

Those are the limitations I have noted in our OneNote in regards to Generic DataCenter Objects.

The one that we made sure our scripts are checking is the 15K IPs's per JSON change. 

(my biggest file is ~200KB and has 8500lines and it summ's up the following:

NAME[1]="TOR_Exit_Nodes"

NAME[2]="TOR_IPBlacklist_Nodes"

NAME[3]="OpenDBL_TOR_Exit_Nodes"

NAME[4]="OpenDBL_TALOS_Nodes"

NAME[5]="OpenDBL_EmergThreats_Nodes"

NAME[6]="OpenDL_Bruteforce_Nodes"

) 

Also smth that I didn't see it specified in other documents, is that the JSON objects that you have defined in the file, needs to HAVE the same ID (that is an UUID) in order for the update to function. Otherwise the old imported object shows invaid/deleted - try and confirm please 😊 .

Limitations  

• Policy Installation failed when maximum number dynamic objects was reached (The default value is 5000): refer to sk167576.
• We recommend that you avoid more than 30,000 changes per update as it will affect user experience.

From <https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...>

Limitations

• You can make up to 15,000 changes in a JSON file between two time intervals at which the JSON file is sampled.

• The number of generic data center objects + dynamic objects + updatable objects in all policy packages cannot exceed 2,000.

From <https://sc1.checkpoint.com/documents/R81/WebAdminGuides/EN/CP_R81_SecurityManagement_AdminGuide/Topi...>

Thank you,

[Sorin_Gogean]

PS: as I didn't answer all your questions 😊 .

So, our approach, was NOT TO HAVE BIG files, as that needs to be loaded in a SHORT time, therefore I think you can split them - you know better.  

Changing the update/check time to 10sec, is NoK, I would prefer 60sec, as it takes couple of second to load those files and also update them over to the other GW's where they are used. So think about that.

I'm not sure about JSON inclusion, what we're doing is to generate the JSON on the fly .

In order to keep the same ID (UUID) on the objects, we've adapted the script, that uses JQ to look for the names and grabs the UID , so we're making sure we use the same ...

#grab and set the .id
   UUID=$(jq '.objects[] | select(.name == "'${NAME[$z]}'") | .id' $JSON_FILEBK)

As for the case we're hitting and UID that already exists in CheckPoint DB, I did not faced that yet, but we've set an SmartEvent alert if we're seeing some JSON related error messages in the 'blade:"CloudGuard IaaS"' .

Hope it helps..

[Jonathan]

Hi Sorin,

Thanks for the detailed answer.

Regarding the JSON inclusion, what I ment was IP exceptions. Something like Palo Alto has in it's EDL mechanism.

Meaning, somehow mark specific IPs, so that even if they are added to the JSON file they will be ignored.

[Sorin_Gogean · Accepted solution]

Hey @Jonathan ,

So, I really don't see the problem here, you just need to format your JSON with two or more objects like (adjusted from sk167210😞

{
      "version": "1.0",     
      "description": "Generic Data Center file example",
      "objects": [
                          {
                               "name": "GOOD Object A name",
                               "id": "e7f18b60-f22d-4f42-8dc2-050490ecf6d5",
                               "description": "Example for IPv4 addresses",
                               "ranges": [
                                                     "91.198.174.192",
                                                     "20.0.0.0/24",                        
                                                     "10.1.1.2-10.1.1.10"
                               ]              
                          },
                          {
                                "name": "BAD Object B name",
                                "id": "a46f02e6-af56-48d2-8bfb-f9e8738f2bd0",
                                "description": "Example for IPv6 addresses",
                                "ranges": [
                                                     "2001:0db8:85a3:0000:0000:8a2e:0370:7334",
                                                     "0064:ff9b:0000:0000:0000:0000:1234:5678/96",
                                                     "2001:0db8:85a3:0000:0000:8a2e:2020:0-2001:0db8:85a3:0000:0000:8a2e:2020:5"                                        
                                ]
                          }
     ]
}

So with the "GOOD Object A name" you can do whatever GOOD permissions 😁, while with the "BAD Object B name" you block them or whatever. 

Now, if by "somehow mark specific IPs, so that even if they are added to the JSON file they will be ignored" you meant the other way, like to take action when the JSON is generated, for that you will have to perform it from the scripting/programming end where the JSON is generated.  We did smth like that as we're generating some JSON with BAD IP addresses (like SCAN, MALICIOUS, etc.) that our SIEM sees (and not only the SIEM), and just as a failsafe; we build a function that gets each IP address (or network) and compares it with the WHITELIST IPs and networks that we have defined, and if that is matching, it's not adding the IP or the network to the JSON (in order to be blocked) but it's notifying us in order to be aware that a IP of ours was added on the list that we use to generate the JSON. Hopefully it's understandable, if not let me know and I'll try to explain better. 
(Our JSON is with IP's or networks that we BLOCK)

Thank you,

[Jonathan]

Hi Sorin,

So yeh, I actualy meant what you referred to in the second half of your answer, using scripting,

We use automation based on SIEM logs to auto block malicious IPs but sometime we need to intentionally exclude some IPs. 

Like you suggested, we'll manage a TXT file containing the IP we want to exclude and we'll have a script that will compare the malicious IPs from the SIEM with that file and if there's a match they won't be inserted to the JSON file.

Thanks for all the help!

[Sorin_Gogean]

Hello @Jonathan ,

Pretty similar with our implementation, but the way we did it was that from SIEM we ingest the "malicious" IPs into an DB from where we generate the JSON file to feed CKP Generic DataCenter object.

Let us know how it goes.

PS: on the comparison, pay attention if you're whitelist IPs are ranges or networks, to have a function that will properly split them 😁. We were using some network library's in python or in VBStudio that made it easier.
PS2: what about Corelated events from CheckPoint SmartEvents blade, or are you not using that blade ?

[Jonathan]

We currently don't have the SmartEvent blade.

[Jonathan]

Hi Sorin,

From your and Chris's replies, I understand that if my interval is 10 seconds for example, I can't add/delete/change more than 15,000 changes to the JSON file until the next 10 seconds. 

I still don't see any formal refference to max number of IP addresses per JSON file/JSON object (physical limit or performance limit).

Also, any thoughts about my other questions?

[Sorin_Gogean]

Like I said before, you can create different test files, with 10K lines (can be an IP or ranges), with 30K lines and with 55K lines .

(just choose some numbers so you can do a proper check)

Then load those new files into GDC objects, set them to 10sec or so, and see what you get in logs. 

Like I said earlier, I was looking for the FULL file load to be performed in under 2 to 5 sec, so it would be acceptable, and for that we agreed that for us 10-15K lines are OK. (we'll see in the future how it's going) .

(I'll come back with proper screenshots.... - sorry)

Thank you, 

PS: I've seen that the "CheckPoint Updatable Objects" are checked every 30sec, so I guess that's a recommended refresh timer...