R81.20 TechTalk webinar - Network Feed Objects

[Simon_Macpherso]

Hello,

This is related Network Feeds object mentioned in the What's New in R81.20 TechTalk webinar this week.  

I have a few questions re the Network Feeds object. 

What file type is the file the Network Feeds object?

If there is no strict formatting;

• How can you trust the data input is valid data?
• Is there a built-in validation process to ensure the data is valid?
• Also is there a constraints mechanism i.e. restrict what values can will be accepted ion the file e.g. a specific IP range?

We just started using generic data center objects block malicious IPs from verified threat intelligence feeds. As you stated, the generic data center object references a JSON file with strict formatting requirements. However, there is still no built-in protection for data validation. 

To mitigate input errors i.e. input data that doesn't conform to the strict formatting, we validate the JSON against a schema before copying the file to a web or the management server.  

In terms of scalability,  the JSON should be able to handle a lot of IPs. Can you explain the advantage of the new object in further detail here?  

I would be interested to look at any additional information you're able to provide on the Network Feeds object.  

@Tomer_Noy are you able to shed some more light on these objects?

Regards,

Simon    

[PhoneBoy · Accepted solution]

In addition to being JSON, for which you can specify a 'jq' query to pull out the precise fields you're interested in, a "flat file" type is supported.
You specify the precise formatting when you define the object:

The Data Type of the feed can be:

• IP Address
• Domain Name
• IP Address or Domain Name

As for the benefit of this new method, there are a few:

• Generic Datacenter objects use the CloudGuard Connector backend, which relies on the management server being active to feed the gateways. Network Feeds are fetched from the gateways directly.
• Generic Datacenter objects only support IPs, Network Feeds also support domains.
• Network Feeds should be significantly faster and more scalable than either Generic Datacenter objects or IOC Feeds in terms of how quickly they are read in and enforced on the gateway.

Hopefully I got everything that's pertinent 🙂

Edit: See also the official documentation on this feature
https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_SecurityManagement_AdminGuid... 

[Simon_Macpherso]

@PhoneBoy 

Thanks for the information. 

If flat file format is selected, do you know if any validation is done on the IP or domain values to ensure they are valid inputs? If not, and an invalid entry is added, does this break the entire network feed object?  

With JSON selection, if there is no built-in validation, we can perform our own validation against a JSON schema which we currently do.  

Regards,

Simon

[Tomer_Noy · Accepted solution]

All of @PhoneBoy's comments are accurate.

In addition, to answer your remaining questions:

An invalid entry will not break the entire feed and will simply be skipped. I just tried it to verify 😀
(take a look at the attached pictures)

Also, if the feed is completely broken or inaccessible, the gateway will continue to use cached contents from the last successful fetch.

You can of course add your custom validations to some CI/CD pipeline before updating the feed files (whether JSON or flat files).

[Simon_Macpherso]

Thanks @Tomer_Noy 

Good to know the feed will still function in those scenarios. Does that apply to both JSON and flat file format types? 

Do you know if the the same applies to generic data center objects in R81.10?

[Tomer_Noy]

Yes, it should apply to both flat file and JSON.

I don't have confirmation regarding the generic data center...

[Simon_Macpherso]

Hi @Tomer_Noy assuming the gateways needs to be on R81.20 in addition to the management server? 

[PhoneBoy]

Unlike Generic Data Center objects, which is a management feature (leverages the CloudGuard Controller infrastructure), this is a gateway feature.
That means the gateways need to be on R81.20 or above.

[Simon_Macpherso]

Thanks for confirming @PhoneBoy 

[_khard]

Hi @PhoneBoy , What would happen if the external http/https website used to fetch the domains/Urls/IP-Add is unavailable ? 

Would Gateways use the last cached list ? 

[PhoneBoy]

Yes, the last version of the list should be used.

[PhoneBoy]

See also the official documentation on this feature now that R81.20 is GA: https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_SecurityManagement_AdminGuid... 

[Simon_Macpherso]

For Network Feeds objects, can standard JSON syntax be used when selecting JSON as the feed format? Or does it have to be in the JSON file structure used when creating generic data center objects per sk167210 (https://support.checkpoint.com/results/sk/sk167210)?

I am trying to parse demo.json using a JSON file using JSON query jq -r '.addresses[]' demo.json to extract  CIDR ranges. The JSON file is stored on an internal web server my that the gateway I am testing the feed against has access to. i.e. http://1.1.1.1/whitelisting/demo/demo.json 

The contents of demo.json are in standard JSON file structure as follows. 

{

"addresses": [

"23.235.32.0/20",

"43.249.72.0/22",

"103.244.50.0/24",

"103.245.222.0/23",

"103.245.224.0/24",

"104.156.80.0/20",

"151.101.0.0/16",

"157.52.64.0/18",

"172.111.64.0/18",

"185.31.16.0/22",

"199.27.72.0/21",

"199.232.0.0/16"

]

}

The test is failing returning 'Test failed because the JSON query execution finished with errors.'

[PhoneBoy]

Standard JSON is expected here, so that SHOULD work...

[_khard]

Hi @Simon_Macpherso , 

Try the below json 

[
{
"value": "23.235.32.0/20",
"comment": "Network Block A"
},
{
"value": "43.249.72.0/22",
"comment": "Network Block B"
}
]

I've published the same on https://akhrd.github.io/ipadd.json using github pages. 

Now from the smartconsole, if you put this query, it should work.

.[].value

Do sharing your observations. 

[Simon_Macpherso]

Hi @_khard 

Your query works using the JSON format you supplied. 

It turns out my query syntax was wrong - the 'jq' is not required in the JSON query field. If I use .addresses[] against the following format the query is successful. 

{

"addresses": [

"23.235.32.0/20",

"43.249.72.0/22"

]

}

Thanks for your response. 

[Thomas_Eichelbu]

Hello folks ...
one question, 
we just stumbled over this limitation on VSX.

https://support.checkpoint.com/results/sk/sk79700

Blade / Feature	R81.20	R81.10	R81	R80.40	R80.30SP	R80.30	R80.20SP	R80.20
External Network Feeds	No	No	No	No	No	No	No	No

external Network Feeds are not supported on VSX? YES/NO?
I must admit i have not tried it yet ... 
Anybody who has tried it and can make some statements about it?

What could be the technical reason why it is not "supported"/"running" on VSX?

best regards

[Thomas_Eichelbu]

just found this:
https://community.checkpoint.com/t5/General-Topics/Network-Feeds-and-VSX/td-p/212877

so Network Feeds will come for VSX with a future HFA ... lets hope for the best.

[PhoneBoy]

This has been discussed recently here: https://community.checkpoint.com/t5/General-Topics/Network-Feeds-and-VSX/m-p/213560/highlight/true#M... 
They will work with VSX provided you have a non-VSX gateway to perform the "test feed" option with on initial setup.
We plan to fix this limitation in R82 and in an upcoming R81.20 JHF.