Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Simon_Macpherso
Advisor
Jump to solution

R81.20 TechTalk webinar - Network Feed Objects

Hello,

This is related Network Feeds object mentioned in the What's New in R81.20 TechTalk webinar this week.  

I have a few questions re the Network Feeds object. 

What file type is the file the Network Feeds object?

If there is no strict formatting;

  • How can you trust the data input is valid data?
  • Is there a built-in validation process to ensure the data is valid?
  • Also is there a constraints mechanism i.e. restrict what values can will be accepted ion the file e.g. a specific IP range?

We just started using generic data center objects block malicious IPs from verified threat intelligence feeds. As you stated, the generic data center object references a JSON file with strict formatting requirements. However, there is still no built-in protection for data validation. 

To mitigate input errors i.e. input data that doesn't conform to the strict formatting, we validate the JSON against a schema before copying the file to a web or the management server.  

In terms of scalability,  the JSON should be able to handle a lot of IPs. Can you explain the advantage of the new object in further detail here?  

I would be interested to look at any additional information you're able to provide on the Network Feeds object.  

@Tomer_Noy are you able to shed some more light on these objects?

Regards,

Simon    

0 Kudos
2 Solutions

Accepted Solutions
PhoneBoy
Admin
Admin

In addition to being JSON, for which you can specify a 'jq' query to pull out the precise fields you're interested in, a "flat file" type is supported.
You specify the precise formatting when you define the object:

image.png

The Data Type of the feed can be:

  • IP Address
  • Domain Name
  • IP Address or Domain Name

As for the benefit of this new method, there are a few:

  • Generic Datacenter objects use the CloudGuard Connector backend, which relies on the management server being active to feed the gateways. Network Feeds are fetched from the gateways directly.
  • Generic Datacenter objects only support IPs, Network Feeds also support domains.
  • Network Feeds should be significantly faster and more scalable than either Generic Datacenter objects or IOC Feeds in terms of how quickly they are read in and enforced on the gateway.

Hopefully I got everything that's pertinent 🙂

Edit: See also the official documentation on this feature
https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_SecurityManagement_AdminGuid... 

View solution in original post

Tomer_Noy
Employee
Employee

All of @PhoneBoy's comments are accurate.

In addition, to answer your remaining questions:

An invalid entry will not break the entire feed and will simply be skipped. I just tried it to verify 😀
(take a look at the attached pictures)

Also, if the feed is completely broken or inaccessible, the gateway will continue to use cached contents from the last successful fetch.

You can of course add your custom validations to some CI/CD pipeline before updating the feed files (whether JSON or flat files).

View solution in original post

18 Replies
PhoneBoy
Admin
Admin

In addition to being JSON, for which you can specify a 'jq' query to pull out the precise fields you're interested in, a "flat file" type is supported.
You specify the precise formatting when you define the object:

image.png

The Data Type of the feed can be:

  • IP Address
  • Domain Name
  • IP Address or Domain Name

As for the benefit of this new method, there are a few:

  • Generic Datacenter objects use the CloudGuard Connector backend, which relies on the management server being active to feed the gateways. Network Feeds are fetched from the gateways directly.
  • Generic Datacenter objects only support IPs, Network Feeds also support domains.
  • Network Feeds should be significantly faster and more scalable than either Generic Datacenter objects or IOC Feeds in terms of how quickly they are read in and enforced on the gateway.

Hopefully I got everything that's pertinent 🙂

Edit: See also the official documentation on this feature
https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_SecurityManagement_AdminGuid... 

Simon_Macpherso
Advisor

@PhoneBoy 

Thanks for the information. 

If flat file format is selected, do you know if any validation is done on the IP or domain values to ensure they are valid inputs? If not, and an invalid entry is added, does this break the entire network feed object?  

With JSON selection, if there is no built-in validation, we can perform our own validation against a JSON schema which we currently do.  

Regards,

Simon

0 Kudos
Tomer_Noy
Employee
Employee

All of @PhoneBoy's comments are accurate.

In addition, to answer your remaining questions:

An invalid entry will not break the entire feed and will simply be skipped. I just tried it to verify 😀
(take a look at the attached pictures)

Also, if the feed is completely broken or inaccessible, the gateway will continue to use cached contents from the last successful fetch.

You can of course add your custom validations to some CI/CD pipeline before updating the feed files (whether JSON or flat files).

Simon_Macpherso
Advisor

Thanks @Tomer_Noy 

Good to know the feed will still function in those scenarios. Does that apply to both JSON and flat file format types? 

Do you know if the the same applies to generic data center objects in R81.10?

0 Kudos
Tomer_Noy
Employee
Employee

Yes, it should apply to both flat file and JSON.

I don't have confirmation regarding the generic data center...

0 Kudos
Simon_Macpherso
Advisor

Hi @Tomer_Noy assuming the gateways needs to be on R81.20 in addition to the management server? 

0 Kudos
PhoneBoy
Admin
Admin

Unlike Generic Data Center objects, which is a management feature (leverages the CloudGuard Controller infrastructure), this is a gateway feature.
That means the gateways need to be on R81.20 or above.

Simon_Macpherso
Advisor

Thanks for confirming @PhoneBoy 

0 Kudos
_khard
Employee Alumnus
Employee Alumnus

Hi @PhoneBoy , What would happen if the external http/https website used to fetch the domains/Urls/IP-Add is unavailable ? 

Would Gateways use the last cached list ? 

0 Kudos
PhoneBoy
Admin
Admin

Yes, the last version of the list should be used.

0 Kudos
PhoneBoy
Admin
Admin

See also the official documentation on this feature now that R81.20 is GA: https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_SecurityManagement_AdminGuid... 

0 Kudos
Simon_Macpherso
Advisor

For Network Feeds objects, can standard JSON syntax be used when selecting JSON as the feed format? Or does it have to be in the JSON file structure used when creating generic data center objects per sk167210 (https://support.checkpoint.com/results/sk/sk167210)?

I am trying to parse demo.json using a JSON file using JSON query jq -r '.addresses[]' demo.json to extract  CIDR ranges. The JSON file is stored on an internal web server my that the gateway I am testing the feed against has access to. i.e. http://1.1.1.1/whitelisting/demo/demo.json 

The contents of demo.json are in standard JSON file structure as follows. 

{
"addresses": [
"23.235.32.0/20",
"43.249.72.0/22",
"103.244.50.0/24",
"103.245.222.0/23",
"103.245.224.0/24",
"104.156.80.0/20",
"151.101.0.0/16",
"157.52.64.0/18",
"172.111.64.0/18",
"185.31.16.0/22",
"199.27.72.0/21",
"199.232.0.0/16"
]
}
 
The test is failing returning 'Test failed because the JSON query execution finished with errors.'

 

0 Kudos
PhoneBoy
Admin
Admin

Standard JSON is expected here, so that SHOULD work...

0 Kudos
_khard
Employee Alumnus
Employee Alumnus

Hi @Simon_Macpherso , 

Try the below json 

[
{
"value": "23.235.32.0/20",
"comment": "Network Block A"
},
{
"value": "43.249.72.0/22",
"comment": "Network Block B"
}
]

 

I've published the same on https://akhrd.github.io/ipadd.json using github pages. 

Now from the smartconsole, if you put this query, it should work.

.[].value

 

Do sharing your observations. 

0 Kudos
Simon_Macpherso
Advisor

Hi @_khard 

Your query works using the JSON format you supplied. 

It turns out my query syntax was wrong - the 'jq' is not required in the JSON query field. If I use .addresses[] against the following format the query is successful. 

{

"addresses": [
"23.235.32.0/20",
"43.249.72.0/22"
]
}
 
Thanks for your response. 
Thomas_Eichelbu
Advisor
Advisor

Hello folks ...
one question, 
we just stumbled over this limitation on VSX.

https://support.checkpoint.com/results/sk/sk79700

Blade / Feature

R81.20

R81.10

R81

R80.40

R80.30SP

R80.30

R80.20SP

R80.20

External Network Feeds

No

No

No

No

No

No

No

No


external Network Feeds are not supported on VSX? YES/NO?
I must admit i have not tried it yet ... 
Anybody who has tried it and can make some statements about it?

What could be the technical reason why it is not "supported"/"running" on VSX?

best regards

Thomas_Eichelbu
Advisor
Advisor

just found this:
https://community.checkpoint.com/t5/General-Topics/Network-Feeds-and-VSX/td-p/212877

so Network Feeds will come for VSX with a future HFA ... lets hope for the best.

PhoneBoy
Admin
Admin

This has been discussed recently here: https://community.checkpoint.com/t5/General-Topics/Network-Feeds-and-VSX/m-p/213560/highlight/true#M... 
They will work with VSX provided you have a non-VSX gateway to perform the "test feed" option with on initial setup.
We plan to fix this limitation in R82 and in an upcoming R81.20 JHF.

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events