Create objects for Azure Data-Center IP ranges - Python script

[Uri_Bialik]

Overview

This script generate group objects with the IP addresses of Microsoft Azure.

Note:

R80.20 has built-in functionality for addressing Azure's public IP addresses.
There new functionality allows you to use Azure, AWS and Office365 objects in your security policy using the GUI.

There's no need to use scripts like this one and the updates happens automatically (no need to publish  policy).

Description

Download Microsoft's Azure Datacenter IP ranges from: https://www.microsoft.com/en-us/download/details.aspx?id=41653

Run the attached Python script (the script does not have to run on the management server).

Provide the script with: the management's server IP address, username, password, the path for the downloaded file from Microsoft.

The script will now generate:

• Over 3000 networks (for example: azure_network_104.208.0.0/19)
• about 30 Group objects, one for each Azure region (for example: azure_region_useast)
• and a group object called azure_region_all - a group object that contain all the group region objects.

When you get an updated file from Microsoft, you can run the tool again. When running for the second time the script will work much faster: instead of creating thousands of objects, it will only process the changes.

Instructions

Download the attached zip and extract it on any machine with Python 2.7.x.

run:

python azure.py

Code Version

Code version 1.2

Tested on version

R80.10, API version 1.1

NOTICE: By using this sample code you agree to terms and conditions in this Terms and Conditions

...

[Raj_Khatri]

This script is working great on our Multi-Domain environment.  Out of curiosity, is there a python script available to do the same job, but for AWS? AWS Public IP Address Ranges Now Available in JSON Form | AWS Blog

[Todd_Larson]

First, thank you for taking the time to create the code to create all these objects.  I am experiencing an issue getting the objects to appear in the Security Management Server (not MDM).  Had a few issues first, then remembered I had to enable access to the API from the  GUI clients.  I then run the script and everything appears to complete, no errors in the script but no objects appear in SMS.

- There is an audit event showing the account login, Super User Role, has Management API permission.  No events for object creation.

- One item I did change, I disable requirement for session ID and description < running test now.

? Where would I look for some type of event log to determine what is occurring?

[Todd_Larson]

Update:

• IF SMS/MDM require a session name/description the script completes but you have to manually publish the changes after the script completes.
• Logs/results are in api_calls.json file
• How can I change the color of the network object and group during the next script execution?  Looks like it's possible using set-if-exists, not certain of syntax though.
  • I attempted to modify the line below, 
  • res = client.api_call("add-network", {"name":network_name, "subnet": networks[network_name]["subnet"], "mask-length": networks[network_name]["mask-length"]})

[Todd_Larson]

Uri Bialik‌ - I'm trying to run the azure.py script on the Security Management server.  To do this, I created a  directory in home/admin/scripts and unpacked the files there.  I then set azure.py to execute.  Below is the result of that effort.  Are scripts like this not intended to run from an SMS or MDM?  Would the next step be to place the missing module in a subdirectory ?

[Expert@dayprdchkptmgt01:0]# $FWDIR/Python/bin/python2.7 azure.py

Traceback (most recent call last):

  File "azure.py", line 15, in <module>

    from mgmt_api_lib import cp_mgmt_api

  File "/home/admin/scripts/azure_ip/python_bundle_1_2_2/mgmt_api_lib/__init__.py", line 1, in <module>

    from cp_mgmt_api import APIClient

  File "/home/admin/scripts/azure_ip/python_bundle_1_2_2/mgmt_api_lib/cp_mgmt_api.py", line 21, in <module>

    from distutils.version import LooseVersion

[Raj_Khatri]

Uri, I have seen the script for creating the Azure and AWS networks via script.  Is there an equivalent for creating the Office 365 networks?

Office 365 URLs and IP address ranges - Office 365 

[Bernhard_Sayer]

Is anyone able to adapt this script for creating Office365-Objects?

[Richard_Cove]

There is a hotfix being developed that will dynamically update O365 without running a script, it may be released soon

Talk to your local CP SE or support if you want to try it (internal sk119562)

[Bob_Ramsdell]

I am attempting to run this script against the Global domain.  The output of the script indicates objects and groups are being published, but the objects never appear in the Global network objects.  This works as advertised in other domains.

Has anyone encounter this issue?

[Robert_Decker]

If SMS/MDM requires a session name/description, the script completes but you have to manually publish the changes after the script completes.

Robert.

[Bob_Ramsdell]

I removed the requirement for session name/description and the objects were created as expected!

When running the script with the session name/description enabled, the 'Publish' button is greyed out.  Is manual publish something that needs to be done on the using the cli/web API?

[Robert_Decker]

there are commands that also do the publish operation, like assign-global-policy.

regular CRUD commands require publish.

Robert.

[Preecha_Sitthib]

Found the same issue here on SMS R80.10, has this one been addressed?

[Rick_Rodrix]

Has anyone tried it on r77.30?

[Usman_Shaikh]

The script works fine; however I have one minor (rather cosmetic issue).. Inline with our naming convention, I would like to add objects using the name N-AZE-<IPSubnet>_m<mask-length> e.g. N-AZE-104.1.1.0_m23 and also the groups to be G-AZURE-PREFIXES. Also how do I change the color of these objects to be Cyan

I am not familiar with Python so any help here would be greatly appreciated

[Orlando_de_Bien]

I can't get this script to create the objects.

I am running python from my windows 10 pc running python 2.7
i run the script and enter fields.

This is the output i see.

Saving data to debug file api_calls.json

Traceback (most recent call last):
File "azure.py", line 78, in <module>
login_res = client.login(username=username, password=password, domain=domain_name)
File "c:\inetpub\ftproot\azure\lib\mgmt_api.py", line 154, in login
login_res = self.api_call("login", credentials)
File "c:\inetpub\ftproot\azure\lib\mgmt_api.py", line 225, in api_call
self.check_fingerprint()
File "c:\inetpub\ftproot\azure\lib\mgmt_api.py", line 522, in check_fingerprint
server_fingerprint = self.get_server_fingerprint()
File "c:\inetpub\ftproot\azure\lib\mgmt_api.py", line 414, in get_server_fingerprint
context = ssl.create_default_context()
AttributeError: 'module' object has no attribute 'create_default_context'

the api_calls.json file only has []

Nothing is created in my manager.

Thanks

[Orlando_de_Bien]

I was able to get this working. I ran the script from a suse12 box and the script ran perfect without any changes.

My best bet is it will not run from windows, use linux.

[Ismail_Shaik]

Hello, I  ran python script in MDM R80.10 and the result showed as publishing 2300 out of 2332 however i am not able to see those groups from Smart console 

does anyone experienced this ?

Any helpful suggestions please.

Thanks in advance,

[Oscar_Figueruel]

Dear Uri,

First, I would like to thanks you for this post, it's very usefull for us, especially for the customers that don't have yet the R80.20 running on the MGMT and GWs.

I ran the script against our MDM R80.10 however we were not able to see all the "azure" networks and then some of the azure_region groups are not populated either.

Looking on the API.ELG Files, I found these two lines (taking 1 subnet and 1 group as an example)

*********************************   Creation of the azure network   *********************************

ID: 3188

Address: http://127.0.0.1:50276/web_api/v1.1/add-network

Encoding: ISO-8859-1

Http-Method: POST

Content-Type: application/json

Headers: {Accept=[*/*], accept-encoding=[identity], connection=[keep-alive], Content-Length=[86], content-type=[application/json], Host=[127.0.0.1:50276], User-Agent=[python-api-wrapper], X-chkp-sid=[************], X-Forwarded-For=[************], X-Forwarded-Host=[************], X-Forwarded-Host-Port=[443], X-Forwarded-Server=[************]}

Payload: {"subnet": "40.86.192.0", "name": "azure_network_40.86.192.0/18", "mask-length": "18"}

--------------------------------------

2019-08-01 08:53:12,721  INFO com.checkpoint.management.web_api_is.utils.helpers.ApiCache.:13 [qtp101599496-11543] - Cache created and initialized

2019-08-01 08:53:12,721  INFO com.checkpoint.management.web_api.web_services.WebApiEntryPoint.logRequestedCommandInfo:54 [qtp101599496-11543] - Executing [add-network] of version 1.1

2019-08-01 08:53:12,761 ERROR com.checkpoint.management.web_api.utils.WebApiCommandExceptionUtils.getErrorReply:110 [qtp101599496-11543] -

com.checkpoint.web_services.faults.ValidationRemoteFault: A blocking validation error was found: More than one network have the same IP 40.86.192.0/255.255.192.0

                at sun.reflect.GeneratedConstructorAccessor463.newInstance(Unknown Source)

                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57)

                at java.lang.reflect.Constructor.newInstance(Constructor.java:437)

                at org.apache.cxf.interceptor.ClientFaultConverter.processFaultDetail(ClientFaultConverter.java:182)

                at org.apache.cxf.interceptor.ClientFaultConverter.handleMessage(ClientFaultConverter.java:82)

                at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)

                at org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:113)

**************** Addition of a subnet on the corresponding Region Group: ************

----------------------------

ID: 4335

Address: http://127.0.0.1:50276/web_api/v1.1/set-group

Encoding: ISO-8859-1

Http-Method: POST

Content-Type: application/json

Headers: {Accept=[*/*], accept-encoding=[identity], connection=[keep-alive], Content-Length=[1089], content-type=[application/json], Host=[127.0.0.1:50276], User-Agent=[python-api-wrapper], X-chkp-sid=[********], X-Forwarded-For=[********], X-Forwarded-Host=[********], X-Forwarded-Host-Port=[443], X-Forwarded-Server=[********]}

Payload: {"name": "azure_region_canadaeast", "members": ["azure_network_40.86.192.0/18", "azure_network_40.90.147.32/27", "azure_network_104.44.93.64/27", "azure_network_40.90.138.64/27", "azure_network_52.108.232.0/23", "azure_network_52.245.32.0/22", "azure_network_52.139.64.0/18", "azure_network_40.80.40.0/22", "azure_network_52.114.164.0/22", "azure_network_13.104.154.128/25", "azure_network_40.90.17.128/28", "azure_network_40.89.0.0/19", "azure_network_40.90.130.192/28", "azure_network_52.235.0.0/18", "azure_network_52.239.164.128/26", "azure_network_52.109.96.0/22", "azure_network_52.229.64.0/18", "azure_network_20.38.121.128/25", "azure_network_40.79.217.0/24", "azure_network_52.232.128.0/21", "azure_network_40.126.11.128/25", "azure_network_52.136.22.0/24", "azure_network_40.90.156.96/27", "azure_network_40.90.140.0/27", "azure_network_40.80.240.0/20", "azure_network_40.69.96.0/19", "azure_network_20.190.139.128/25", "azure_network_52.242.0.0/18", "azure_network_52.155.0.0/19", "azure_network_104.44.95.32/28", "azure_network_20.150.1.0/25", "azure_network_52.239.190.0/25"]}

--------------------------------------

2019-08-01 09:07:45,671  INFO com.checkpoint.management.web_api_is.utils.helpers.ApiCache.:13 [qtp101599496-12497] - Cache created and initialized

2019-08-01 09:07:45,672  INFO com.checkpoint.management.web_api.web_services.WebApiEntryPoint.logRequestedCommandInfo:13 [qtp101599496-12497] - Executing [set-group] of version 1.1 (references 1)

2019-08-01 09:07:46,122  WARN com.checkpoint.management.web_api_is.utils.RemoteDetailsLevelUtils.getApiReplyObjectsByCpmObjects_aroundBody4:105 [qtp101599496-11677] - GatewayNonSupportedException was caught for object: [8c9abce5-c960-428d-a57a-0f50f2078cb8]

2019-08-01 09:07:46,223 ERROR com.checkpoint.management.web_api_is.exceptions.WebApiGeneralException.log:42 [qtp101599496-12497] - Error code: [GENERIC_ERR_OBJECT_NOT_FOUND]

2019-08-01 09:07:46,224 ERROR com.checkpoint.management.web_api_is.exceptions.WebApiGeneralException.log:43 [qtp101599496-12497] - Error message: [Requested object [azure_network_40.86.192.0/18] not found]

2019-08-01 09:07:46,224 ERROR com.checkpoint.management.web_api.utils.WebApiCommandExceptionUtils.getErrorReply:110 [qtp101599496-12497] -

com.checkpoint.management.web_api_is.exceptions.WebApiObjectNotFoundException

Looks like the issue is caused because there are many subnets which already exists on our MDM DB but with different name.

*** My question ***

Is there any way/parameter to force the creation of the duplicate entries trough the API ?   

I have tested modifying the azure-py script. including the "ignore-warnings": true at the end of line 112,

-------------------------

for network_name in networks:
i=i+1
if network_name not in existing_networks:
#print str(i) + " " + network_name
res = client.api_call("add-network", {"name":network_name, "subnet": networks[network_name]["subnet"], "mask-length": networks[network_name]["mask-length"]}, "ignore-warnings": true)
if i % 50 == 0:
print "publishing " + str(i) + " out of " + str(len(networks))
res = client.api_call("publish",{})

print "Publishing last networks"
res = client.api_call("publish",{})

But I get an error saying that this parameter is not valid.

Traceback (most recent call last):
File "azure.py", line 112, in <module>
res = client.api_call("add-network", {"name":network_name, "subnet": networks[network_name]["subnet"], "mask-length": networks[network_name]["mask-length"], "ignore-warnings": true })
NameError: name 'true' is not defined
[Expert@HQEBGM0081:0]#
[Expert@HQEBGM0081:0]#
[Expert@HQEBGM0081:0]#
[Expert@HQEBGM0081:0]# vi azure.py
[Expert@HQEBGM0081:0]# /opt/CPsuite-R80/fw1/Python/bin/python azure.py
File "azure.py", line 112
res = client.api_call("add-network", {"name":network_name, "subnet": networks[network_name]["subnet"], "mask-length": networks[network_name]["mask-length"], "ignore-warnings" true })
^

thanks so much in advance!

Oscar

[Raj_Khatri]

Since the old 'Microsoft Azure Datacenter IP Ranges' list is now been deprecated, with the last file being 2020-08-24 (PublicIPs_20200824.xml), will the new 'Azure IP address ranges' page which uses json format, work with the existing Python script?

We still have quite a few SMB gateways which run 77.20 and don't support R80.20+.