Python tool for exporting/importing a policy package or parts of it

[Inbar_Moskovich]

Overview

ExportImportPolicyPackage tool enables you to export a policy package from a R80.x management database to a .tar.gz file, which can then be imported into any other R8x management database.

This tool can be used for backups, database transfers, testing, and more.

In the case you are exporting a policy package from a CMA, please verify that a global policy was NOT assigned to that CMA.
The tool doesn't support exporting a policy with global policy assigned!

The tool is referenced in https://support.checkpoint.com/results/sk/sk180923 

Description

This tool enables you to export a policy package (Access Policy, Threat Policy or both) from a management server into a .tar.gz file.

Notice

There are some types of objects that the script might not be able to export.
In such a case, an appropriate dummy object will be exported instead, and a message will be logged into the log files to notify you of this.
In the Check Point SmartConsole you can easily replace each of these objects by searching "export_error" in the search field, see where each object is used, create the necessary object manually, then replace it.

Instructions

Download the latest version from our GitHub repository: https://github.com/CheckPointSW/ExportImportPolicyPackage 
First, make sure you have [2.7.9 <= Python <= 2.7.14] installed on the machine running the script.
To export a package, run the import_export_package.py script. An interactive menu will guide you the rest of the way.
Command line flags may also be set in order to skip some or all of the menu.
A lot more details can of course be accessed with the [-h] option. This option also prints the current version of the tool.

Current tool version is V3.0.

Limitations

This export/import script does not gather all data from a given management server/CMA.
In general, it is limited by the R80.x Management APIs.
Specifically, this means:

• CMAs with a Global Policy assigned cannot be exported
  • Workaround: unassign the Global Policy prior to export
• Gateway/Cluster objects have to be recreated
  • Placeholder objects will be created
• UserCheck messages have to be recreated
  • Placeholder objects will be created
• The Internal Certificate Authority will not be copied. This means:
  • Re-establishing SIC with the appropriate gateways
  • Re-generating VPN certificates
  • Manually recreating HTTPS Inspection and DLP Rules
• Other objects not currently readable/writable via the R80.x API will not be copied

Tested on version

R8x
Releases earlier than R80 lack the necessary API support and are not supported.

Source Code Availability

The source code is available through GitHub: https://github.com/CheckPointSW/ExportImportPolicyPackage 

FAQ

Replies to this thread have locked.
Please refer to the FAQ below before you create a new post with your question.

When I run this tool, I get the message: APIResponse received a response which is not a valid JSON.

This most likely means you haven't enabled the API server yet.
See: https://community.checkpoint.com/t5/API-CLI-Discussion-and-Samples/Enabling-web-api/m-p/32641

I get an error message related to server fingerprint

Use the --unsafe option to ignore this error.

Can this tool export more than one policy package at a time?

Not currently, but you could call the tool in a script multiple times.

[Martin_Valenta]

Yes it is, when exporting access layers it will export object related to it. There are other tools to simply export/import network objects..

[Ankur_Datta]

Hi All,

I am facing issue while running this package. Kindly guide me how to run the script. 

We have a customer with MDS environment. we need to migrate policy package from one CMA to another hosted on same MDS. Before running it in production environment. I need to test this in lab. I placed the complete package in tmp folder in particular CMA:

/opt/CPmds-R80/customers/CMA_POD_1_Server/CPsuite-R80/fw1/tmp/export/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master

-rw-r----- 1 admin root 11558 Sep 26 2017 LICENSE
-rw-r----- 1 admin root 2357 Sep 26 2017 README.md
-rw-r----- 1 admin root 14 Sep 26 2017 __init__.py
drwxr-xr-x 5 admin root 4096 Dec 31 21:37 cp_mgmt_api_python_sdk
drwxr-xr-x 2 admin root 4096 Dec 31 21:37 exporting
-rwxrwxrwx 1 admin root 2161 Sep 26 2017 import_export_package.py
drwxr-xr-x 2 admin root 4096 Dec 31 21:37 importing
-rw-r----- 1 admin root 12885 Sep 26 2017 lists_and_dictionaries.py
-rw-r----- 1 admin root 12052 Sep 26 2017 menu.py
-rw-r----- 1 admin root 22461 Sep 26 2017 utils.py

When i run import_export_package.py script from the directory where the package is it through me error:

[Expert@MDS:0]# ./import_export_package.py
./import_export_package.py: line 1: from: command not found
: command not foundkage.py: line 2:
./import_export_package.py: line 3: import: command not found
: command not foundkage.py: line 4:
./import_export_package.py: line 5: from: command not found
./import_export_package.py: line 6: from: command not found
./import_export_package.py: line 7: from: command not found
./import_export_package.py: line 8: from: command not found
: command not foundkage.py: line 9:
./import_export_package.py: line 10: debug: command not found
./import_export_package.py: line 11: log_file: command not found
./import_export_package.py: line 12: output_file: command not found
./import_export_package.py: line 13: client: command not found
: command not foundkage.py: line 14:
./import_export_package.py: line 15: from: command not found
: command not foundkage.py: line 16:
./import_export_package.py: line 19: syntax error near unexpected token `('
'/import_export_package.py: line 19: ` arg_parser = argparse.ArgumentParser(description="R80.X Policy Package Export/Import Tool, V3.0")

run another command:

[Expert@MDS:0]# ./opt/CPsuite-R80/fw1/Python/bin/python2.7 /home/admin//home/admin/ExportImportPolicyPackage-master/import_export_package.py -m 192.168.155.5
-bash: ./opt/CPsuite-R80/fw1/Python/bin/python2.7: No such file or directory

I think i am not following the correct command to run the script. 

Please guide.

Thanks

[PhoneBoy]

The dot as the first character of the command is definitely wrong unless you are currently in / (which you're probably not).

You're also including /home/admin twice.

Try running as $MDS_FWDIR/Python/bin/python2.7 /home/admin/ExportImportPolicyPackage-master/import_export_package.py -m 192.168.155.5

[Martin_Valenta]

If you have global policies assigned on CMA you will not be able to successfully export that policy package. This tool doesn’t support this. It’s mentioned in documentation.

[Ankur_Datta]

Thanks Dameon for guiding me. I didn't notice i was including /home/admin twice.

I tested now with the command you provided and it worked. But when i am importing the package. i can see gateway object is also included. is there any way we can exclude gateway object.

Further, i created manual hide nat but it didn't came into new CMA when i imported the policy package.

Kindly advise.

Thanks

[Ankur_Datta]

Hi Martin,

Global Policies are not applied at the moment.

[PhoneBoy]

The script will export the policy and any object required to support the policy.

This includes the gateway, but as I recall the gateway is a placeholder object you can safely remove after the fact (after you resolve any dependecies on said object).

As far as I know, NAT rules should come over with his script,

Was it just one NAT rule that didn't come over, all NAT rules?

[Ankur_Datta]

Hi Dameon,

In my lab I created only 1 manual nat rule but it didn't come over. I tried automatic nat rule as well but same result. customer has around 70 nat rules doing automatic NAT.

Thanks

[PhoneBoy]

NAT rules should get exported (both automatic and manual ones).

Anything in the output of running the script that might give a clue?

[Ankur_Datta]

Hi Dameon,

NAT rules are exported completely fine but failing when i import policy package. i get below error:

Adding nat-rules

Failed to import nat-rule. Error: Requested object [hello] not found

Failed to import nat-rule. Error: Requested object [hello] not found

Failed to import nat-rule. Error: Requested object [hello] not found

Failed to import nat-rule. Error: Requested object [hello] not found

Failed to import nat-rule. Error: Requested object [hello] not found

where hello is name of policy package on existing CMA.

Rules screenshot:

[PhoneBoy]

Did the object hello successfully import to the target system?

If it did not, neither did the automatic NAT rule, which is stored in the object. 

Can you please output the object in question from the original CMA?

Output should look like below.

Maybe Robert Decker‌ has an idea what happened. 

[Expert@DEMO:0]# mgmt_cli -r true show host name hello
uid: "33e57b6f-7b34-41f7-9c01-44b80a6b1f5d"
name: "hello"
type: "host"
domain: 
 uid: "41e821a0-3720-11e3-aa6e-0800200c9fde"
 name: "SMC User"
 domain-type: "domain"
ipv4-address: "1.1.1.1"
interfaces: []
nat-settings: 
 auto-rule: true
 ipv4-address: "2.2.2.2"
 ipv6-address: ""
 install-on: "All"
 method: "static"
groups: []
comments: ""
color: "black"
icon: "Objects/host"
tags: []
meta-info: 
 lock: "unlocked"
 validation-state: "ok"
 last-modify-time: 
 posix: 1546409680479
 iso-8601: "2019-01-02T06:14+0000"
 last-modifier: "admin"
 creation-time: 
 posix: 1546409628160
 iso-8601: "2019-01-02T06:13+0000"
 creator: "admin"
read-only: false

[Ankur_Datta]

Hi Dameon,

There is no object hello in original CMA.

output:

[Expert@MDS:0]# mgmt_cli -r true show host name hello
code: "generic_err_object_not_found"
message: "Requested object [hello] not found"

[Amiad_Stern]

Hi Ankur Datta‌ , 

Since you wrote 'hello' is a policy package , the result of 'object not found' is clear (for the execution of show host name hello commad)

If i understand right, you didn't get any errors for the export part, am i right?

Can you please share 'import_export.log' (should reside in same folder where 'import_export_package.py' exist)

In addition, i would like to get API logs for the 'import' failure.

Please connect to Management Server via SSH and execute the following 'api status -s'.

Then send the output (.tgz file) to my mail amiads@checkpoint.com 

Regards,

Amiad.

[Ankur_Datta]

Hi Amid,

I again tried and automatic NAT rules can be imported but not manual NAT rules they are still missing.

There was no error in export part:

Exporting Access Control layers

Exporting Access Layer [CPPFMS Network]

Retrieved 5 out of 5 rules (100%)

Processing rules and sections

Exporting hosts from layer [CPPFMS Network]

Exporting groups from layer [CPPFMS Network]

Exporting hosts from group [Hosts]

Exporting simple-gateways from layer [CPPFMS Network]

Exporting access rules from layer [CPPFMS Network]

Exporting access sections from layer [CPPFMS Network]

Exporting placeholders for unexportable objects from layer [CPPFMS Network]

Exporting layer settings of layer [CPPFMS Network]

Done exporting layer 'CPPFMS Network'.

Exporting Access Layer [application]

Retrieved 1 out of 1 rules (100%)

Processing rules and sections

Exporting hosts from layer [application]

Exporting access rules from layer [application]

Exporting access sections from layer [application]

Exporting placeholders for unexportable objects from layer [application]

Exporting layer settings of layer [application]

Done exporting layer 'application'.

Exporting NAT policy

Getting information from show-nat-rulebase

Retrieved 16 out of 16 rules (100%)

Processing rules and sections

Exporting address_ranges

Exporting hosts

Exporting networks

Exporting simple-gateways

Exporting NAT rules

Exporting placeholders for unexportable objects from NAT rulebase

Done exporting NAT rulebase.

I am sending you the google drive link. i uploaded the required files on it.

Thanks.

[Brian_Deutmeyer]

Robert Decker‌ - I'm looking to move a layer (MDS R80.10 JHF 154) from one CMA to another CMA.  I have the same global policy assigned to both CMAs.  I know it says global policy assignment isn't supported, but it isn't feasible unassigned global policy.  If I don't use global objects on the layer I'm wanting to export, will this method work?

[Amiad_Stern]

Hi Brian, we will check the exact limitation on Sunday and will update on this thread. 

[Ankur_Datta]

Hi, I tested on customer backup in lab environment but export completed with errors below, The export didn't said done exporting NAT rulebase whereas i got done for access and application. 

Export logs:

Traceback (most recent call last):
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/import_export_package.py", line 45, in <module>
export_package(client, args)
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/exporting/export_package.py", line 59, in export_package
nat_data_dict, nat_unexportable_objects = export_nat_rulebase(show_package.data["name"], client)
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/exporting/export_nat_rulebase.py", line 13, in export_nat_rulebase
rulebase_rules, general_objects = get_query_nat_rulebase_data(client, {"package": package})
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/exporting/export_objects.py", line 187, in get_query_nat_rulebase_data
rulebase_items[len(rulebase_items) - 1]["rulebase"].extend(non_empty_rulebase_items[0]["rulebase"])
KeyError: 'rulebase'

Import failed:

Creating a Policy Package named [****]

Importing general objects

Traceback (most recent call last):
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/import_export_package.py", line 47, in <module>
import_package(client, args)
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/importing/import_package.py", line 52, in import_package
layers_to_attach = import_objects(args.file, client, {})
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/importing/import_objects.py", line 19, in import_objects
export_tar = tarfile.open(file_name, "r:gz")
File "/opt/CPsuite-R80/fw1/Python/lib/python2.7/tarfile.py", line 1693, in open
return func(name, filemode, fileobj, **kwargs)
File "/opt/CPsuite-R80/fw1/Python/lib/python2.7/tarfile.py", line 1751, in gzopen
raise ReadError("not a gzip file")
tarfile.ReadError: not a gzip file

I gave the same name as the policy package name is in both the output file during export and then in import.

Kindly advise.

[Ankur_Datta]

Any suggestions please.

Thanks

[Amiad_Stern]

Hi Ankur Datta‌,

Following our session, we managed to make the process work on your lab. I'm sharing the information here.

When you are asked to enter Management Server IP you need to set it to Multi Domain IP and not domain IP. 

See image below:

Comment: we will open Jira task to improve phrasing so it will be clearer

As for your issue on production, as discussed, now that we managed to make it work in your lab, do the same in production, if it fails, send by mail these log files:

1. 'import_export.log' (reside in same folder where 'import_export_package.py' exist)

2. output of api status -s

Regards,

Amiad.

[Aidan_Luby]

I'd like to thank Robert Decker and everyone else who has spent time on this, I found it immeasurably helpful. At the same time the process I used this in should be MUCH easier.

I needed to take two R77.30 2200 gateways that operate separately and upgrade them to R80.10. Unfortunately I don't have the budget to just buy two new management servers for both sites and the 2200's have to be distributed so I needed to add these gateways into my central R80.10 SMS server. I also wanted to have the firewall configuration ready on an SMS before wiping the standalone appliances and installing R80.10 fresh. The process I went through for two of my gateways recently was as such:

• Grab migrate export of first gateway using R77.30 migration tools
• Import that into a new R77.30 SMS VM in our core after editing the .tgz with the sk85900 to mark it as a mgmt only device 
• cp_merge the database of the other standalone device
• Fix inconsistencies with import and merge
• Update R77.30 VM to R80.10 with CPUSE offline clish method since you can't use Gaia web for this when you're on a trial version disconnected from the internet 
• Update to the same hotfix version as my production SMS
• Use the exportimportpolicypackage tool to export both policies
• Import the policies into the production SMS
• Fix inconsistencies from export and import

I still need to:

• Wipe the standalone boxes
• Install R80.10
• Get connected to the HQ SMS server for configuration
• Add back in any OS or Gaia settings like DHCP or NTP using clish configuration
• Add back in any custom kern.conf or crypto settings through expert

Pain Points:

• Gaia ISO default partitioning saved 69% of the 100 GB vm disk I made for the R77.30 SMS for the upgrade partition causing me constantly failures with imports and exports due to filling up the /var/log and root partitions. I recreated the vm with 25 GB partitions for /var/log and root to resolve this issue.
• The fact migrate export will allow you to delete db_revisions from both the production device and from the export but you can't choose to exclude them just from the export. I didn't want to get rid of revisions which I may need before upgrading the gateway to R80.10. For this reason I had a 6 GB export with most of it being revisions. People also said to untar the export and remove these but I found no instructions on how to do this properly. 
• Inability to use Gaia Web CPUSE, especially annoying since the R77.30 to R80.10 upgrade rebooted then kept telling me the upgrade was still in progress and to check the Gaia Web CPUSE status page which I was not allowed access to, I had to keep SSH'ing in hoping it wouldn't say in progress anymore and using the CPUSE offline clish tool to see the package status.
• exportimportpolicypackage JSON error I had to fix both on the exporting SMS and the importing SMS by allowing the management API from all IP's
• exportimportpolicypackage authentication error which seemed to be due to the fact I had SmartConsole open with the same credentials (I don't see this listed in the instructions, or whether you need to use Security Admin/Regular clish Admin/or a /bin/bash admin for this tool).
• The fact every vpn community, legacy user, gateway, LDAP AU, and other settings imported with errors.
• Having to setup python on my windows machine for this tool instead of being able to run it off the SMS itself.

This whole process has taken about 4+ days not including the several hours I will need per appliance to finalize the upgrade. I'd be willing to help anyone trying to complete this same task or to take advice from the community on how I could've done this better.

[Marco_Valenti]

Hey all

It is supposed to work with r80.20 database too right?

There are a list of items that are not supported therefore they won't be part of the archive to import?

Thanks

[PhoneBoy]

Yes, it should.

In a previous comment to this thread, I had posted some limitations of this script.

As this has come up more than once and it might be difficult to find said comment, I will add to the top-level post.

[Marco_Valenti]

thanks , appreciate it

[Francisco_Jose1]

Hi Dor. Did you fix the issue? I am suffering exactly the same

[Aidan_Luby]

The export/import tool messed up the import of all the gateways and VPN communities. Even after recreating the necessary objects and deleting the objects with errors I still have Implied rules based on the VPN Communities with import errors that I can't delete. Now I have implied rules dropping traffic incorrectly and I feel it might be related to the corrupted implied rules, is there anything I can do?

The SK where I was told my configuration was unsupported it: 6-0001478291

[Sergio_lopez]

Hello guys, i need help with the migration of policies, from a management 80.10 to a multidomain 80.10, the export goes well, also the import of the first policy goes ok, but when i try to import a second policy the process fails after failing to import several objects that were previously imported on the first policy package(i guess is ok those errors cause the object already exist), have someone encounter this issue before?

this is the output of the terminal when fails

Traceback (most recent call last):
File "C:\xxxxxxxxxxr\import_export_package.py", line 47, in <module>
import_package(client, args)
File "C:\xxxxxxxxxx\import_package.py", line 52, in import_package
layers_to_attach = import_objects(args.file, client, {})
File "C:\xxxxxxxxxx\import_objects.py", line 78, in import_objects
changed_layer_names, api_call, num_objects, client)
File "C:\xxxxxxxxxx\import_objects.py", line 221, in add_object
"name"] + "]" if "name" in payload else "", error_msg)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 24: ordinal not in range(128)

regards

[Neville_Kuo]

Dear all,

Normally this script is okay, but today we've encountered session timeout proble, due to large policy package(over 7000 rules):

Is there anyway to extend session timeout setting or send keepalive like web api does?

[PhoneBoy]

The script would have to be modified to request a longer timeout with the login API call.

The default is 600 seconds (10 minutes).

[Neville_Kuo]

Yes, that's what I meant, now I'm trying with the following modification:

[Neville_Kuo]

Still saw so many objects failed to import, but not sure about the root cause:

Any Ideas?