Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted

Re: Python tool for exporting/importing a policy package or parts of it

Yes it is, when exporting access layers it will export object related to it. There are other tools to simply export/import network objects..

0 Kudos
Highlighted
Copper

Re: Python tool for exporting/importing a policy package or parts of it

Hi All,

I am facing issue while running this package. Kindly guide me how to run the script. 

We have a customer with MDS environment. we need to migrate policy package from one CMA to another hosted on same MDS. Before running it in production environment. I need to test this in lab. I placed the complete package in tmp folder in particular CMA:

/opt/CPmds-R80/customers/CMA_POD_1_Server/CPsuite-R80/fw1/tmp/export/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master

-rw-r----- 1 admin root 11558 Sep 26 2017 LICENSE
-rw-r----- 1 admin root 2357 Sep 26 2017 README.md
-rw-r----- 1 admin root 14 Sep 26 2017 __init__.py
drwxr-xr-x 5 admin root 4096 Dec 31 21:37 cp_mgmt_api_python_sdk
drwxr-xr-x 2 admin root 4096 Dec 31 21:37 exporting
-rwxrwxrwx 1 admin root 2161 Sep 26 2017 import_export_package.py
drwxr-xr-x 2 admin root 4096 Dec 31 21:37 importing
-rw-r----- 1 admin root 12885 Sep 26 2017 lists_and_dictionaries.py
-rw-r----- 1 admin root 12052 Sep 26 2017 menu.py
-rw-r----- 1 admin root 22461 Sep 26 2017 utils.py

When i run import_export_package.py script from the directory where the package is it through me error:

[Expert@MDS:0]# ./import_export_package.py
./import_export_package.py: line 1: from: command not found
: command not foundkage.py: line 2:
./import_export_package.py: line 3: import: command not found
: command not foundkage.py: line 4:
./import_export_package.py: line 5: from: command not found
./import_export_package.py: line 6: from: command not found
./import_export_package.py: line 7: from: command not found
./import_export_package.py: line 8: from: command not found
: command not foundkage.py: line 9:
./import_export_package.py: line 10: debug: command not found
./import_export_package.py: line 11: log_file: command not found
./import_export_package.py: line 12: output_file: command not found
./import_export_package.py: line 13: client: command not found
: command not foundkage.py: line 14:
./import_export_package.py: line 15: from: command not found
: command not foundkage.py: line 16:
./import_export_package.py: line 19: syntax error near unexpected token `('
'/import_export_package.py: line 19: ` arg_parser = argparse.ArgumentParser(description="R80.X Policy Package Export/Import Tool, V3.0")

run another command:

[Expert@MDS:0]# ./opt/CPsuite-R80/fw1/Python/bin/python2.7 /home/admin//home/admin/ExportImportPolicyPackage-master/import_export_package.py -m 192.168.155.5
-bash: ./opt/CPsuite-R80/fw1/Python/bin/python2.7: No such file or directory

I think i am not following the correct command to run the script. 

Please guide.

Thanks

0 Kudos
Highlighted
Admin
Admin

Re: Python tool for exporting/importing a policy package or parts of it

The dot as the first character of the command is definitely wrong unless you are currently in / (which you're probably not).

You're also including /home/admin twice.

Try running as $MDS_FWDIR/Python/bin/python2.7 /home/admin/ExportImportPolicyPackage-master/import_export_package.py -m 192.168.155.5

0 Kudos
Highlighted

Re: Python tool for exporting/importing a policy package or parts of it

If you have global policies assigned on CMA you will not be able to successfully export that policy package. This tool doesn’t support this. It’s mentioned in documentation.


0 Kudos
Highlighted
Copper

Re: Python tool for exporting/importing a policy package or parts of it

Thanks Dameon for guiding me. I didn't notice i was including /home/admin twice.

I tested now with the command you provided and it worked. But when i am importing the package. i can see gateway object is also included. is there any way we can exclude gateway object.

Further, i created manual hide nat but it didn't came into new CMA when i imported the policy package.

Kindly advise.

Thanks

0 Kudos
Highlighted
Copper

Re: Python tool for exporting/importing a policy package or parts of it

Hi Martin,

Global Policies are not applied at the moment.

0 Kudos
Highlighted
Admin
Admin

Re: Python tool for exporting/importing a policy package or parts of it

The script will export the policy and any object required to support the policy.

This includes the gateway, but as I recall the gateway is a placeholder object you can safely remove after the fact (after you resolve any dependecies on said object).

As far as I know, NAT rules should come over with his script,

Was it just one NAT rule that didn't come over, all NAT rules?

0 Kudos
Highlighted
Copper

Re: Python tool for exporting/importing a policy package or parts of it

Hi Dameon,

In my lab I created only 1 manual nat rule but it didn't come over. I tried automatic nat rule as well but same result. customer has around 70 nat rules doing automatic NAT.

Thanks

0 Kudos
Highlighted
Admin
Admin

Re: Python tool for exporting/importing a policy package or parts of it

NAT rules should get exported (both automatic and manual ones).

Anything in the output of running the script that might give a clue?

0 Kudos
Highlighted
Copper

Re: Python tool for exporting/importing a policy package or parts of it

Hi Dameon,

NAT rules are exported completely fine but failing when i import policy package. i get below error:

Adding nat-rules

Failed to import nat-rule. Error: Requested object [hello] not found

Failed to import nat-rule. Error: Requested object [hello] not found

Failed to import nat-rule. Error: Requested object [hello] not found

Failed to import nat-rule. Error: Requested object [hello] not found

Failed to import nat-rule. Error: Requested object [hello] not found

where hello is name of policy package on existing CMA.

Rules screenshot:

0 Kudos
Highlighted
Admin
Admin

Re: Python tool for exporting/importing a policy package or parts of it

Did the object hello successfully import to the target system?

If it did not, neither did the automatic NAT rule, which is stored in the object. 

Can you please output the object in question from the original CMA?

Output should look like below.

Maybe Robert Decker‌ has an idea what happened. 

[Expert@DEMO:0]# mgmt_cli -r true show host name hello

uid: "33e57b6f-7b34-41f7-9c01-44b80a6b1f5d"
name: "hello"
type: "host"
domain:
uid: "41e821a0-3720-11e3-aa6e-0800200c9fde"
name: "SMC User"
domain-type: "domain"
ipv4-address: "1.1.1.1"
interfaces: []
nat-settings:
auto-rule: true
ipv4-address: "2.2.2.2"
ipv6-address: ""
install-on: "All"
method: "static"
groups: []
comments: ""
color: "black"
icon: "Objects/host"
tags: []
meta-info:
lock: "unlocked"
validation-state: "ok"
last-modify-time:
posix: 1546409680479
iso-8601: "2019-01-02T06:14+0000"
last-modifier: "admin"
creation-time:
posix: 1546409628160
iso-8601: "2019-01-02T06:13+0000"
creator: "admin"
read-only: false
0 Kudos
Highlighted
Copper

Re: Python tool for exporting/importing a policy package or parts of it

Hi Dameon,

There is no object hello in original CMA.

output:

[Expert@MDS:0]# mgmt_cli -r true show host name hello
code: "generic_err_object_not_found"
message: "Requested object [hello] not found"

0 Kudos
Highlighted

Re: Python tool for exporting/importing a policy package or parts of it

Hi Ankur Datta‌ , 

Since you wrote 'hello' is a policy package , the result of 'object not found' is clear (for the execution of show host name hello commad)

If i understand right, you didn't get any errors for the export part, am i right?

Can you please share 'import_export.log' (should reside in same folder where 'import_export_package.py' exist)

In addition, i would like to get API logs for the 'import' failure.

Please connect to Management Server via SSH and execute the following 'api status -s'.

Then send the output (.tgz file) to my mail amiads@checkpoint.com 

Regards,

Amiad.

0 Kudos
Highlighted
Copper

Re: Python tool for exporting/importing a policy package or parts of it

Hi Amid,

I again tried and automatic NAT rules can be imported but not manual NAT rules they are still missing.

There was no error in export part:

Exporting Access Control layers

Exporting Access Layer [CPPFMS Network]

Retrieved 5 out of 5 rules (100%)

Processing rules and sections

Exporting hosts from layer [CPPFMS Network]

Exporting groups from layer [CPPFMS Network]

Exporting hosts from group [Hosts]

Exporting simple-gateways from layer [CPPFMS Network]

Exporting access rules from layer [CPPFMS Network]

Exporting access sections from layer [CPPFMS Network]

Exporting placeholders for unexportable objects from layer [CPPFMS Network]

Exporting layer settings of layer [CPPFMS Network]

Done exporting layer 'CPPFMS Network'.


Exporting Access Layer [application]

Retrieved 1 out of 1 rules (100%)

Processing rules and sections

Exporting hosts from layer [application]

Exporting access rules from layer [application]

Exporting access sections from layer [application]

Exporting placeholders for unexportable objects from layer [application]

Exporting layer settings of layer [application]

Done exporting layer 'application'.


Exporting NAT policy

Getting information from show-nat-rulebase

Retrieved 16 out of 16 rules (100%)

Processing rules and sections

Exporting address_ranges

Exporting hosts

Exporting networks

Exporting simple-gateways

Exporting NAT rules

Exporting placeholders for unexportable objects from NAT rulebase

Done exporting NAT rulebase.

I am sending you the google drive link. i uploaded the required files on it.

Thanks.

0 Kudos
Highlighted

Re: Python tool for exporting/importing a policy package or parts of it

Robert Decker‌ - I'm looking to move a layer (MDS R80.10 JHF 154) from one CMA to another CMA.  I have the same global policy assigned to both CMAs.  I know it says global policy assignment isn't supported, but it isn't feasible unassigned global policy.  If I don't use global objects on the layer I'm wanting to export, will this method work?

Highlighted

Re: Python tool for exporting/importing a policy package or parts of it

Hi Brian, we will check the exact limitation on Sunday and will update on this thread. 

0 Kudos
Highlighted
Copper

Re: Python tool for exporting/importing a policy package or parts of it

Hi, I tested on customer backup in lab environment but export completed with errors below, The export didn't said done exporting NAT rulebase whereas i got done for access and application. 

Export logs:

Traceback (most recent call last):
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/import_export_package.py", line 45, in <module>
export_package(client, args)
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/exporting/export_package.py", line 59, in export_package
nat_data_dict, nat_unexportable_objects = export_nat_rulebase(show_package.data["name"], client)
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/exporting/export_nat_rulebase.py", line 13, in export_nat_rulebase
rulebase_rules, general_objects = get_query_nat_rulebase_data(client, {"package": package})
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/exporting/export_objects.py", line 187, in get_query_nat_rulebase_data
rulebase_items[len(rulebase_items) - 1]["rulebase"].extend(non_empty_rulebase_items[0]["rulebase"])
KeyError: 'rulebase'

Import failed:

Creating a Policy Package named [****]

Importing general objects

Traceback (most recent call last):
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/import_export_package.py", line 47, in <module>
import_package(client, args)
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/importing/import_package.py", line 52, in import_package
layers_to_attach = import_objects(args.file, client, {})
File "/home/admin/ExportImportPolicyPackage-master/ExportImportPolicyPackage-master/importing/import_objects.py", line 19, in import_objects
export_tar = tarfile.open(file_name, "r:gz")
File "/opt/CPsuite-R80/fw1/Python/lib/python2.7/tarfile.py", line 1693, in open
return func(name, filemode, fileobj, **kwargs)
File "/opt/CPsuite-R80/fw1/Python/lib/python2.7/tarfile.py", line 1751, in gzopen
raise ReadError("not a gzip file")
tarfile.ReadError: not a gzip file

I gave the same name as the policy package name is in both the output file during export and then in import.

Kindly advise.

0 Kudos
Highlighted
Copper

Re: Python tool for exporting/importing a policy package or parts of it

Any suggestions please.

Thanks

0 Kudos
Highlighted

Re: Python tool for exporting/importing a policy package or parts of it

Hi Ankur Datta‌,

Following our session, we managed to make the process work on your lab. I'm sharing the information here.

When you are asked to enter Management Server IP you need to set it to Multi Domain IP and not domain IP. 

See image below:

Comment: we will open Jira task to improve phrasing so it will be clearer

As for your issue on production, as discussed, now that we managed to make it work in your lab, do the same in production, if it fails, send by mail these log files:

1. 'import_export.log' (reside in same folder where 'import_export_package.py' exist)

2. output of api status -s

Regards,

Amiad.

0 Kudos
Highlighted
Copper

Re: Python tool for exporting/importing a policy package or parts of it

I'd like to thank Robert Decker and everyone else who has spent time on this, I found it immeasurably helpful. At the same time the process I used this in should be MUCH easier.

I needed to take two R77.30 2200 gateways that operate separately and upgrade them to R80.10. Unfortunately I don't have the budget to just buy two new management servers for both sites and the 2200's have to be distributed so I needed to add these gateways into my central R80.10 SMS server. I also wanted to have the firewall configuration ready on an SMS before wiping the standalone appliances and installing R80.10 fresh. The process I went through for two of my gateways recently was as such:


  • Grab migrate export of first gateway using R77.30 migration tools
  • Import that into a new R77.30 SMS VM in our core after editing the .tgz with the sk85900 to mark it as a mgmt only device 
  • cp_merge the database of the other standalone device
  • Fix inconsistencies with import and merge
  • Update R77.30 VM to R80.10 with CPUSE offline clish method since you can't use Gaia web for this when you're on a trial version disconnected from the internet 
  • Update to the same hotfix version as my production SMS
  • Use the exportimportpolicypackage tool to export both policies
  • Import the policies into the production SMS
  • Fix inconsistencies from export and import

I still need to:

  • Wipe the standalone boxes
  • Install R80.10
  • Get connected to the HQ SMS server for configuration
  • Add back in any OS or Gaia settings like DHCP or NTP using clish configuration
  • Add back in any custom kern.conf or crypto settings through expert

Pain Points:

  • Gaia ISO default partitioning saved 69% of the 100 GB vm disk I made for the R77.30 SMS for the upgrade partition causing me constantly failures with imports and exports due to filling up the /var/log and root partitions. I recreated the vm with 25 GB partitions for /var/log and root to resolve this issue.
  • The fact migrate export will allow you to delete db_revisions from both the production device and from the export but you can't choose to exclude them just from the export. I didn't want to get rid of revisions which I may need before upgrading the gateway to R80.10. For this reason I had a 6 GB export with most of it being revisions. People also said to untar the export and remove these but I found no instructions on how to do this properly. 
  • Inability to use Gaia Web CPUSE, especially annoying since the R77.30 to R80.10 upgrade rebooted then kept telling me the upgrade was still in progress and to check the Gaia Web CPUSE status page which I was not allowed access to, I had to keep SSH'ing in hoping it wouldn't say in progress anymore and using the CPUSE offline clish tool to see the package status.
  • exportimportpolicypackage JSON error I had to fix both on the exporting SMS and the importing SMS by allowing the management API from all IP's
  • exportimportpolicypackage authentication error which seemed to be due to the fact I had SmartConsole open with the same credentials (I don't see this listed in the instructions, or whether you need to use Security Admin/Regular clish Admin/or a /bin/bash admin for this tool).
  • The fact every vpn community, legacy user, gateway, LDAP AU, and other settings imported with errors.
  • Having to setup python on my windows machine for this tool instead of being able to run it off the SMS itself.

This whole process has taken about 4+ days not including the several hours I will need per appliance to finalize the upgrade. I'd be willing to help anyone trying to complete this same task or to take advice from the community on how I could've done this better.

0 Kudos
Highlighted

Re: Python tool for exporting/importing a policy package or parts of it

Hey all

It is supposed to work with r80.20 database too right?

There are a list of items that are not supported therefore they won't be part of the archive to import?

Thanks

0 Kudos
Highlighted
Admin
Admin

Re: Python tool for exporting/importing a policy package or parts of it

Yes, it should.

In a previous comment to this thread, I had posted some limitations of this script.

As this has come up more than once and it might be difficult to find said comment, I will add to the top-level post.

0 Kudos
Highlighted

Re: Python tool for exporting/importing a policy package or parts of it

thanks , appreciate it

0 Kudos
Highlighted

Re: Python tool for exporting/importing a policy package or parts of it

Hi Dor. Did you fix the issue? I am suffering exactly the same

0 Kudos
Highlighted
Copper

Re: Python tool for exporting/importing a policy package or parts of it

The export/import tool messed up the import of all the gateways and VPN communities. Even after recreating the necessary objects and deleting the objects with errors I still have Implied rules based on the VPN Communities with import errors that I can't delete. Now I have implied rules dropping traffic incorrectly and I feel it might be related to the corrupted implied rules, is there anything I can do?

 

The SK where I was told my configuration was unsupported it: 6-0001478291

0 Kudos
Highlighted

Re: Python tool for exporting/importing a policy package or parts of it

Hello guys, i need help with the migration of policies, from a management 80.10 to a multidomain 80.10, the export goes well, also the import of the first policy goes ok, but when i try to import a second policy the process fails after failing to import several objects that were previously imported on the first policy package(i guess is ok those errors cause the object already exist), have someone encounter this issue before?

this is the output of the terminal when fails

Traceback (most recent call last):
File "C:\xxxxxxxxxxr\import_export_package.py", line 47, in <module>
import_package(client, args)
File "C:\xxxxxxxxxx\import_package.py", line 52, in import_package
layers_to_attach = import_objects(args.file, client, {})
File "C:\xxxxxxxxxx\import_objects.py", line 78, in import_objects
changed_layer_names, api_call, num_objects, client)
File "C:\xxxxxxxxxx\import_objects.py", line 221, in add_object
"name"] + "]" if "name" in payload else "", error_msg)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 24: ordinal not in range(128)

regards

Highlighted
Silver

Re: Python tool for exporting/importing a policy package or parts of it

Dear all,

Normally this script is okay, but today we've encountered session timeout proble, due to large policy package(over 7000 rules):

Is there anyway to extend session timeout setting or send keepalive like web api does?

0 Kudos
Highlighted
Admin
Admin

Re: Python tool for exporting/importing a policy package or parts of it

The script would have to be modified to request a longer timeout with the login API call.

The default is 600 seconds (10 minutes).

Highlighted
Silver

Re: Python tool for exporting/importing a policy package or parts of it

Yes, that's what I meant, now I'm trying with the following modification:

0 Kudos
Highlighted
Silver

Re: Python tool for exporting/importing a policy package or parts of it

Still saw so many objects failed to import, but not sure about the root cause:

Any Ideas?

0 Kudos