CLI API Example for exporting, importing, and deleting different objects using CSV files

[Eric_Beasley]

Check out GITHUB for latest versions : GitHub - mybasementcloud/R8x-export-import-api-scripts: Check Point R8x Export, Import, [and more] A... (version 00.60.08.075 as of 2022-03-14)

The post below is not manageable and won't be updated any more, all specifics and improved documentation is provided on GitHub.

BR

Eric @Eric_Beasley 

Overview

The export, import, delete using CSV files scripts in this post, currently version 00.33.00 and later, dated 2019-01-19 and later, are intended to allow operations on an existing R80, R80.10, R80.20[.Mx], R80.30[.Mx] Check Point management server (SMS or MDM) from bash on the management server or a management server able authenticate and reach the target management server.

These scripts show examples of:

• an export of objects and services with full and standard json output
• an export of hosts, networks, groups, groups-with-exclusion, address-ranges, dns-domains, host interfaces, and group members to csv output [future services and other object dumps to csv is pending further research]
• an import of hosts, networks, groups, groups-with-exclusion, address-ranges, dns-domains, host interfaces, and group members from csv output generated by the export to csv operations above or custom csv files with valid objects.
• a set operation (change existing objects) for hosts, networks, groups, groups-with-exclusion, address-ranges, dns-domains, host interfaces, and group members from csv output generated by the export to csv operations above or custom csv files with valid objects.  (As of version 00.24.00).  Script added to specifically set values using CSV file for all objects for which a CSV file is found, specifically the:   "cli_api_set-update_objects_from_csv.sh"
• a script to delete groups-with-exclusion, groups, address-ranges, dns-domains, networks, hosts, using csv files created by an object name export to csv for the respective items deleted.  NOTE:  DANGER!, DANGER!, DANGER!  Use at own risk with extreme care!
• scripts provided to just count the objects the script could find in the environment (as of version 00.23.00), specifically the:  
  "cli_api_get_object_totals.sh" and "cli_api_get_object_totals_w_group_member_count.sh".
• script operations for export of more than 500 objects of a specific type.
• export complex object elements, like Group Members and Host Interfaces to CSV files for import via the --batch option.
• export of full JSON and CSV for only objects that are not created by "System", so exporting only custom added or administrator modified objects.
• MDM tool to export domains to domains_list.txt file for reference in other script calls

NOTE: 

• As of version 00.29.00 all cli_api*.sh scripts require "common" script folder to handle command line parameters!  Don't forget to copy this folder also.
• EXPORT SCRIPTS NOW HANDLE > 500 OBJECTS FOR ALL CSV  AND JSON EXPORTS (version 00.24.00)
• EXPORT SCRIPTS NOW DEFAULT TO EXPORTING ALL OBJECTS INSTEAD OF NON-SYSTEM OBJECTS FOR FULL JSON AND CSV OPERATION.  CLI COMMANDS ENABLE FULL DUMP OF ALL OBJECTS INSTEAD OF THOSE CREATED BY "System":  --SO and --NSO for show System Objects and No Show System Objects.  (version 00.29.02)
• AS OF VERSION 00.33.00, THE PORT FOR MANAGEMENT WEB SSL PORT IS IDENTIFIED AUTOMATICALLY IF EXECUTING AGAINST THE LOCAL MANAGEMENT HOST.
  

Where to Find, Contact, Support, and Terms of Service

Scripts are provided as-is without express or implied warranty, guarantee, assumption of liability, or SLA for resolution as they are examples of how to use the API and may not always apply to the situation they are used in and are subject to limitations of the API engine or utilized Check Point management version.

At no time is this post an implied assumption of duty to change the example scripts IAW requests.  If you need an API script developed, contact Check Point Professional Services or a qualified Check Point Channel Partner with DevOps capabilities.

CLI API Script approach may change at any time, which may require accessing older GitHub branches or releases.

CLI API Scripts are provided with stipulation that the implementor is capable of using CLI and understands API operations.

For direct questions, you can hit me up at ericb@checkpoint.com  Response time may vary based on schedule, availability, and issue presented.

For questions about issues with scripts please provide the information identified in the FEEDBACK section at the bottom of this article. 

Description

This post includes a set of scripts in two (2) packages, a Development Branch which may be an advanced version still under construction and an Operations Branch that should work as expected.  All script files end with .sh for shell and are intended for Check Point bash implementation on R80, R80.10, R80.20[.Mx], and R80.30[Mx] or later.  Scripts in the packages have specific purposes and scripts call sub-scripts for extensive repeated operations and basic actions (e.g. handling CLI parameters).  The packages also include specific expected default directory folders that are not created by the script action.

General Information:

<yyyy-mm-dd-hhmm-tz> is a date time group (DTG) generated at time of execution and used for the full operation of the respective script, providing consistent information for a specific script run.  Example:  2017-01-05-1346CST  for January 5, 2017, at 13:46 hrs CST.

Output Generated by Scripts:

Output from the scripts is directed to a sub-folder (default is a dump folder with DTG sub-folder, e.g. ./dump/<yyyy-mm-dd-hhmm-tz>) and further placed in sub-folders based on script:  csv, full, standard, import, delete. 

Example:

Multi-Domain Management operations scripts that handle multiple domains in their operation will create a sub-folder for the domain and then create the specific output based on that domain in the domain specific folder.  Note that the scripts automatically handle the default "System Data" and "Global" domains.  When executing the scripts with specific domain selected, that domain is the folder name between the DTG folder and the output folders.

Example:

NOTE:  Current CSV output includes additional files used in the process that are raw data in a WIP folder, sorted raw data, csv header, and the original data with header.

NOTE:  When operating on MDM, the stated domain in the -d parameter is used as a subfolder to collect the specific data for that dumped domain.

Scripts

Scripts a provided in an Operations and Development branch folder structure.  Operations branch implies that these are stable operational scripts for their stated purpose, and expected to work as such under the tested versions where applicable.  Development branch may include advanced, early availability version of the scripts where development is not yet complete for migration to Operations branch.

Main script types provided are MDM scripts, Session Cleanup scripts, common scripts used by other scripts, Export Import scripts, and script Templates.

Script Type (main folder)	Script Name	Script Purpose	Output formats
_templates	Templates for developing scripts with the basic capabilities of the provided version level of the template.
	api_mgmt_cli_shell_template_with_cmd_line_parameters.template.<version>.sh	Template for scripts using provided approach that includes built-in command line parameter handler operations.	Text log file Output dependent on chosen template implementation
	api_mgmt_cli_shell_template_with_cmd_line_parameters_script.template.<versions>.sh	Template for scripts using common scripts approach for handling command line parameters	Text log file Output dependent on chosen template implementation
	api_mgmt_cli_shell_template_action_handler.template.<version>.sh	Template for action sub-script called by another script, with basic handling for version mismatch	log to originating script based on verbose level setting
	test._templates.<version>.sh	Rough testing script for validating if templates function	Text log file Output dependent on chosen template implementation
common	Common scripts called by the scripts and utilized by the templates in the _template folder.  This folder and expected scripts are replicated to the utilizing script folder for direct access, so will be found under _templates, export_import, and Session_Cleanup
	cmd_line_parameters_handler.action.common.<level>.<version>.sh	Action sub-script called to execute operations to handle command line parameters standard to all scripts in that version	log to originating script based on verbose level setting
	identify_gaia_and_installation.action.common.<level>.<version>.sh	Action sub-script called to execute operations to identify version of Gaia of host and Check Point installation type	log to originating script based on verbose level setting
export_import	Scripts for export, import, set, and delete operations
	cli_api_export_objects.sh	Script to export all supported objects to JSON (full and standard) and CSV	JSON Full, JSON Standard, CSV Text log file
	cli_api_export_objects_to_json_full.sh	Script to export all supported objects to JSON full	JSON Full, JSON Standard, CSV Text log file
	cli_api_export_objects_to_json_standard.sh	Script to export all supported objects to JSON standard	JSON Full, JSON Standard, CSV Text log file
	cli_api_export_objects_actions.sh	Action sub-script to execute export of objects to JSON (full or standard) for calling script.	log to originating script based on verbose level setting
	cli_api_export_objects_actions_to_csv.sh	Action sub-script to execute export of objects to CSV (full or standard) for calling script.	log to originating script based on verbose level setting
	cli_api_export_objects_to_csv.sh	Script to export all supported objects to CSV	CSV Text log file
	cli_api_export_all_domains_objects.sh	Script to export all supported objects in all domains on an MDS to JSON (full and standard) and CSV	JSON Full, JSON Standard, CSV Text log file
	cli_api_export_all_domains_objects_to_csv.sh	Script to export all supported objects in all domains on an MDS to CSV	CSV Text log file
	cli_api_import_objects_from_csv.sh	Script to import all supported objects from supplied CSV files (only supplied files are processed)	JSON results Text log file
	cli_api_set-update_objects_from_csv.sh	Script to set (updated) all supported objects from supplied CSV files (only supplied files are processed)	JSON results Text log file
	cli_api_delete_objects_using_csv.sh	Script to delete all supported objects from supplied CSV files by name (only supplied files are processed)	JSON results Text log file
	cli_api_get_object_totals.sh	Dump the number of each object from all supported objects.	Text log file
	cli_api_get_object_totals_w_group_member_count.sh	Dump the number of each object from all supported objects, including count of group members per group.	Text log file
	api_add_csv_error_handling_to_csv_file.sh	For versions (e.g. R80.10) where CSV import and set operations require that ignore error, ignore warning, and set-if-exists are in the actual CSV file.  Adds necessary columns to front of existing CSV file rows with headers.	CSV Text log file
	api_subpend_csv_error_handling_to_csv_files.sh	For versions (e.g. R80.10) where CSV import and set operations require that ignore error, ignore warning, and set-if-exists are in the actual CSV file.  Adds necessary columns to end (back) of existing CSV file rows with headers.	CSV Text log file
MDM	Multi-Domain Management general scripts
	MDM_Get_Domains_List_<version>.sh	Script will generate a list of current domains on an MDS for manual re-use later	Text
Session_Cleanup	Session Cleanup scripts to first list the current sessions open on a management server and also provide options to clean-up (delete) session that are not locked or pending publish operations (zero locks).
	remove_zerolocks_sessions.<version>.sh	Identify and delete zero lock sessions	Text result of operation
	remove_zerolocks_web_api_sessions.<version>.sh	Identify and delete zero lock sessions with user web_api	Text result of operation
	show_zerolocks_sessions.<version>.sh	Identify zero lock sessions	Text result of operation
	show_zerolocks_web_api_sessions.<version>.sh	Identify zero lock sessions with user web_api	Text result of operation

 

Instructions

To utilize the scripts, download the scripts from this repository post, extract the script files and directory folders [import and delete actions], then upload those files and directory folders to a working target folder location (e.g. /var/tmp/api-scripts) on the target management server where the scripts will execute from.  Once uploaded to a working folder the relevant scripts are executed like any other bash script.  If executing directly from the folder where the script is located use "./<script>.sh" for execution.  If script modifications are made outside of Check Point Linux, it is recommended to first run "dos2unix <script>.sh" to ensure compatibility with bash shell.

Each script accepts command line parameters to control important inputs that have some defined defaults.

If the "-p <password>" parameter is not used, the user is prompted for the console user/administrators password, as in this example:

If the "-r"  or "--root" parameter is used then the above prompt should be skipped as in standard mgmt_cli execution.

Command Line Parameters

The scripts all (except actions sub-scripts) can take Command Line Parameters (CLI parameters).  To get a dump of the active CLI parameters for a specific script run it with "--help" or "-?".  Example  (version 00.23.00):

Command line parameters support multiple input formats as displayed, and can be mixed and matched as needed.

This is the standard help output for cli_api_export_objects.sh script, which is the standard baseline for all scripts in this package release:

[Expert@X:0]# ./cli_api_get_object_totals.sh --help

Script:  cli_api_get_object_totals  Script Version: v00x23x00
API version = 1.1

./cli_api_get_object_totals.sh [-?][-v]|[-r]|[-u <admin_name>] [-p <password>]]|[-P <web ssl port>] [-m <server_IP>] [-d <domain>] [-s <session_file_filepath>]|[-x <export_path>] [-i <import_path>] [-k <delete_path>] [-l <log_path>]

 Script Version:  00.23.00  Date:  2017-07-22

 Standard Command Line Parameters:

  Show Help                  -? | --help
  Verbose mode               -v | --verbose

  Authenticate as root       -r | --root
  Set Console User Name      -u <admin_name> | --user <admin_name> |
                             -u=<admin_name> | --user=<admin_name>
  Set Console User password  -p <password> | --password <password> |
                             -p=<password> | --password=<password>

  Set [web ssl] Port         -P <web-ssl-port> | --port <web-ssl-port> |
                             -P=<web-ssl-port> | --port=<web-ssl-port>
  Set Management Server IP   -m <server_IP> | --management <server_IP> |
                             -m=<server_IP> | --management=<server_IP>
  Set Management Domain      -d <domain> | --domain <domain> |
                             -d=<domain> | --domain=<domain>
  Set session file path      -s <session_file_filepath> |
                             --session-file <session_file_filepath> |
                             -s=<session_file_filepath> |
                             --session-file=<session_file_filepath>

  Set log file path          -l <log_path> | --log-path <log_path> |
                             -l=<log_path> | --log-path=<log_path>

  Set export file path       -x <export_path> | --export <export_path> |
                             -x=<export_path> | --export=<export_path>

  session_file_filepath = fully qualified file path for session file
  log_path = fully qualified folder path for log files
  export_path = fully qualified folder path for export file

 NOTE:  Only use Management Server IP (-m) parameter if operating from a
        different host than the management host itself.

 Example: General :

 ]# cli_api_get_object_totals -u fooAdmin -p voodoo -P 4434 -m 192.168.1.1 -d fooville -s "/var/tmp/id.txt" -l "/var/tmp/script_dump/"

 Example: Export:

 ]# cli_api_get_object_totals -u fooAdmin -p voodoo -P 4434 -m 192.168.1.1 -d fooville -s "/var/tmp/id.txt" -l "/var/tmp/script_dump/" -x "/var/tmp/script_dump/export/"

Standard Command Line Parameters:

Parameter Purpose	Parameter value and options	Default Value	Description
Show Help	-? | --help	n/a	Show help for script
Verbose Mode	-v | --verbose	n/a (not set)	Show details of operations and values during execution.  bash environment variable APISCRIPTVERBOSE can be set to TRUE to run in verbose mode from start without Command Line Parameter. Example: export APISCRIPTVERBOSE=TRUE
Authenticate as root	-r | --root	n/a	Instead of using administrator user name and password operate as root
Set Console User Name	-u <admin_name> --user <admin_name> -u=<admin_name> --user=<admin_name>	administrator	Set the username of console user/administrator executing the script. <admin_name> username for console/administrator, e.g. admin
Set Console User password	-p <password> --password <password> -p=<password> --password=<password>	n/a	Set the password to be used for console user/administrator authentication. If not used the default operation will prompt for the console user/administrator password. <password> password to use for console user/administrator. NOTE:  Entry is visible when used.
Set Management Server IP	-m <server_IP> --management <server_IP> -m=<server_IP> --management=<server_IP>	localhost	Set the IP address of the management server to use for this operation. <server_IP> is the TCP/IP address of the target management server, e.g. 10.10.100.66 NOTE:  DO NOT USE THIS PARAMETER IF OPERATING THE SCRIPT FROM THE HOSTING MDS OR SMS, SINCE AUTHENTICTATION WILL FAIL.
Set Management Domain	-d <domain> --domain <domain> -d=<domain> --domain=<domain>	not set	Set the management domain to use for this operation on a Multi-Domain Management Server <domain> is the domain to use for the operation, e.g. fooville
Set session file path	-s <session_file_filepath> -session-file <session_file_filepath> -s=<session_file_filepath> -session-file=<session_file_filepath>	./id.txt	Set the full path and file name to the session ID file <session_file_filepath> full path to the session ID file, e.g. /var/tmp/id.txt
Web SSL Port NEW	-P <web_ssl_port> | --port <web_ssl_port> | -P=<web_ssl_port> | --port=<web_ssl_port>	443	Web SSL Port of the Management server, default is 443, but can be set explicity to address changes to multiportal, thus change to API web port.
Session Timeout	--session-timeout <session_time_out>	10 - 3600 600 default seconds	Configure session timeout value for login operation executed
System Object Export	--NSO | --no-system-objects --SO | --system-objects	--NSO	Configure export of System Objects created by "System".  By Default this value is set to --NSO or --no-system-objects and objects created by "System" are ignored during the export of full JSON or CSV information.  Standard JSON export always will do all objects found since the search for "System" objects is not possible with the supplied JSON stream.  To enable export of "System" created objects, utilise the --SO or --system-objects parameter.  For JSON output --NSO will generate zero length files as the dump.
Log File path	-l <log_path> --log-path <log_path> -l=<log_path> --log-path =<log_path>	folder path	Set the path for log files generated by the script. <log_path> path (no following "/"), e.g. "./var/tmp/script/logs"
Output File path	-o <output_path> --output <output_path> -o=<output_path> --output =<output_path>	folder path	Set the path for output files generated by the script. <export_path> path (no following "/"), e.g. "./var/tmp/script"
Set export file path CHANGED	-x <export_path> --export <export_path> -x=<export_path> --export =<export_path>	./dump/<yyyy-mm-dd-hhmm-tz>	Set the path for export files generated by the script. <export_path> path (no following "/"), e.g. "./var/tmp/script"
Set import file path	-i <import_path> --import-path <import_path> -i=<import_path> --import-path=<import_path>	./import.csv	Set the path for input files required by the script. <import_path> path (no following "/"), e.g. "./var/tmp/script/input"
Set delete file path	-k <delete_path> --delete-path <delete_path> -k=<delete_path> --delete-path=<delete_path>	./delete.csv	Set the path for input files required by the script to identify what to delete. <delete_path> path (no following "/"), e.g. "./var/tmp/script/input"
--NOWAIT			Skip waiting for key input on some operations or when running in verbose mode
--CLEANUPWIP			Remove the WIP folder created under some operational output operations (e.g. CSV exports) - PENDING IMPLEMENTATION
--NODOMAINFOLDERS			Don't generate the domain specific folders, files are domain specific so all collected - PENDING IMPLEMENTATION
--CSVEXPORTADDIGNOREERR			Automatically modify the CSV file to include the presumed ignore error, warning, or set-if-exist values - PENDING IMPLEMENTATION

Configuration Parameters in the Script [this section needs more work]

NOTE:  This predicates some scripting ability and capability to use a text editor.  I recommend using the dos2unix command on any updated scripts once uploaded to the target management server host to ensure compatibility.

These script examples attempt to provide some detail tailoring and configuration via variables set for the specific script.  Some of these configuration values are influenced by the Command Line Parameters that can be passed to the script.  This version does not make the approach overly generic (e.g. name of exported CSV file is hardcode in the import), and future versions of this example set may clearly abstract and configure command line input variables.

Key values to configure for:

Export, Import, and Delete scripts

Variable	Definition
APICLIadmin	SmartConsole administrator name to use for operations
APICLIsessionfile	filename and path to mgmt_cli session ID file generated by login and used for all subsequent mgmt_cli operations

Export Scripts

Variable	Definition
APICLIpathroot	root of path for output files
APICLIpathbase	base path for output files, generally uses $APICLIpathroot and for operations time delineation can utilize the $DATE variable
APICLIfileoutputpre	General prefix for the output file, prefixes the filename in the full output file path
APICLIfileoutputext	File extension for operational output file, default is .txt
APICLIfileoutputsufix	File suffix for the operational output file, default is $DATE.$APICLIfileoutputext so generally<date_time_group>.txt
APICLIJSONfileoutputext	File extension for mgmt_cli json output file, default is .json NOTE:  this is not used in this example
APICLIJSONfileoutputsufix	File suffix for the mgmt_cli json output file, default is $DATE.$APICLIJSONfileoutputext so generally<date_time_group>.json NOTE:  this is not used in this example
APICLICSVfileoutputext	File extension for generated CSV file, default is .csv
APICLICSVfileoutputsufix	File suffix for the operational output file, default is $DATE.$APICLICSVfileoutputext so generally<date_time_group>.csv NOTE:  this was purposely done for the work utilizing this example, which stipulates a defined state of CSV output to export based on the time of execution.  For those wanting a generic approach, the value can be set to be more static and not include the $DATE value element.
APICLIObjectLimit (DO NOT MODIFY THIS VALUE)	This is the maximum number of groups to export, providing the limit value for the mgmt_cli show groups command to populate the array of groups to export members from.  The API supports a "limit" value of 0 to 500, and the default is set to 500 to ensure the maximum number of objects is collected.
APICLIoutput	full file path to operational output file for later review of actions

Import Scripts

Header 1	Header 2
APICLIfileoutputpre	General prefix for the output file, prefixes the filename in the full output file path
APICLIfileoutputext	File extension for mgmt_cli json output file, default is .json
APICLIfileoutputsufix	File suffix for the mgmt_cli json output file, default is $DATE.$APICLIfileoutputext so generally<date_time_group>.json
OutputPathRoot	root of path for output files
OutputPathBase	base path for output files, generally uses $OutputPathRoot and for operations time delineation can utilize the $DATE variable
CSVImportType	mgmt_cli type for import operation, in this example it is group
CSVImportPathRoot	This is the path root for the location of the CSV file to import, in the example it is a sub-directory relative to the location of the script
CSVImportPathFile	This is the file name of the CSV file to import, in this case hard-coded based the CSV output generated by the export operation. NOTE:  this was purposely done for the work utilizing this example, which stipulates a defined state of CSV output to import.  For those wanting a generic approach, the value can be set to be more static and not include the $DATE value element.
CSVImportPath	This is the path to the CSV file to import based on the $CSVImportPathRoot and $CSVImportPathFile variables.
OutputPath	full file path to operational output file for later review of actions

Delete Scripts

Header 1	Header 2
APICLIfileoutputpre	General prefix for the output file, prefixes the filename in the full output file path
APICLIfileoutputext	File extension for mgmt_cli json output file, default is .json
APICLIfileoutputsufix	File suffix for the mgmt_cli json output file, default is $DATE.$APICLIfileoutputext so generally<date_time_group>.json
OutputPathRoot	root of path for output files
OutputPathBase	base path for output files, generally uses $OutputPathRoot and for operations time delineation can utilize the $DATE variable
CSVImportType	mgmt_cli type for import operation, in this example it is group
CSVImportPathRoot	This is the path root for the location of the CSV file to import, in the example it is a sub-directory relative to the location of the script
CSVImportPathFile	This is the file name of the CSV file to import, in this case hard-coded based the CSV output generated by the export operation. NOTE:  this was purposely done for the work utilizing this example, which stipulates a defined state of CSV output to import.  For those wanting a generic approach, the value can be set to be more static and not include the $DATE value element.
CSVImportPath	This is the path to the CSV file to import based on the $CSVImportPathRoot and $CSVImportPathFile variables.
OutputPath	full file path to operational output file for later review of actions

Modification of the script sections to suit personal preference and requirements is strongly encouraged via the copy-paste operation.

I may be updating these later, with some harmonization of common variables required and some abstraction options via command line parameters.

Why and What for...

These scripts were developed to address a pressing need in my own basement cloud laboratory, after some issues cropped up with my migrated management server, which has an original data base starting from R70 and migrated, upgraded, imported to Multi-Domain Management, and now exported from Multi-Domain Management, which has left the system a bit wonky and questionable.  By creating scripts to handle the output of objects from my existing management server, I can then use the CSV data to import to a clean, new installation, where I can start fresh, with all my objects, but probably none of the baggage or garbage from almost 9 years of lab/home use operations.  It is an excellent learning opportunity and mentors like Uri Bialik help with this very much.

However, these scripts can also help with some other operations that may be necessary, probably requiring some tweaks, but the example can help a bunch for starting out, operations like:

• Duplicating group members after group import for laboratory environments, when building a from-scratch test environment, but wanting to use familiar objects
• Duplicating group members after group import to a different Domain in Multi-Domain Management, when not wanting to use global objects.  This will require some adjustments to handle the Domain specification variables, but should not be rocket magic to make happen.
• Operations in Professional Services for either recovering from an exported baseline, or assisting with pre-migration testing operations, and rebuild operations
• Having a backup of objects on hand to utilize for bare-metal rebuild where a import or restore is not plausible or advisable.

Future Improvements and Extensions:

• PARTIALLY Implemented - Implementing and improving error handling, currently omitted due to time constraint and for actual mgmt_cli operations a concept for approach
• JSON Implemented - Export, Import, and [yes also] Delete of Services, but very much focused on those created by the user, not the native Services delivered in the installation
• Configuration of exported fields to CSV for more control at import
• Interactive selection of what items to export, import, or delete depending on script
• Port to Windows Power Shell scripting, once I figure out how to handled date-time-group values and jq in Windows Power Shell.
• Configuration of system objects creators to exclude, and parameter control to select specific creator objects.
• PARTIALLY Implemented - Common bash script element framework to simplify script development, which may mean more detailed script installation requirements.  Later with Power Shell port also for Power Shell.
• Addition of a global [or any use] naming pre-fix to each object exported to CSV to allow alternative import, e.g. to move objects from local domain to global domain on Multi Domain Management implementation.
• Addition of specific information fields for CSV export to help identify objects for other operations (e.g. not direct import), like uid, creator, state, etc.  These dumps may require more work to prepare if import is desired

Now on GITHUB

GitHub - mybasementcloud/R8x-export-import-api-scripts: Check Point R8x Export, Import, [and more] A...

 

FEEDBACK

If you need help with a problem using the script, please provide the following in any communication:

• R80.x version being used and management type (SMS or MDM)
• CLI level text capture (copy/paste of terminal window for execution) with the script run in "-v" (verbose mode) to ensure we see all comments and information from the execution
  • Do not use the "--password" option to ensure that you're password is not provided, I don't want to know
  • Ensure all command line parameters passed are correct
• any CSV or JSON output generated for the problem object(s), this can be compressed to save e-mail space
• if importing, setting, or deleting values the utilized CSV file(s)
• a short explanation of what you are trying to accomplish and what is not being met, since we may be operating on different assumptions

Code Version

Code version 0.29.02 and later

Tested on version

R80, API version 1.0

R80.10 EA, API version 1.? [2016-12 EA package]

R80.10 GA, API version 1.1

R80.10 New Kernel EA, API version 1.1

R80.10 GA for Smart-1 525, 5050, 5150, API version 1.1

R80.20 EA T354, EA T395 MDM and SMS, R80.20 GoGo EA 3.10 kernel gateways

R80.20 GA T101

R80.30 EA (Public)

Known Limitations

Code version	Limitation Identified
All up to current	RESOLVED with v00.29.02 release Using the -r or --root command line operation fails to execute authentication due to a limitation in approach with parametrized command line parameters for the mgmt_cli command.  This issue is being escalated for technical clarification since the same values entered directly (instead of by evaluation parameter) works.
All	Using the -m or --management command line parameter is only supported when script is not executed on the actual management server host that is the target.  When executing the script on the actual management server host, DO NOT use the -m or --management command line parameter with management IP address.

 

Change Log

Code version	Key Changes
0.17.25	Updated scripts to include comprehensive Command Line parameter handling (CLI parameters) Added specific scripts for explicit object export to csv. Added delete objects package for clean-up opererations Refined operation of scripts to leverage sub routines for repeated operations with extensive parameterization to simplify adding more objects and services Solved the way to pass variable to JQ element in export operation Think I've solved the MDS JQ location problem with older scripts. Providing packaged sets for export, export of specific objects, import, delete, and template shell version 0.5.0
0.21.00	Updated to correct issues with assumptions around how command line parameters for management, domain, and port work. Now able to handle domain, management server
0.22.00	Corrected issue with changes to file naming that caused action scripts to fail
0.23.00	New scripts provided to just count the objects the script could find in the environment (as of version 00.23.00), specifically the "cli_api_get_object_totals.sh" and "cli_api_get_object_totals_w_group_member_count.sh".  Which is useful to determine if what the api sees given the input parameters and user rights is what the admin expects or should see, specifically to help with Multi-Domain operations, which may still need some more tweaking. -P | --port <web ssl port> parameter added to support other than port 443 for management hosts -o | --output CLI parameter renamed to -x | --export to conform with actual purpose Check for zero size groups added to export operations for group members Export scripts now handled > 500 objects (current maximum api limit value) and will iterate in 500 chunk steps to process object sets larger than 500 objects.  json files will be broken out in 500 object sets with the name adding an increment value in sets of 500.  CSV files are still single, since they are built differently. For import added version 1.1 api support for "set-if-exists" option, with api version checking. Added more dependency checking before start and overhauled some operations to enable easier changes.
0.24.00	New script added to set values of existing objects (as of version 00.24.00), specifically the "cli_api_set-update_objects_from_csv.sh" in the Import Objects set.  It will process all expected input files, and skip operations where it does not find a file. Added handling for export, import, and set for host's interface objects.  NOTE:  The .interfaces[].subnet-mask value is not utilized, instead explict .interfaces[].mask-length4 and .interfaces[].mask-length6 are utilized. Export scripts not handle > 500 objects on all exported object types and formats (both CSV and JSON), this now includes the handling of group members and host interfaces, which required more programming thought to identify the approach. Reduced the default wait time in "read -t <waittime>" from 600 to 15 and added the value $WAITTIME that can be set higher if desired.  Future addition of --waittime <wait-time> and --nowait options are under consideration.
0.25.00	Corrected semantic issue with detecting empty object types (i.e. no objects of that type), now the CSV export will not exit the whole script if no objects are found Extracted the Command Line Parameter Handler to a dedicated subscript to simplify editing the other scripts. Addressed a semantic issue with web ssl-port not available before identifing the API version and not having the jq location available on MDM versus SMS installations Prepared some main export files for future object handling expansion NOTE:  This is the last version that will have dedicated specific object handling scripts, the next script set will have an option to configure which objects to handle interactively
0.25.01	Correct spelling mistakes for output [an ongoing process--always]
0.27.05	Major overhaul of script operational approach in some functions with logical fixes to semantic approach. Updated command line parameter handler to varriant 003.  As of version 00.27.05 all cli_api*.sh scripts require "cmd_line_parameters_handler.action.common.003.sh" script to handle command line parameters!  Don't forget to copy this file also. Added ability to configure export of objects created by "System" to remove potential for import of system objects and reduce size of output.  By Default this value is set to --NSO or --no-system-objects and objects created by "System" are ignored during the export of full JSON or CSV information.  Standard JSON export always will do all objects found since the search for "System" objects is not possible with the supplied JSON stream.  To enable export of "System" created objects, utilise the --SO or --system-objects parameter.  For JSON output --NSO will generate zero length files as the dump. Templates were also updated to reflect changes in command line handler and approach.
0.29.02	Resolved issues blocking use of -r parameter for local administrator authentication.  Only usable on the actual host and will not work with -m option (will indicate if identified). Corrected approach to selection of Check Point Data objects, since System creator is set by upgrade for all objects when upgrading from R77.30 and prior versions of management. Implemented "common" folder for common operations, where future common code elements will live their own version lives, like the CLI parameter handler in the current release. Restructured much of the internal code to fix issues and simplify future updates. Release folder structure and pressed files shared now reflects actual approach used in operation and provides total package, not just specifics.
0.29.05	Moved to providing Operations and Development branches for scripts in GitHub. General corrections and reworking operations.  Last release to include development of individual scripts for import and export of explicit objects.
0.31.00	Removed explicit objects import and export scripts from development operations and updates, and now legacy (version 00.29.05) variants of those scripts are provided under the Development branch under ../export_import.wip/_Deprecated_Scripts. Continued corrections and refactoring.  First release of the cli_api_export_all_domains_objects.sh and cli_api_export_all_domains_objects_to_csv_files.sh scripts to handle all objects in all domains on an MDS the user has rights to.
0.33.00	Production release with updates and working identification of local management host web ssl port for API login configuration.  Operational improvements and template corrections in preparation to adding more objects supported for CSV export and future changes to allow selection of operations.

[Suleyman_B_T]

Hi,

Thank you for export import script.

I have a question about script.

Export object limit 500.

export APICLIObjectLimit=500

I increased the value through script.(vi editor)

Script did not work after editing

How can I increase the export&import object limit?

Thank you

Best Regards

Suleyman Toreli

SECURRENT

[Eric_Beasley]

Hello,

API currently has a object limit value of 1-500, so can't go higher than 500 as far as I can tell.  I'll see if I can do something to determine number of objects and slice the content into 500 objects slices to get output to csv for all the objects.  Import operations via add don't apparently use the limit entry value so don't have the problem, but show operations do.

Will look into this for the next version of script, version 00.23.00 or later.

Eric

[Eric_Beasley]

Hello again,

Version 00.23.00 now handles more than 500 objects in 500 object slices for json output and it was tested with ~1200 objects to csv (single file).  This works for all objects except for group members, where only 500 groups are handled, but I don't have data to test whether it can handle more than 500 members for a group, since that handling is different from the limit based show "objects" operation.

Check GitHub for version 00.23.00 or later to test it out.

Also added script "cli_api_get_object_totals.sh" and "cli_api_get_object_totals_w_group_member_count.sh" that will just provide a count of objects and a log of operations to query group members.  These are located under the Export Objects folder or 7z file.

Eric

[Eric_Beasley]

And one more update on this question.  Script version 00.24.00 now handles all processed objects for export with > 500 objects, even the tougher group members and new host interfaces settings.

Eric

[Tung_Nguyen_Son]

 Phuong Nguyen Le  Phi Thanh Tung Good script!

[Julien_Tissot]

Hello!

If there is no object of 1 category (for example no address range) the script terminates. Are you aware of that?

Otherwise it works really good, thank you!

[Eric_Beasley]

Can you e-mail me an execution where it terminates like that?  Also run it with the -v (verbose) parameter for more details, I need to see what the issue is.

[Eric_Beasley]

OK, after some testing on a new system, identified the issue and fixed it and the initial problem on an MDS in version 00.25.00 posted to GitHub.

[John_Tammaro1]

Fantastic work. Just what we needed.

Just a heads up for the author that there is a spelling error in the output. This does not break any functionality at all but thought I would share this.

Dump times to /home/admin/results/2017-08-31-1518BST/full/dump_times_full_2017-08-31-1518BST.json
Now processing up to next 500 objects starting with object 0 of 3 remainging!

Should be "remaining" not "remainging" . Perhaps this could be fixed upstream ?

[Eric_Beasley]

Hi John,

Thanks for that, I’ll fix that in v00.25.01, typing fast can lead to great typos, and a standard text editor doesn’t spell check—so a little one like this is easy to miss. It’s more fun when you flub a command argument or variable name, that usually comes back around in testing.

Eric

[Zbynek_Michl]

Hello,

I have an issues with cli_api_export_object_group-members_to_csv.sh script. If I set APICLIObjectLimit=100, then:

- Each particular pair ("name","members.add") in group-members_standard_csv.csv is listed 6x, e.g.:

MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"

- List in group-members_standard_csv.csv is not complete, it contains just a part of the all records (I know, our db is quite big). Despite the script finishes with no error.

Could you have a look on it, please? Script version is 00.25.00.

Thanks

Zbynek

[Eric_Beasley]

Hi,

Why are you modifying the limit value, which identifies how many groups will be added to the array that the group members are searched from?

Leave the value where it was and also use the latest version of the scripts.

Group Members is built by reading the full list of groups into an array, then processing the array of groups for each group, for its respective members, such that we get all of the members listed for each group.

The proper example of a group members CSV file:

"name","members.add"

"TEST_GROUP_01","test-addr-range-01-062.128.100.000-062.128.101.255"

"TEST_GROUP_01","test-addr-range-02-010.128.100.000-010.128.101.255"

"TEST_GROUP_01","test-addr-range-03-010.128.102.000-010.128.103.255"

"TEST_GROUP_01","test-addr-range-04-010.128.104.000-010.128.105.255"

"TEST_GROUP_01","test-addr-range-05-010.128.106.000-010.128.107.255"

"TEST_GROUP_02","TEST-HOST-01"

"TEST_GROUP_02","TEST-HOST-02"

"TEST_GROUP_02","TEST-HOST-03"

"TEST_GROUP_02","TEST-HOST-04"

"TEST_GROUP_02","TEST-HOST-05"

"TEST_GROUP_03","TEST-HOST-02"

"TEST_GROUP_03","TEST-HOST-04"

"TEST_GROUP_04","netw_test_LAN_01"

"TEST_GROUP_04","netw_test_LAN_02"

"TEST_GROUP_04","netw_test_LAN_03"

"TEST_GROUP_04","netw_test_LAN_04"

"TEST_GROUP_04","netw_test_LAN_05"

"TEST_GROUP_05",".1.test.domain.local"

"TEST_GROUP_05",".2.test.domain.local"

"TEST_GROUP_05",".3.test.domain.local"

For best results in determining if and what is going wrong, provide a executing of the script with the “-v” (verbose mode) and a full copy of the script execution from the CLI. I recommend not using the “—password” option, but entering that at the prompt when the mgmt_cli login executes, since I don’t need to know any passwords.

That should indicate if the processing is happening as expected.

Latest version is updated on GitHub: https://github.com/mybasementcloud/R8x-export-import-api-scripts, right now that is version 00.25.01

Versions prior to 00.23.00 do not execute operations for more than either 500 (API limitation) or the set APICLIObjectLimit value objects, and version 00.24.00 and higher handle group members and host interfaces for more than 500 groups or hosts, respectively.

BR

Eric Beasley

Security Engineer, North America Channel Sales

Check Point Software Technologies Ltd.

Cell: +1 708-224-7724 E-mail: ericb@checkpoint.com

Additional Contact Informaton:

Support : +1 (888) 361 5030 or +1 (972) 444 6600 Technical Assistance Center (TAC)

Partner Sales Assistance: (866) 488-6689 or cpp@checkpoint.com

Account Services: +1 (972) 444-6600, option 5 or AccountServices@checkpoint.com

Licensing Advocacy: +1 (800) 429-4391 or advocacy@checkpoint.com

Inside Sales: +1 (650) 628-2225

Check Point Incident Response Team: (866) 923-0907 > initial consultation is free

emergency-response@checkpoint.com<mailto:emergency-response@checkpoint.com> > you do not need to be an existing Check Point customer

[Michael_Vogler]

Hi Eric, 

first of all thank you for the great scripts!

I wanted to use the Script cli_api_export_object_group-members_to_csv.sh, too. But I have the same Problems like Zbynek. To get full information I run the script in Verbose Mode and i noticed, that the script stop exporting exactly after 1000 Elements. Perhaps there is a Limit of 1000 anywhere?

Best regards

Michael

[Zbynek_Michl]

I am modifying the APICLIObjectLimit because the default value (500) crashes the API java server with our DB. And we do not have any fix from CP yet. So the only chance for me is to decrease the value to 100 now, which seems to be safe.

Regards

Zbynek

[Paul_Warnagiris]

Hi Eric.  I must admit I am no coder, I just do engineering.  I'm trying to export R80.10 objects and move it to another 80.10 setup.  When I run your script cli_api_export_objects_to_csv.sh v00.26.01  I get the following error.  Any help would be appreciated:

Done dumping hosts

Use array of hosts to generate host interfaces CSV

!!!! Error data file missing : ./dump/2017-11-06-1111UTC/csv/wip/hosts_full_csv.csv.data
Terminating!

[Eric_Beasley]

This issue was identified and fixed in 00.26.07 and again tested in 00.27.05.

[Joshua_Hatter]

Gotta say kudos just for the documentation and write up. Great work!

[Martin_Valenta]

Great stuff man, what is ETA for services exports?

[Brandon_Cotter]

I successfully produced an export and then trimmed down the csv (hosts_full_csv.csv) output to just one host as an experiment. I ran the import script and the single host was not imported. It output a file called "dump_add_host_json" that contained the text

Line 2: {
"code" : "generic_err_invalid_parameter",
"message" : "Parameter [auto-rule] value is not valid (full path: nat-settings.auto-rule)"
}

{ "response" : []}

What am I doing wrong? Is partial import not supported?

Also, if I do a full import, what is the overwrite setting for existing objects?

Thanks!

[Bob_Stevens1]

Thanks for these awesome scripts!  They have come in super helpful in a conversion project!

I do have one request:

Would it be possible to get the script updated to also export any service objects and service groups?  We need to compare these between rulebases and think it will most easily be done in a csv file.  I can export services through SmartConsole, but the groups are what don't come through correctly.

Thanks again for all of your hard work!

[Brandon_Cotter]

I started using this script with not much knowledge of how to use the API, and was able to make it work despite that. Awesome! I had a couple questions going into this that I have now come up with answers to, so I thought I'd add them. Apologies in advance for any dimness.

Can I use this script to just import selected objects?

Basically, no. But you can edit down an outputted csv from "cli_api_import_objects_from_csv.sh" and then import it manually with a pretty simple command. I dove into the script and found the basic behavior.

mgmt_cli add [type] --batch [path to modified csv file for that type from the export script] set-if-exists true ignore-warnings true ignore-errors true --ignore-errors true -f json -s [path to session file]

Values for [type]:

host | network | group | group-with-exclusion | address-range | dns-domain | security-zone | dynamic-object | application-site | application-site-category | application-site-group | tag | simple-gateway | time | time-group | access-role | opsec-application | service-tcp | service-udp | service-icmp | service-icmp6 | service-sctp | service-other | service-dce-rpc | service-rpc | service-group

Does this script overwrite existing objects?

"set-if-exists true" is declared in the script, and will modify existing objects to match the imported form. Changing the line

export MgmtCLI_Add_OpParms="set-if-exists true $MgmtCLI_IgnoreErr_OpParms $MgmtCLI_Base_OpParms"

to

export MgmtCLI_Add_OpParms="set-if-exists false $MgmtCLI_IgnoreErr_OpParms $MgmtCLI_Base_OpParms"

should leave existing objects alone.

Hope this isn't misleading anyone.

[VincentBacher]

Hi,

just tested those tools and exporting importing seems to work pretty well.

I am just wondering that not all objects have been imported from dumped csv files.

Csv files have exactly the number of objects of the original smartcenter, so exporting was fine.

I was searching around but did not see in the console output nor in logfile why some objects were not imported.
870 of 877 host objects have been imported and 256 of 300 nets.

Would be nice to know if there any issues in the data to be imported so i could check and modify the objects in the csv prior to import.
Any hints would be appreciated

cheers

Vincent

[VincentBacher]

Hi all,

did anybody already test it on R80.40 ?

I assume it should work, right?

br

Vincent

[Security_Consul]

I trying to add firewall rule via text/csv file in check with sk113078 and https://sc1.checkpoint.com/documents/latest/APIs/index.html#clish/add-access-rule~v1.5%20
But still dont understand.
So my example script:
layer,position,name,action,source,destination,service,track,install-on
Network,1,Rule1,accept,source.1Host_10.10.10.10source.2Host15.15.15.15,Host_20.20.20.20,server.1tcp-22service.2tcp-443,logs,Firewall1

[Workz]

This script is so good, appreciate the effort taken and sharing with all. Do we have any update on getting the services plus services groups also?

[stallwoodj]

Excellent scripts! Thank you!

Just a small one, expanding groups with space in its name during the cli_api_export_objects_to_csv script throws an error (target R80.40):

Group Franking Machines number of members = 2
Error: The parameters of show-group command should be provided in pairs (key and value). You have provided an odd number of parameters which suggests that you are probably missing a parameter.

Thanks Jamie

[Benjamin_Weiss]

Hi!

I see that this will work for R80.30, but don't see anything about R81.10. I'm creating a new FWMS on R81.10 and would like to move my network objects. Can I use these scripts to do so?

Thanks!

Ben

[PhoneBoy]

The API is versioned, which means they should work fine with R81.10.

[the_rock]

Yes, you can. I used it few times and works fine.

Andy