Check out GITHUB for latest versions : GitHub - mybasementcloud/R8x-export-import-api-scripts: Check Point R8x Export, Import, [and more] A... (version v00.60.12.000.050 as of 2022-10-27)
https://github.com/mybasementcloud/R8x-export-import-api-scripts/releases
The post below is not manageable and won't be updated any more, all specifics and improved documentation is provided on GitHub.
BR
Eric @Eric_Beasley
Overview
The export, import, delete using CSV files scripts in this post, currently version 00.33.00 and later, dated 2019-01-19 and later, are intended to allow operations on an existing R80, R80.10, R80.20[.Mx], R80.30[.Mx] Check Point management server (SMS or MDM) from bash on the management server or a management server able authenticate and reach the target management server.
These scripts show examples of:
- an export of objects and services with full and standard json output
- an export of hosts, networks, groups, groups-with-exclusion, address-ranges, dns-domains, host interfaces, and group members to csv output [future services and other object dumps to csv is pending further research]
- an import of hosts, networks, groups, groups-with-exclusion, address-ranges, dns-domains, host interfaces, and group members from csv output generated by the export to csv operations above or custom csv files with valid objects.
- a set operation (change existing objects) for hosts, networks, groups, groups-with-exclusion, address-ranges, dns-domains, host interfaces, and group members from csv output generated by the export to csv operations above or custom csv files with valid objects. (As of version 00.24.00). Script added to specifically set values using CSV file for all objects for which a CSV file is found, specifically the: "cli_api_set-update_objects_from_csv.sh"
- a script to delete groups-with-exclusion, groups, address-ranges, dns-domains, networks, hosts, using csv files created by an object name export to csv for the respective items deleted. NOTE: DANGER!, DANGER!, DANGER! Use at own risk with extreme care!
- scripts provided to just count the objects the script could find in the environment (as of version 00.23.00), specifically the:
"cli_api_get_object_totals.sh" and "cli_api_get_object_totals_w_group_member_count.sh".
- script operations for export of more than 500 objects of a specific type.
- export complex object elements, like Group Members and Host Interfaces to CSV files for import via the --batch option.
- export of full JSON and CSV for only objects that are not created by "System", so exporting only custom added or administrator modified objects.
- MDM tool to export domains to domains_list.txt file for reference in other script calls
NOTE:
- As of version 00.29.00 all cli_api*.sh scripts require "common" script folder to handle command line parameters! Don't forget to copy this folder also.
- EXPORT SCRIPTS NOW HANDLE > 500 OBJECTS FOR ALL CSV AND JSON EXPORTS (version 00.24.00)
- EXPORT SCRIPTS NOW DEFAULT TO EXPORTING ALL OBJECTS INSTEAD OF NON-SYSTEM OBJECTS FOR FULL JSON AND CSV OPERATION. CLI COMMANDS ENABLE FULL DUMP OF ALL OBJECTS INSTEAD OF THOSE CREATED BY "System": --SO and --NSO for show System Objects and No Show System Objects. (version 00.29.02)
- AS OF VERSION 00.33.00, THE PORT FOR MANAGEMENT WEB SSL PORT IS IDENTIFIED AUTOMATICALLY IF EXECUTING AGAINST THE LOCAL MANAGEMENT HOST.
Where to Find, Contact, Support, and Terms of Service
Scripts are provided as-is without express or implied warranty, guarantee, assumption of liability, or SLA for resolution as they are examples of how to use the API and may not always apply to the situation they are used in and are subject to limitations of the API engine or utilized Check Point management version.
At no time is this post an implied assumption of duty to change the example scripts IAW requests. If you need an API script developed, contact Check Point Professional Services or a qualified Check Point Channel Partner with DevOps capabilities.
CLI API Script approach may change at any time, which may require accessing older GitHub branches or releases.
CLI API Scripts are provided with stipulation that the implementor is capable of using CLI and understands API operations.
For direct questions, you can hit me up at ericb@checkpoint.com Response time may vary based on schedule, availability, and issue presented.
For questions about issues with scripts please provide the information identified in the FEEDBACK section at the bottom of this article.
Description
This post includes a set of scripts in two (2) packages, a Development Branch which may be an advanced version still under construction and an Operations Branch that should work as expected. All script files end with .sh for shell and are intended for Check Point bash implementation on R80, R80.10, R80.20[.Mx], and R80.30[Mx] or later. Scripts in the packages have specific purposes and scripts call sub-scripts for extensive repeated operations and basic actions (e.g. handling CLI parameters). The packages also include specific expected default directory folders that are not created by the script action.
General Information:
<yyyy-mm-dd-hhmm-tz> is a date time group (DTG) generated at time of execution and used for the full operation of the respective script, providing consistent information for a specific script run. Example: 2017-01-05-1346CST for January 5, 2017, at 13:46 hrs CST.
Output Generated by Scripts:
Output from the scripts is directed to a sub-folder (default is a dump folder with DTG sub-folder, e.g. ./dump/<yyyy-mm-dd-hhmm-tz>) and further placed in sub-folders based on script: csv, full, standard, import, delete.
Example:
Multi-Domain Management operations scripts that handle multiple domains in their operation will create a sub-folder for the domain and then create the specific output based on that domain in the domain specific folder. Note that the scripts automatically handle the default "System Data" and "Global" domains. When executing the scripts with specific domain selected, that domain is the folder name between the DTG folder and the output folders.
Example:
NOTE: Current CSV output includes additional files used in the process that are raw data in a WIP folder, sorted raw data, csv header, and the original data with header.
NOTE: When operating on MDM, the stated domain in the -d parameter is used as a subfolder to collect the specific data for that dumped domain.
Scripts
Scripts a provided in an Operations and Development branch folder structure. Operations branch implies that these are stable operational scripts for their stated purpose, and expected to work as such under the tested versions where applicable. Development branch may include advanced, early availability version of the scripts where development is not yet complete for migration to Operations branch.
Main script types provided are MDM scripts, Session Cleanup scripts, common scripts used by other scripts, Export Import scripts, and script Templates.
|
Script Type
(main folder)
|
Script Name |
Script Purpose |
Output formats |
| _templates |
Templates for developing scripts with the basic capabilities of the provided version level of the template. |
| api_mgmt_cli_shell_template_with_cmd_line_parameters.template.<version>.sh |
Template for scripts using provided approach that includes built-in command line parameter handler operations. |
Text log file
Output dependent on chosen template implementation
|
| api_mgmt_cli_shell_template_with_cmd_line_parameters_script.template.<versions>.sh |
Template for scripts using common scripts approach for handling command line parameters |
Text log file
Output dependent on chosen template implementation
|
| api_mgmt_cli_shell_template_action_handler.template.<version>.sh |
Template for action sub-script called by another script, with basic handling for version mismatch |
log to originating script based on verbose level setting |
| test._templates.<version>.sh |
Rough testing script for validating if templates function |
Text log file
Output dependent on chosen template implementation
|
| common |
Common scripts called by the scripts and utilized by the templates in the _template folder. This folder and expected scripts are replicated to the utilizing script folder for direct access, so will be found under _templates, export_import, and Session_Cleanup |
| cmd_line_parameters_handler.action.common.<level>.<version>.sh |
Action sub-script called to execute operations to handle command line parameters standard to all scripts in that version |
log to originating script based on verbose level setting |
| identify_gaia_and_installation.action.common.<level>.<version>.sh |
Action sub-script called to execute operations to identify version of Gaia of host and Check Point installation type |
log to originating script based on verbose level setting |
| export_import |
Scripts for export, import, set, and delete operations |
| cli_api_export_objects.sh |
Script to export all supported objects to JSON (full and standard) and CSV |
JSON Full, JSON Standard, CSV
Text log file
|
| cli_api_export_objects_to_json_full.sh |
Script to export all supported objects to JSON full |
JSON Full, JSON Standard, CSV
Text log file
|
| cli_api_export_objects_to_json_standard.sh |
Script to export all supported objects to JSON standard |
JSON Full, JSON Standard, CSV
Text log file
|
| cli_api_export_objects_actions.sh |
Action sub-script to execute export of objects to JSON (full or standard) for calling script. |
log to originating script based on verbose level setting |
| cli_api_export_objects_actions_to_csv.sh |
Action sub-script to execute export of objects to CSV (full or standard) for calling script. |
log to originating script based on verbose level setting |
| cli_api_export_objects_to_csv.sh |
Script to export all supported objects to CSV |
CSV
Text log file
|
| cli_api_export_all_domains_objects.sh |
Script to export all supported objects in all domains on an MDS to JSON (full and standard) and CSV |
JSON Full, JSON Standard, CSV
Text log file
|
| cli_api_export_all_domains_objects_to_csv.sh |
Script to export all supported objects in all domains on an MDS to CSV |
CSV
Text log file
|
| cli_api_import_objects_from_csv.sh |
Script to import all supported objects from supplied CSV files (only supplied files are processed) |
JSON results
Text log file
|
| cli_api_set-update_objects_from_csv.sh |
Script to set (updated) all supported objects from supplied CSV files (only supplied files are processed) |
JSON results
Text log file
|
| cli_api_delete_objects_using_csv.sh |
Script to delete all supported objects from supplied CSV files by name (only supplied files are processed) |
JSON results
Text log file
|
| cli_api_get_object_totals.sh |
Dump the number of each object from all supported objects. |
Text log file
|
| cli_api_get_object_totals_w_group_member_count.sh |
Dump the number of each object from all supported objects, including count of group members per group. |
Text log file
|
| api_add_csv_error_handling_to_csv_file.sh |
For versions (e.g. R80.10) where CSV import and set operations require that ignore error, ignore warning, and set-if-exists are in the actual CSV file. Adds necessary columns to front of existing CSV file rows with headers. |
CSV
Text log file
|
| api_subpend_csv_error_handling_to_csv_files.sh |
For versions (e.g. R80.10) where CSV import and set operations require that ignore error, ignore warning, and set-if-exists are in the actual CSV file. Adds necessary columns to end (back) of existing CSV file rows with headers. |
CSV
Text log file
|
| MDM |
Multi-Domain Management general scripts |
| MDM_Get_Domains_List_<version>.sh |
Script will generate a list of current domains on an MDS for manual re-use later |
Text |
| Session_Cleanup |
Session Cleanup scripts to first list the current sessions open on a management server and also provide options to clean-up (delete) session that are not locked or pending publish operations (zero locks). |
| remove_zerolocks_sessions.<version>.sh |
Identify and delete zero lock sessions |
Text result of operation |
| remove_zerolocks_web_api_sessions.<version>.sh |
Identify and delete zero lock sessions with user web_api |
Text result of operation |
| show_zerolocks_sessions.<version>.sh |
Identify zero lock sessions |
Text result of operation |
| show_zerolocks_web_api_sessions.<version>.sh |
Identify zero lock sessions with user web_api |
Text result of operation |
Instructions
To utilize the scripts, download the scripts from this repository post, extract the script files and directory folders [import and delete actions], then upload those files and directory folders to a working target folder location (e.g. /var/tmp/api-scripts) on the target management server where the scripts will execute from. Once uploaded to a working folder the relevant scripts are executed like any other bash script. If executing directly from the folder where the script is located use "./<script>.sh" for execution. If script modifications are made outside of Check Point Linux, it is recommended to first run "dos2unix <script>.sh" to ensure compatibility with bash shell.
Each script accepts command line parameters to control important inputs that have some defined defaults.
If the "-p <password>" parameter is not used, the user is prompted for the console user/administrators password, as in this example:
If the "-r" or "--root" parameter is used then the above prompt should be skipped as in standard mgmt_cli execution.
Command Line Parameters
The scripts all (except actions sub-scripts) can take Command Line Parameters (CLI parameters). To get a dump of the active CLI parameters for a specific script run it with "--help" or "-?". Example (version 00.23.00):
Command line parameters support multiple input formats as displayed, and can be mixed and matched as needed.
This is the standard help output for cli_api_export_objects.sh script, which is the standard baseline for all scripts in this package release:
[Expert@X:0]# ./cli_api_get_object_totals.sh --help
Script: cli_api_get_object_totals Script Version: v00x23x00
API version = 1.1
./cli_api_get_object_totals.sh [-?][-v]|[-r]|[-u <admin_name>] [-p <password>]]|[-P <web ssl port>] [-m <server_IP>] [-d <domain>] [-s <session_file_filepath>]|[-x <export_path>] [-i <import_path>] [-k <delete_path>] [-l <log_path>]
Script Version: 00.23.00 Date: 2017-07-22
Standard Command Line Parameters:
Show Help -? | --help
Verbose mode -v | --verbose
Authenticate as root -r | --root
Set Console User Name -u <admin_name> | --user <admin_name> |
-u=<admin_name> | --user=<admin_name>
Set Console User password -p <password> | --password <password> |
-p=<password> | --password=<password>
Set [web ssl] Port -P <web-ssl-port> | --port <web-ssl-port> |
-P=<web-ssl-port> | --port=<web-ssl-port>
Set Management Server IP -m <server_IP> | --management <server_IP> |
-m=<server_IP> | --management=<server_IP>
Set Management Domain -d <domain> | --domain <domain> |
-d=<domain> | --domain=<domain>
Set session file path -s <session_file_filepath> |
--session-file <session_file_filepath> |
-s=<session_file_filepath> |
--session-file=<session_file_filepath>
Set log file path -l <log_path> | --log-path <log_path> |
-l=<log_path> | --log-path=<log_path>
Set export file path -x <export_path> | --export <export_path> |
-x=<export_path> | --export=<export_path>
session_file_filepath = fully qualified file path for session file
log_path = fully qualified folder path for log files
export_path = fully qualified folder path for export file
NOTE: Only use Management Server IP (-m) parameter if operating from a
different host than the management host itself.
Example: General :
]# cli_api_get_object_totals -u fooAdmin -p voodoo -P 4434 -m 192.168.1.1 -d fooville -s "/var/tmp/id.txt" -l "/var/tmp/script_dump/"
Example: Export:
]# cli_api_get_object_totals -u fooAdmin -p voodoo -P 4434 -m 192.168.1.1 -d fooville -s "/var/tmp/id.txt" -l "/var/tmp/script_dump/" -x "/var/tmp/script_dump/export/"
Standard Command Line Parameters:
| Parameter Purpose |
Parameter value and options |
Default Value |
Description |
| Show Help |
-? | --help |
n/a |
Show help for script |
| Verbose Mode |
-v | --verbose |
n/a (not set) |
Show details of operations and values during execution. bash environment variable APISCRIPTVERBOSE can be set to TRUE to run in verbose mode from start without Command Line Parameter.
Example:
export APISCRIPTVERBOSE=TRUE
|
| Authenticate as root |
-r | --root |
n/a |
Instead of using administrator user name and password operate as root |
| Set Console User Name |
-u <admin_name>
--user <admin_name>
-u=<admin_name>
--user=<admin_name>
|
administrator |
Set the username of console user/administrator executing the script.
<admin_name> username for console/administrator, e.g. admin
|
| Set Console User password |
-p <password>
--password <password>
-p=<password>
--password=<password>
|
n/a |
Set the password to be used for console user/administrator authentication.
If not used the default operation will prompt for the console user/administrator password.
<password> password to use for console user/administrator.
NOTE: Entry is visible when used.
|
| Set Management Server IP |
-m <server_IP>
--management <server_IP>
-m=<server_IP>
--management=<server_IP>
|
localhost |
Set the IP address of the management server to use for this operation.
<server_IP> is the TCP/IP address of the target management server, e.g. 10.10.100.66
NOTE: DO NOT USE THIS PARAMETER IF OPERATING THE SCRIPT FROM THE HOSTING MDS OR SMS, SINCE AUTHENTICTATION WILL FAIL.
|
| Set Management Domain |
-d <domain>
--domain <domain>
-d=<domain>
--domain=<domain>
|
not set |
Set the management domain to use for this operation on a Multi-Domain Management Server
<domain> is the domain to use for the operation, e.g. fooville
|
| Set session file path |
-s <session_file_filepath>
-session-file <session_file_filepath>
-s=<session_file_filepath>
-session-file=<session_file_filepath>
|
./id.txt |
Set the full path and file name to the session ID file
<session_file_filepath> full path to the session ID file, e.g. /var/tmp/id.txt
|
|
Web SSL Port
NEW
|
-P <web_ssl_port> | --port <web_ssl_port> | -P=<web_ssl_port> | --port=<web_ssl_port>
|
443 |
Web SSL Port of the Management server, default is 443, but can be set explicity to address changes to multiportal, thus change to API web port.
|
|
Session Timeout
|
--session-timeout <session_time_out>
|
10 - 3600
600 default
seconds
|
Configure session timeout value for login operation executed
|
|
System Object Export
|
--NSO | --no-system-objects
--SO | --system-objects
|
--NSO |
Configure export of System Objects created by "System". By Default this value is set to --NSO or --no-system-objects and objects created by "System" are ignored during the export of full JSON or CSV information. Standard JSON export always will do all objects found since the search for "System" objects is not possible with the supplied JSON stream. To enable export of "System" created objects, utilise the --SO or --system-objects parameter. For JSON output --NSO will generate zero length files as the dump.
|
|
Log File path
|
-l <log_path>
--log-path <log_path>
-l=<log_path>
--log-path =<log_path>
|
folder path |
Set the path for log files generated by the script.
<log_path> path (no following "/"), e.g. "./var/tmp/script/logs"
|
|
Output File path
|
-o <output_path>
--output <output_path>
-o=<output_path>
--output =<output_path>
|
folder path |
Set the path for output files generated by the script.
<export_path> path (no following "/"), e.g. "./var/tmp/script"
|
|
Set export file path
CHANGED
|
-x <export_path>
--export <export_path>
-x=<export_path>
--export =<export_path>
|
./dump/<yyyy-mm-dd-hhmm-tz> |
Set the path for export files generated by the script.
<export_path> path (no following "/"), e.g. "./var/tmp/script"
|
| Set import file path |
-i <import_path>
--import-path <import_path>
-i=<import_path>
--import-path=<import_path>
|
./import.csv |
Set the path for input files required by the script.
<import_path> path (no following "/"), e.g. "./var/tmp/script/input" |
| Set delete file path |
-k <delete_path>
--delete-path <delete_path>
-k=<delete_path>
--delete-path=<delete_path>
|
./delete.csv |
Set the path for input files required by the script to identify what to delete.
<delete_path> path (no following "/"), e.g. "./var/tmp/script/input" |
| --NOWAIT |
|
|
Skip waiting for key input on some operations or when running in verbose mode
|
| --CLEANUPWIP |
|
|
Remove the WIP folder created under some operational output operations (e.g. CSV exports) - PENDING IMPLEMENTATION
|
| --NODOMAINFOLDERS |
|
|
Don't generate the domain specific folders, files are domain specific so all collected - PENDING IMPLEMENTATION
|
| --CSVEXPORTADDIGNOREERR |
|
|
Automatically modify the CSV file to include the presumed ignore error, warning, or set-if-exist values - PENDING IMPLEMENTATION
|
Configuration Parameters in the Script [this section needs more work]
NOTE: This predicates some scripting ability and capability to use a text editor. I recommend using the dos2unix command on any updated scripts once uploaded to the target management server host to ensure compatibility.
These script examples attempt to provide some detail tailoring and configuration via variables set for the specific script. Some of these configuration values are influenced by the Command Line Parameters that can be passed to the script. This version does not make the approach overly generic (e.g. name of exported CSV file is hardcode in the import), and future versions of this example set may clearly abstract and configure command line input variables.
Key values to configure for:
Export, Import, and Delete scripts
| APICLIadmin |
SmartConsole administrator name to use for operations |
| APICLIsessionfile |
filename and path to mgmt_cli session ID file generated by login and used for all subsequent mgmt_cli operations |
Export Scripts
| APICLIpathroot |
root of path for output files |
| APICLIpathbase |
base path for output files, generally uses $APICLIpathroot and for operations time delineation can utilize the $DATE variable |
| APICLIfileoutputpre |
General prefix for the output file, prefixes the filename in the full output file path |
| APICLIfileoutputext |
File extension for operational output file, default is .txt |
| APICLIfileoutputsufix |
File suffix for the operational output file, default is $DATE.$APICLIfileoutputext so generally<date_time_group>.txt |
| APICLIJSONfileoutputext |
File extension for mgmt_cli json output file, default is .json
NOTE: this is not used in this example
|
| APICLIJSONfileoutputsufix |
File suffix for the mgmt_cli json output file, default is $DATE.$APICLIJSONfileoutputext so generally<date_time_group>.json
NOTE: this is not used in this example
|
| APICLICSVfileoutputext |
File extension for generated CSV file, default is .csv |
| APICLICSVfileoutputsufix |
File suffix for the operational output file, default is $DATE.$APICLICSVfileoutputext so generally<date_time_group>.csv
NOTE: this was purposely done for the work utilizing this example, which stipulates a defined state of CSV output to export based on the time of execution. For those wanting a generic approach, the value can be set to be more static and not include the $DATE value element.
|
|
APICLIObjectLimit
(DO NOT MODIFY THIS VALUE)
|
This is the maximum number of groups to export, providing the limit value for the mgmt_cli show groups command to populate the array of groups to export members from. The API supports a "limit" value of 0 to 500, and the default is set to 500 to ensure the maximum number of objects is collected. |
| APICLIoutput |
full file path to operational output file for later review of actions |
Import Scripts
| APICLIfileoutputpre |
General prefix for the output file, prefixes the filename in the full output file path |
| APICLIfileoutputext |
File extension for mgmt_cli json output file, default is .json |
| APICLIfileoutputsufix |
File suffix for the mgmt_cli json output file, default is $DATE.$APICLIfileoutputext so generally<date_time_group>.json |
| OutputPathRoot |
root of path for output files |
| OutputPathBase |
base path for output files, generally uses $OutputPathRoot and for operations time delineation can utilize the $DATE variable |
| CSVImportType |
mgmt_cli type for import operation, in this example it is group |
| CSVImportPathRoot |
This is the path root for the location of the CSV file to import, in the example it is a sub-directory relative to the location of the script |
| CSVImportPathFile |
This is the file name of the CSV file to import, in this case hard-coded based the CSV output generated by the export operation.
NOTE: this was purposely done for the work utilizing this example, which stipulates a defined state of CSV output to import. For those wanting a generic approach, the value can be set to be more static and not include the $DATE value element.
|
| CSVImportPath |
This is the path to the CSV file to import based on the $CSVImportPathRoot and $CSVImportPathFile variables. |
| OutputPath |
full file path to operational output file for later review of actions |
Delete Scripts
| APICLIfileoutputpre |
General prefix for the output file, prefixes the filename in the full output file path |
| APICLIfileoutputext |
File extension for mgmt_cli json output file, default is .json |
| APICLIfileoutputsufix |
File suffix for the mgmt_cli json output file, default is $DATE.$APICLIfileoutputext so generally<date_time_group>.json |
| OutputPathRoot |
root of path for output files |
| OutputPathBase |
base path for output files, generally uses $OutputPathRoot and for operations time delineation can utilize the $DATE variable |
| CSVImportType |
mgmt_cli type for import operation, in this example it is group |
| CSVImportPathRoot |
This is the path root for the location of the CSV file to import, in the example it is a sub-directory relative to the location of the script |
| CSVImportPathFile |
This is the file name of the CSV file to import, in this case hard-coded based the CSV output generated by the export operation.
NOTE: this was purposely done for the work utilizing this example, which stipulates a defined state of CSV output to import. For those wanting a generic approach, the value can be set to be more static and not include the $DATE value element.
|
| CSVImportPath |
This is the path to the CSV file to import based on the $CSVImportPathRoot and $CSVImportPathFile variables. |
| OutputPath |
full file path to operational output file for later review of actions |
Modification of the script sections to suit personal preference and requirements is strongly encouraged via the copy-paste operation.
I may be updating these later, with some harmonization of common variables required and some abstraction options via command line parameters.
Why and What for...
These scripts were developed to address a pressing need in my own basement cloud laboratory, after some issues cropped up with my migrated management server, which has an original data base starting from R70 and migrated, upgraded, imported to Multi-Domain Management, and now exported from Multi-Domain Management, which has left the system a bit wonky and questionable. By creating scripts to handle the output of objects from my existing management server, I can then use the CSV data to import to a clean, new installation, where I can start fresh, with all my objects, but probably none of the baggage or garbage from almost 9 years of lab/home use operations. It is an excellent learning opportunity and mentors like Uri Bialik help with this very much.
However, these scripts can also help with some other operations that may be necessary, probably requiring some tweaks, but the example can help a bunch for starting out, operations like:
- Duplicating group members after group import for laboratory environments, when building a from-scratch test environment, but wanting to use familiar objects
- Duplicating group members after group import to a different Domain in Multi-Domain Management, when not wanting to use global objects. This will require some adjustments to handle the Domain specification variables, but should not be rocket magic to make happen.
- Operations in Professional Services for either recovering from an exported baseline, or assisting with pre-migration testing operations, and rebuild operations
- Having a backup of objects on hand to utilize for bare-metal rebuild where a import or restore is not plausible or advisable.
Future Improvements and Extensions:
- PARTIALLY Implemented - Implementing and improving error handling, currently omitted due to time constraint and for actual mgmt_cli operations a concept for approach
- JSON Implemented - Export, Import, and [yes also] Delete of Services, but very much focused on those created by the user, not the native Services delivered in the installation
- Configuration of exported fields to CSV for more control at import
- Interactive selection of what items to export, import, or delete depending on script
- Port to Windows Power Shell scripting, once I figure out how to handled date-time-group values and jq in Windows Power Shell.
- Configuration of system objects creators to exclude, and parameter control to select specific creator objects.
- PARTIALLY Implemented - Common bash script element framework to simplify script development, which may mean more detailed script installation requirements. Later with Power Shell port also for Power Shell.
- Addition of a global [or any use] naming pre-fix to each object exported to CSV to allow alternative import, e.g. to move objects from local domain to global domain on Multi Domain Management implementation.
- Addition of specific information fields for CSV export to help identify objects for other operations (e.g. not direct import), like uid, creator, state, etc. These dumps may require more work to prepare if import is desired
Now on GITHUB
GitHub - mybasementcloud/R8x-export-import-api-scripts: Check Point R8x Export, Import, [and more] A...
FEEDBACK
If you need help with a problem using the script, please provide the following in any communication:
- R80.x version being used and management type (SMS or MDM)
- CLI level text capture (copy/paste of terminal window for execution) with the script run in "-v" (verbose mode) to ensure we see all comments and information from the execution
- Do not use the "--password" option to ensure that you're password is not provided, I don't want to know
- Ensure all command line parameters passed are correct
- any CSV or JSON output generated for the problem object(s), this can be compressed to save e-mail space
- if importing, setting, or deleting values the utilized CSV file(s)
- a short explanation of what you are trying to accomplish and what is not being met, since we may be operating on different assumptions
Code Version
Code version 0.29.02 and later
Tested on version
R80, API version 1.0
R80.10 EA, API version 1.? [2016-12 EA package]
R80.10 GA, API version 1.1
R80.10 New Kernel EA, API version 1.1
R80.10 GA for Smart-1 525, 5050, 5150, API version 1.1
R80.20 EA T354, EA T395 MDM and SMS, R80.20 GoGo EA 3.10 kernel gateways
R80.20 GA T101
R80.30 EA (Public)
Known Limitations
| Code version |
Limitation Identified |
| All up to current |
RESOLVED with v00.29.02 release
Using the -r or --root command line operation fails to execute authentication due to a limitation in approach with parametrized command line parameters for the mgmt_cli command. This issue is being escalated for technical clarification since the same values entered directly (instead of by evaluation parameter) works.
|
| All |
Using the -m or --management command line parameter is only supported when script is not executed on the actual management server host that is the target. When executing the script on the actual management server host, DO NOT use the -m or --management command line parameter with management IP address. |
Change Log
| Code version |
Key Changes |
| 0.17.25 |
Updated scripts to include comprehensive Command Line parameter handling (CLI parameters)
Added specific scripts for explicit object export to csv.
Added delete objects package for clean-up opererations
Refined operation of scripts to leverage sub routines for repeated operations with extensive parameterization to simplify adding more objects and services
Solved the way to pass variable to JQ element in export operation
Think I've solved the MDS JQ location problem with older scripts.
Providing packaged sets for export, export of specific objects, import, delete, and template shell version 0.5.0
|
| 0.21.00 |
Updated to correct issues with assumptions around how command line parameters for management, domain, and port work.
Now able to handle domain, management server
|
| 0.22.00 |
Corrected issue with changes to file naming that caused action scripts to fail |
| 0.23.00 |
New scripts provided to just count the objects the script could find in the environment (as of version 00.23.00), specifically the "cli_api_get_object_totals.sh" and "cli_api_get_object_totals_w_group_member_count.sh". Which is useful to determine if what the api sees given the input parameters and user rights is what the admin expects or should see, specifically to help with Multi-Domain operations, which may still need some more tweaking.
-P | --port <web ssl port> parameter added to support other than port 443 for management hosts
-o | --output CLI parameter renamed to -x | --export to conform with actual purpose
Check for zero size groups added to export operations for group members
Export scripts now handled > 500 objects (current maximum api limit value) and will iterate in 500 chunk steps to process object sets larger than 500 objects. json files will be broken out in 500 object sets with the name adding an increment value in sets of 500. CSV files are still single, since they are built differently.
For import added version 1.1 api support for "set-if-exists" option, with api version checking.
Added more dependency checking before start and overhauled some operations to enable easier changes.
|
| 0.24.00 |
New script added to set values of existing objects (as of version 00.24.00), specifically the "cli_api_set-update_objects_from_csv.sh" in the Import Objects set. It will process all expected input files, and skip operations where it does not find a file.
Added handling for export, import, and set for host's interface objects. NOTE: The .interfaces[].subnet-mask value is not utilized, instead explict .interfaces[].mask-length4 and .interfaces[].mask-length6 are utilized.
Export scripts not handle > 500 objects on all exported object types and formats (both CSV and JSON), this now includes the handling of group members and host interfaces, which required more programming thought to identify the approach.
Reduced the default wait time in "read -t <waittime>" from 600 to 15 and added the value $WAITTIME that can be set higher if desired. Future addition of --waittime <wait-time> and --nowait options are under consideration.
|
| 0.25.00 |
Corrected semantic issue with detecting empty object types (i.e. no objects of that type), now the CSV export will not exit the whole script if no objects are found
Extracted the Command Line Parameter Handler to a dedicated subscript to simplify editing the other scripts.
Addressed a semantic issue with web ssl-port not available before identifing the API version and not having the jq location available on MDM versus SMS installations
Prepared some main export files for future object handling expansion
NOTE: This is the last version that will have dedicated specific object handling scripts, the next script set will have an option to configure which objects to handle interactively
|
| 0.25.01 |
Correct spelling mistakes for output [an ongoing process--always]
|
| 0.27.05 |
Major overhaul of script operational approach in some functions with logical fixes to semantic approach.
Updated command line parameter handler to varriant 003. As of version 00.27.05 all cli_api*.sh scripts require "cmd_line_parameters_handler.action.common.003.sh" script to handle command line parameters! Don't forget to copy this file also.
Added ability to configure export of objects created by "System" to remove potential for import of system objects and reduce size of output. By Default this value is set to --NSO or --no-system-objects and objects created by "System" are ignored during the export of full JSON or CSV information. Standard JSON export always will do all objects found since the search for "System" objects is not possible with the supplied JSON stream. To enable export of "System" created objects, utilise the --SO or --system-objects parameter. For JSON output --NSO will generate zero length files as the dump.
Templates were also updated to reflect changes in command line handler and approach.
|
| 0.29.02 |
Resolved issues blocking use of -r parameter for local administrator authentication. Only usable on the actual host and will not work with -m option (will indicate if identified).
Corrected approach to selection of Check Point Data objects, since System creator is set by upgrade for all objects when upgrading from R77.30 and prior versions of management.
Implemented "common" folder for common operations, where future common code elements will live their own version lives, like the CLI parameter handler in the current release.
Restructured much of the internal code to fix issues and simplify future updates.
Release folder structure and pressed files shared now reflects actual approach used in operation and provides total package, not just specifics.
|
| 0.29.05 |
Moved to providing Operations and Development branches for scripts in GitHub.
General corrections and reworking operations. Last release to include development of individual scripts for import and export of explicit objects.
|
| 0.31.00 |
Removed explicit objects import and export scripts from development operations and updates, and now legacy (version 00.29.05) variants of those scripts are provided under the Development branch under ../export_import.wip/_Deprecated_Scripts.
Continued corrections and refactoring. First release of the cli_api_export_all_domains_objects.sh and cli_api_export_all_domains_objects_to_csv_files.sh scripts to handle all objects in all domains on an MDS the user has rights to.
|
| 0.33.00 |
Production release with updates and working identification of local management host web ssl port for API login configuration. Operational improvements and template corrections in preparation to adding more objects supported for CSV export and future changes to allow selection of operations.
|
