It's Here!
CPX 360 2021 Content
Check Point Harmony
Highest Level of Security for Remote Users
Important certificate update to CloudGuard Controller, CME,
and Azure HA Security Gateways
Advanced Protection for
Small and Medium Business
Important! R80 and R80.10
End Of Support around the corner (May 2021)
Forrester Study Finds
CloudGuard Generates 169% ROI
Eric_Beasley
The export, import, delete using CSV files scripts in this post, currently version 00.33.00 and later, dated 2019-01-19 and later, are intended to allow operations on an existing R80, R80.10, R80.20[.Mx], R80.30[.Mx] Check Point management server (SMS or MDM) from bash on the management server or a management server able authenticate and reach the target management server.
These scripts show examples of:
NOTE:
Scripts are provided as-is without express or implied warranty, guarantee, assumption of liability, or SLA for resolution as they are examples of how to use the API and may not always apply to the situation they are used in and are subject to limitations of the API engine or utilized Check Point management version.
At no time is this post an implied assumption of duty to change the example scripts IAW requests. If you need an API script developed, contact Check Point Professional Services or a qualified Check Point Channel Partner with DevOps capabilities.
CLI API Script approach may change at any time, which may require accessing older GitHub branches or releases.
CLI API Scripts are provided with stipulation that the implementor is capable of using CLI and understands API operations.
Check out GITHUB for latest versions : GitHub - mybasementcloud/R8x-export-import-api-scripts: Check Point R8x Export, Import, [and more] A... (version 00.33.00 as of 2019-01-19)
For direct questions, you can hit me up at ericb@checkpoint.com Response time may vary based on schedule, availability, and issue presented.
For questions about issues with scripts please provide the information identified in the FEEDBACK section at the bottom of this article.
This post includes a set of scripts in two (2) packages, a Development Branch which may be an advanced version still under construction and an Operations Branch that should work as expected. All script files end with .sh for shell and are intended for Check Point bash implementation on R80, R80.10, R80.20[.Mx], and R80.30[Mx] or later. Scripts in the packages have specific purposes and scripts call sub-scripts for extensive repeated operations and basic actions (e.g. handling CLI parameters). The packages also include specific expected default directory folders that are not created by the script action.
General Information:
<yyyy-mm-dd-hhmm-tz> is a date time group (DTG) generated at time of execution and used for the full operation of the respective script, providing consistent information for a specific script run. Example: 2017-01-05-1346CST for January 5, 2017, at 13:46 hrs CST.
Output from the scripts is directed to a sub-folder (default is a dump folder with DTG sub-folder, e.g. ./dump/<yyyy-mm-dd-hhmm-tz>) and further placed in sub-folders based on script: csv, full, standard, import, delete.
Example:
Multi-Domain Management operations scripts that handle multiple domains in their operation will create a sub-folder for the domain and then create the specific output based on that domain in the domain specific folder. Note that the scripts automatically handle the default "System Data" and "Global" domains. When executing the scripts with specific domain selected, that domain is the folder name between the DTG folder and the output folders.
Example:
NOTE: Current CSV output includes additional files used in the process that are raw data in a WIP folder, sorted raw data, csv header, and the original data with header.
NOTE: When operating on MDM, the stated domain in the -d parameter is used as a subfolder to collect the specific data for that dumped domain.
Scripts
Scripts a provided in an Operations and Development branch folder structure. Operations branch implies that these are stable operational scripts for their stated purpose, and expected to work as such under the tested versions where applicable. Development branch may include advanced, early availability version of the scripts where development is not yet complete for migration to Operations branch.
Main script types provided are MDM scripts, Session Cleanup scripts, common scripts used by other scripts, Export Import scripts, and script Templates.
Script Type (main folder) | Script Name | Script Purpose | Output formats |
|---|---|---|---|
| _templates | Templates for developing scripts with the basic capabilities of the provided version level of the template. | ||
| api_mgmt_cli_shell_template_with_cmd_line_parameters.template.<version>.sh | Template for scripts using provided approach that includes built-in command line parameter handler operations. | Text log file Output dependent on chosen template implementation | |
| api_mgmt_cli_shell_template_with_cmd_line_parameters_script.template.<versions>.sh | Template for scripts using common scripts approach for handling command line parameters | Text log file Output dependent on chosen template implementation | |
| api_mgmt_cli_shell_template_action_handler.template.<version>.sh | Template for action sub-script called by another script, with basic handling for version mismatch | log to originating script based on verbose level setting | |
| test._templates.<version>.sh | Rough testing script for validating if templates function | Text log file Output dependent on chosen template implementation | |
| common | Common scripts called by the scripts and utilized by the templates in the _template folder. This folder and expected scripts are replicated to the utilizing script folder for direct access, so will be found under _templates, export_import, and Session_Cleanup | ||
| cmd_line_parameters_handler.action.common.<level>.<version>.sh | Action sub-script called to execute operations to handle command line parameters standard to all scripts in that version | log to originating script based on verbose level setting | |
| identify_gaia_and_installation.action.common.<level>.<version>.sh | Action sub-script called to execute operations to identify version of Gaia of host and Check Point installation type | log to originating script based on verbose level setting | |
| export_import | Scripts for export, import, set, and delete operations | ||
| cli_api_export_objects.sh | Script to export all supported objects to JSON (full and standard) and CSV | JSON Full, JSON Standard, CSV Text log file | |
| cli_api_export_objects_to_json_full.sh | Script to export all supported objects to JSON full | JSON Full, JSON Standard, CSV Text log file | |
| cli_api_export_objects_to_json_standard.sh | Script to export all supported objects to JSON standard | JSON Full, JSON Standard, CSV Text log file | |
| cli_api_export_objects_actions.sh | Action sub-script to execute export of objects to JSON (full or standard) for calling script. | log to originating script based on verbose level setting | |
| cli_api_export_objects_actions_to_csv.sh | Action sub-script to execute export of objects to CSV (full or standard) for calling script. | log to originating script based on verbose level setting | |
| cli_api_export_objects_to_csv.sh | Script to export all supported objects to CSV | CSV Text log file | |
| cli_api_export_all_domains_objects.sh | Script to export all supported objects in all domains on an MDS to JSON (full and standard) and CSV | JSON Full, JSON Standard, CSV Text log file | |
| cli_api_export_all_domains_objects_to_csv.sh | Script to export all supported objects in all domains on an MDS to CSV | CSV Text log file | |
| cli_api_import_objects_from_csv.sh | Script to import all supported objects from supplied CSV files (only supplied files are processed) | JSON results Text log file | |
| cli_api_set-update_objects_from_csv.sh | Script to set (updated) all supported objects from supplied CSV files (only supplied files are processed) | JSON results Text log file | |
| cli_api_delete_objects_using_csv.sh | Script to delete all supported objects from supplied CSV files by name (only supplied files are processed) | JSON results Text log file | |
| cli_api_get_object_totals.sh | Dump the number of each object from all supported objects. | Text log file | |
| cli_api_get_object_totals_w_group_member_count.sh | Dump the number of each object from all supported objects, including count of group members per group. | Text log file | |
| api_add_csv_error_handling_to_csv_file.sh | For versions (e.g. R80.10) where CSV import and set operations require that ignore error, ignore warning, and set-if-exists are in the actual CSV file. Adds necessary columns to front of existing CSV file rows with headers. | CSV Text log file | |
| api_subpend_csv_error_handling_to_csv_files.sh | For versions (e.g. R80.10) where CSV import and set operations require that ignore error, ignore warning, and set-if-exists are in the actual CSV file. Adds necessary columns to end (back) of existing CSV file rows with headers. | CSV Text log file | |
| MDM | Multi-Domain Management general scripts | ||
| MDM_Get_Domains_List_<version>.sh | Script will generate a list of current domains on an MDS for manual re-use later | Text | |
| Session_Cleanup | Session Cleanup scripts to first list the current sessions open on a management server and also provide options to clean-up (delete) session that are not locked or pending publish operations (zero locks). | ||
| remove_zerolocks_sessions.<version>.sh | Identify and delete zero lock sessions | Text result of operation | |
| remove_zerolocks_web_api_sessions.<version>.sh | Identify and delete zero lock sessions with user web_api | Text result of operation | |
| show_zerolocks_sessions.<version>.sh | Identify zero lock sessions | Text result of operation | |
| show_zerolocks_web_api_sessions.<version>.sh | Identify zero lock sessions with user web_api | Text result of operation | |
To utilize the scripts, download the scripts from this repository post, extract the script files and directory folders [import and delete actions], then upload those files and directory folders to a working target folder location (e.g. /var/tmp/api-scripts) on the target management server where the scripts will execute from. Once uploaded to a working folder the relevant scripts are executed like any other bash script. If executing directly from the folder where the script is located use "./<script>.sh" for execution. If script modifications are made outside of Check Point Linux, it is recommended to first run "dos2unix <script>.sh" to ensure compatibility with bash shell.
Each script accepts command line parameters to control important inputs that have some defined defaults.
If the "-p <password>" parameter is not used, the user is prompted for the console user/administrators password, as in this example:
If the "-r" or "--root" parameter is used then the above prompt should be skipped as in standard mgmt_cli execution.
The scripts all (except actions sub-scripts) can take Command Line Parameters (CLI parameters). To get a dump of the active CLI parameters for a specific script run it with "--help" or "-?". Example (version 00.23.00):
Command line parameters support multiple input formats as displayed, and can be mixed and matched as needed.
This is the standard help output for cli_api_export_objects.sh script, which is the standard baseline for all scripts in this package release:
[Expert@X:0]# ./cli_api_get_object_totals.sh --help
Script: cli_api_get_object_totals Script Version: v00x23x00
API version = 1.1
./cli_api_get_object_totals.sh [-?][-v]|[-r]|[-u <admin_name>] [-p <password>]]|[-P <web ssl port>] [-m <server_IP>] [-d <domain>] [-s <session_file_filepath>]|[-x <export_path>] [-i <import_path>] [-k <delete_path>] [-l <log_path>]
Script Version: 00.23.00 Date: 2017-07-22
Standard Command Line Parameters:
Show Help -? | --help
Verbose mode -v | --verbose
Authenticate as root -r | --root
Set Console User Name -u <admin_name> | --user <admin_name> |
-u=<admin_name> | --user=<admin_name>
Set Console User password -p <password> | --password <password> |
-p=<password> | --password=<password>
Set [web ssl] Port -P <web-ssl-port> | --port <web-ssl-port> |
-P=<web-ssl-port> | --port=<web-ssl-port>
Set Management Server IP -m <server_IP> | --management <server_IP> |
-m=<server_IP> | --management=<server_IP>
Set Management Domain -d <domain> | --domain <domain> |
-d=<domain> | --domain=<domain>
Set session file path -s <session_file_filepath> |
--session-file <session_file_filepath> |
-s=<session_file_filepath> |
--session-file=<session_file_filepath>
Set log file path -l <log_path> | --log-path <log_path> |
-l=<log_path> | --log-path=<log_path>
Set export file path -x <export_path> | --export <export_path> |
-x=<export_path> | --export=<export_path>
session_file_filepath = fully qualified file path for session file
log_path = fully qualified folder path for log files
export_path = fully qualified folder path for export file
NOTE: Only use Management Server IP (-m) parameter if operating from a
different host than the management host itself.
Example: General :
]# cli_api_get_object_totals -u fooAdmin -p voodoo -P 4434 -m 192.168.1.1 -d fooville -s "/var/tmp/id.txt" -l "/var/tmp/script_dump/"
Example: Export:
]# cli_api_get_object_totals -u fooAdmin -p voodoo -P 4434 -m 192.168.1.1 -d fooville -s "/var/tmp/id.txt" -l "/var/tmp/script_dump/" -x "/var/tmp/script_dump/export/"
| Parameter Purpose | Parameter value and options | Default Value | Description |
|---|---|---|---|
| Show Help | -? | --help | n/a | Show help for script |
| Verbose Mode | -v | --verbose | n/a (not set) | Show details of operations and values during execution. bash environment variable APISCRIPTVERBOSE can be set to TRUE to run in verbose mode from start without Command Line Parameter. Example: export APISCRIPTVERBOSE=TRUE |
| Authenticate as root | -r | --root | n/a | Instead of using administrator user name and password operate as root |
| Set Console User Name | -u <admin_name> --user <admin_name> -u=<admin_name> --user=<admin_name> | administrator | Set the username of console user/administrator executing the script. <admin_name> username for console/administrator, e.g. admin |
| Set Console User password | -p <password> --password <password> -p=<password> --password=<password> | n/a | Set the password to be used for console user/administrator authentication. If not used the default operation will prompt for the console user/administrator password. <password> password to use for console user/administrator. NOTE: Entry is visible when used. |
| Set Management Server IP | -m <server_IP> --management <server_IP> -m=<server_IP> --management=<server_IP> | localhost | Set the IP address of the management server to use for this operation. <server_IP> is the TCP/IP address of the target management server, e.g. 10.10.100.66 NOTE: DO NOT USE THIS PARAMETER IF OPERATING THE SCRIPT FROM THE HOSTING MDS OR SMS, SINCE AUTHENTICTATION WILL FAIL. |
| Set Management Domain | -d <domain> --domain <domain> -d=<domain> --domain=<domain> | not set | Set the management domain to use for this operation on a Multi-Domain Management Server <domain> is the domain to use for the operation, e.g. fooville |
| Set session file path | -s <session_file_filepath> -session-file <session_file_filepath> -s=<session_file_filepath> -session-file=<session_file_filepath> | ./id.txt | Set the full path and file name to the session ID file <session_file_filepath> full path to the session ID file, e.g. /var/tmp/id.txt |
Web SSL Port NEW | -P <web_ssl_port> | --port <web_ssl_port> | -P=<web_ssl_port> | --port=<web_ssl_port> | 443 | Web SSL Port of the Management server, default is 443, but can be set explicity to address changes to multiportal, thus change to API web port. |
Session Timeout | --session-timeout <session_time_out> | 10 - 3600 seconds | Configure session timeout value for login operation executed |
System Object Export | --NSO | --no-system-objects --SO | --system-objects | --NSO | Configure export of System Objects created by "System". By Default this value is set to --NSO or --no-system-objects and objects created by "System" are ignored during the export of full JSON or CSV information. Standard JSON export always will do all objects found since the search for "System" objects is not possible with the supplied JSON stream. To enable export of "System" created objects, utilise the --SO or --system-objects parameter. For JSON output --NSO will generate zero length files as the dump. |
Log File path | -l <log_path> --log-path <log_path> -l=<log_path> --log-path =<log_path> | folder path | Set the path for log files generated by the script. <log_path> path (no following "/"), e.g. "./var/tmp/script/logs" |
Output File path | -o <output_path> --output <output_path> -o=<output_path> --output =<output_path> | folder path | Set the path for output files generated by the script. <export_path> path (no following "/"), e.g. "./var/tmp/script" |
Set export file path CHANGED | -x <export_path> --export <export_path> -x=<export_path> --export =<export_path> | ./dump/<yyyy-mm-dd-hhmm-tz> | Set the path for export files generated by the script. <export_path> path (no following "/"), e.g. "./var/tmp/script" |
| Set import file path | -i <import_path> --import-path <import_path> -i=<import_path> --import-path=<import_path> | ./import.csv | Set the path for input files required by the script. <import_path> path (no following "/"), e.g. "./var/tmp/script/input" |
| Set delete file path | -k <delete_path> --delete-path <delete_path> -k=<delete_path> --delete-path=<delete_path> | ./delete.csv | Set the path for input files required by the script to identify what to delete. <delete_path> path (no following "/"), e.g. "./var/tmp/script/input" |
| --NOWAIT | Skip waiting for key input on some operations or when running in verbose mode | ||
| --CLEANUPWIP | Remove the WIP folder created under some operational output operations (e.g. CSV exports) - PENDING IMPLEMENTATION | ||
| --NODOMAINFOLDERS | Don't generate the domain specific folders, files are domain specific so all collected - PENDING IMPLEMENTATION | ||
| --CSVEXPORTADDIGNOREERR | Automatically modify the CSV file to include the presumed ignore error, warning, or set-if-exist values - PENDING IMPLEMENTATION |
NOTE: This predicates some scripting ability and capability to use a text editor. I recommend using the dos2unix command on any updated scripts once uploaded to the target management server host to ensure compatibility.
These script examples attempt to provide some detail tailoring and configuration via variables set for the specific script. Some of these configuration values are influenced by the Command Line Parameters that can be passed to the script. This version does not make the approach overly generic (e.g. name of exported CSV file is hardcode in the import), and future versions of this example set may clearly abstract and configure command line input variables.
Key values to configure for:
Export, Import, and Delete scripts
| Variable | Definition |
|---|---|
| APICLIadmin | SmartConsole administrator name to use for operations |
| APICLIsessionfile | filename and path to mgmt_cli session ID file generated by login and used for all subsequent mgmt_cli operations |
Export Scripts
| Variable | Definition |
|---|---|
| APICLIpathroot | root of path for output files |
| APICLIpathbase | base path for output files, generally uses $APICLIpathroot and for operations time delineation can utilize the $DATE variable |
| APICLIfileoutputpre | General prefix for the output file, prefixes the filename in the full output file path |
| APICLIfileoutputext | File extension for operational output file, default is .txt |
| APICLIfileoutputsufix | File suffix for the operational output file, default is $DATE.$APICLIfileoutputext so generally<date_time_group>.txt |
| APICLIJSONfileoutputext | File extension for mgmt_cli json output file, default is .json NOTE: this is not used in this example |
| APICLIJSONfileoutputsufix | File suffix for the mgmt_cli json output file, default is $DATE.$APICLIJSONfileoutputext so generally<date_time_group>.json NOTE: this is not used in this example |
| APICLICSVfileoutputext | File extension for generated CSV file, default is .csv |
| APICLICSVfileoutputsufix | File suffix for the operational output file, default is $DATE.$APICLICSVfileoutputext so generally<date_time_group>.csv NOTE: this was purposely done for the work utilizing this example, which stipulates a defined state of CSV output to export based on the time of execution. For those wanting a generic approach, the value can be set to be more static and not include the $DATE value element. |
APICLIObjectLimit (DO NOT MODIFY THIS VALUE) | This is the maximum number of groups to export, providing the limit value for the mgmt_cli show groups command to populate the array of groups to export members from. The API supports a "limit" value of 0 to 500, and the default is set to 500 to ensure the maximum number of objects is collected. |
| APICLIoutput | full file path to operational output file for later review of actions |
Import Scripts
| Header 1 | Header 2 |
|---|---|
| APICLIfileoutputpre | General prefix for the output file, prefixes the filename in the full output file path |
| APICLIfileoutputext | File extension for mgmt_cli json output file, default is .json |
| APICLIfileoutputsufix | File suffix for the mgmt_cli json output file, default is $DATE.$APICLIfileoutputext so generally<date_time_group>.json |
| OutputPathRoot | root of path for output files |
| OutputPathBase | base path for output files, generally uses $OutputPathRoot and for operations time delineation can utilize the $DATE variable |
| CSVImportType | mgmt_cli type for import operation, in this example it is group |
| CSVImportPathRoot | This is the path root for the location of the CSV file to import, in the example it is a sub-directory relative to the location of the script |
| CSVImportPathFile | This is the file name of the CSV file to import, in this case hard-coded based the CSV output generated by the export operation. NOTE: this was purposely done for the work utilizing this example, which stipulates a defined state of CSV output to import. For those wanting a generic approach, the value can be set to be more static and not include the $DATE value element. |
| CSVImportPath | This is the path to the CSV file to import based on the $CSVImportPathRoot and $CSVImportPathFile variables. |
| OutputPath | full file path to operational output file for later review of actions |
Delete Scripts
| Header 1 | Header 2 |
|---|---|
| APICLIfileoutputpre | General prefix for the output file, prefixes the filename in the full output file path |
| APICLIfileoutputext | File extension for mgmt_cli json output file, default is .json |
| APICLIfileoutputsufix | File suffix for the mgmt_cli json output file, default is $DATE.$APICLIfileoutputext so generally<date_time_group>.json |
| OutputPathRoot | root of path for output files |
| OutputPathBase | base path for output files, generally uses $OutputPathRoot and for operations time delineation can utilize the $DATE variable |
| CSVImportType | mgmt_cli type for import operation, in this example it is group |
| CSVImportPathRoot | This is the path root for the location of the CSV file to import, in the example it is a sub-directory relative to the location of the script |
| CSVImportPathFile | This is the file name of the CSV file to import, in this case hard-coded based the CSV output generated by the export operation. NOTE: this was purposely done for the work utilizing this example, which stipulates a defined state of CSV output to import. For those wanting a generic approach, the value can be set to be more static and not include the $DATE value element. |
| CSVImportPath | This is the path to the CSV file to import based on the $CSVImportPathRoot and $CSVImportPathFile variables. |
| OutputPath | full file path to operational output file for later review of actions |
Modification of the script sections to suit personal preference and requirements is strongly encouraged via the copy-paste operation.
I may be updating these later, with some harmonization of common variables required and some abstraction options via command line parameters.
These scripts were developed to address a pressing need in my own basement cloud laboratory, after some issues cropped up with my migrated management server, which has an original data base starting from R70 and migrated, upgraded, imported to Multi-Domain Management, and now exported from Multi-Domain Management, which has left the system a bit wonky and questionable. By creating scripts to handle the output of objects from my existing management server, I can then use the CSV data to import to a clean, new installation, where I can start fresh, with all my objects, but probably none of the baggage or garbage from almost 9 years of lab/home use operations. It is an excellent learning opportunity and mentors like Uri Bialik help with this very much.
However, these scripts can also help with some other operations that may be necessary, probably requiring some tweaks, but the example can help a bunch for starting out, operations like:
If you need help with a problem using the script, please provide the following in any communication:
Code version 0.29.02 and later
R80, API version 1.0
R80.10 EA, API version 1.? [2016-12 EA package]
R80.10 GA, API version 1.1
R80.10 New Kernel EA, API version 1.1
R80.10 GA for Smart-1 525, 5050, 5150, API version 1.1
R80.20 EA T354, EA T395 MDM and SMS, R80.20 GoGo EA 3.10 kernel gateways
R80.20 GA T101
R80.30 EA (Public)
| Code version | Limitation Identified |
|---|---|
| All up to current | RESOLVED with v00.29.02 release Using the -r or --root command line operation fails to execute authentication due to a limitation in approach with parametrized command line parameters for the mgmt_cli command. This issue is being escalated for technical clarification since the same values entered directly (instead of by evaluation parameter) works. |
| All | Using the -m or --management command line parameter is only supported when script is not executed on the actual management server host that is the target. When executing the script on the actual management server host, DO NOT use the -m or --management command line parameter with management IP address. |
| Code version | Key Changes |
|---|---|
| 0.17.25 | Updated scripts to include comprehensive Command Line parameter handling (CLI parameters) Added specific scripts for explicit object export to csv. Added delete objects package for clean-up opererations Refined operation of scripts to leverage sub routines for repeated operations with extensive parameterization to simplify adding more objects and services Solved the way to pass variable to JQ element in export operation Think I've solved the MDS JQ location problem with older scripts. Providing packaged sets for export, export of specific objects, import, delete, and template shell version 0.5.0 |
| 0.21.00 | Updated to correct issues with assumptions around how command line parameters for management, domain, and port work. Now able to handle domain, management server |
| 0.22.00 | Corrected issue with changes to file naming that caused action scripts to fail |
| 0.23.00 | New scripts provided to just count the objects the script could find in the environment (as of version 00.23.00), specifically the "cli_api_get_object_totals.sh" and "cli_api_get_object_totals_w_group_member_count.sh". Which is useful to determine if what the api sees given the input parameters and user rights is what the admin expects or should see, specifically to help with Multi-Domain operations, which may still need some more tweaking. -P | --port <web ssl port> parameter added to support other than port 443 for management hosts -o | --output CLI parameter renamed to -x | --export to conform with actual purpose Check for zero size groups added to export operations for group members Export scripts now handled > 500 objects (current maximum api limit value) and will iterate in 500 chunk steps to process object sets larger than 500 objects. json files will be broken out in 500 object sets with the name adding an increment value in sets of 500. CSV files are still single, since they are built differently. For import added version 1.1 api support for "set-if-exists" option, with api version checking. Added more dependency checking before start and overhauled some operations to enable easier changes. |
| 0.24.00 | New script added to set values of existing objects (as of version 00.24.00), specifically the "cli_api_set-update_objects_from_csv.sh" in the Import Objects set. It will process all expected input files, and skip operations where it does not find a file. Added handling for export, import, and set for host's interface objects. NOTE: The .interfaces[].subnet-mask value is not utilized, instead explict .interfaces[].mask-length4 and .interfaces[].mask-length6 are utilized. Export scripts not handle > 500 objects on all exported object types and formats (both CSV and JSON), this now includes the handling of group members and host interfaces, which required more programming thought to identify the approach. Reduced the default wait time in "read -t <waittime>" from 600 to 15 and added the value $WAITTIME that can be set higher if desired. Future addition of --waittime <wait-time> and --nowait options are under consideration. |
| 0.25.00 | Corrected semantic issue with detecting empty object types (i.e. no objects of that type), now the CSV export will not exit the whole script if no objects are found Extracted the Command Line Parameter Handler to a dedicated subscript to simplify editing the other scripts. Addressed a semantic issue with web ssl-port not available before identifing the API version and not having the jq location available on MDM versus SMS installations Prepared some main export files for future object handling expansion NOTE: This is the last version that will have dedicated specific object handling scripts, the next script set will have an option to configure which objects to handle interactively |
| 0.25.01 | Correct spelling mistakes for output [an ongoing process--always] |
| 0.27.05 | Major overhaul of script operational approach in some functions with logical fixes to semantic approach. Updated command line parameter handler to varriant 003. As of version 00.27.05 all cli_api*.sh scripts require "cmd_line_parameters_handler.action.common.003.sh" script to handle command line parameters! Don't forget to copy this file also. Added ability to configure export of objects created by "System" to remove potential for import of system objects and reduce size of output. By Default this value is set to --NSO or --no-system-objects and objects created by "System" are ignored during the export of full JSON or CSV information. Standard JSON export always will do all objects found since the search for "System" objects is not possible with the supplied JSON stream. To enable export of "System" created objects, utilise the --SO or --system-objects parameter. For JSON output --NSO will generate zero length files as the dump. Templates were also updated to reflect changes in command line handler and approach. |
| 0.29.02 | Resolved issues blocking use of -r parameter for local administrator authentication. Only usable on the actual host and will not work with -m option (will indicate if identified). Corrected approach to selection of Check Point Data objects, since System creator is set by upgrade for all objects when upgrading from R77.30 and prior versions of management. Implemented "common" folder for common operations, where future common code elements will live their own version lives, like the CLI parameter handler in the current release. Restructured much of the internal code to fix issues and simplify future updates. Release folder structure and pressed files shared now reflects actual approach used in operation and provides total package, not just specifics. |
| 0.29.05 | Moved to providing Operations and Development branches for scripts in GitHub. General corrections and reworking operations. Last release to include development of individual scripts for import and export of explicit objects. |
| 0.31.00 | Removed explicit objects import and export scripts from development operations and updates, and now legacy (version 00.29.05) variants of those scripts are provided under the Development branch under ../export_import.wip/_Deprecated_Scripts. Continued corrections and refactoring. First release of the cli_api_export_all_domains_objects.sh and cli_api_export_all_domains_objects_to_csv_files.sh scripts to handle all objects in all domains on an MDS the user has rights to. |
| 0.33.00 | Production release with updates and working identification of local management host web ssl port for API login configuration. Operational improvements and template corrections in preparation to adding more objects supported for CSV export and future changes to allow selection of operations. |
Hi,
Thank you for export import script.
I have a question about script.
Export object limit 500.
export APICLIObjectLimit=500
I increased the value through script.(vi editor)
Script did not work after editing
How can I increase the export&import object limit?
Thank you
Best Regards
Suleyman Toreli
SECURRENT
Eric_Beasley
Hello,
API currently has a object limit value of 1-500, so can't go higher than 500 as far as I can tell. I'll see if I can do something to determine number of objects and slice the content into 500 objects slices to get output to csv for all the objects. Import operations via add don't apparently use the limit entry value so don't have the problem, but show operations do.
Will look into this for the next version of script, version 00.23.00 or later.
Eric
Eric_Beasley
Hello again,
Version 00.23.00 now handles more than 500 objects in 500 object slices for json output and it was tested with ~1200 objects to csv (single file). This works for all objects except for group members, where only 500 groups are handled, but I don't have data to test whether it can handle more than 500 members for a group, since that handling is different from the limit based show "objects" operation.
Check GitHub for version 00.23.00 or later to test it out.
Also added script "cli_api_get_object_totals.sh" and "cli_api_get_object_totals_w_group_member_count.sh" that will just provide a count of objects and a log of operations to query group members. These are located under the Export Objects folder or 7z file.
Eric
Eric_Beasley
And one more update on this question. Script version 00.24.00 now handles all processed objects for export with > 500 objects, even the tougher group members and new host interfaces settings.
Eric
Phuong Nguyen Le Phi Thanh Tung Good script!
Hello!
If there is no object of 1 category (for example no address range) the script terminates. Are you aware of that?
Otherwise it works really good, thank you!
Eric_Beasley
Can you e-mail me an execution where it terminates like that? Also run it with the -v (verbose) parameter for more details, I need to see what the issue is.
Eric_Beasley
OK, after some testing on a new system, identified the issue and fixed it and the initial problem on an MDS in version 00.25.00 posted to GitHub.
Fantastic work. Just what we needed.
Just a heads up for the author that there is a spelling error in the output. This does not break any functionality at all but thought I would share this.
Dump times to /home/admin/results/2017-08-31-1518BST/full/dump_times_full_2017-08-31-1518BST.json
Now processing up to next 500 objects starting with object 0 of 3 remainging!
Should be "remaining" not "remainging" . Perhaps this could be fixed upstream ?
Eric_Beasley
Hi John,
Thanks for that, I’ll fix that in v00.25.01, typing fast can lead to great typos, and a standard text editor doesn’t spell check—so a little one like this is easy to miss. It’s more fun when you flub a command argument or variable name, that usually comes back around in testing.
Eric
Hello,
I have an issues with cli_api_export_object_group-members_to_csv.sh script. If I set APICLIObjectLimit=100, then:
- Each particular pair ("name","members.add") in group-members_standard_csv.csv is listed 6x, e.g.:
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
- List in group-members_standard_csv.csv is not complete, it contains just a part of the all records (I know, our db is quite big). Despite the script finishes with no error.
Could you have a look on it, please? Script version is 00.25.00.
Thanks
Zbynek
Eric_Beasley
Hi,
Why are you modifying the limit value, which identifies how many groups will be added to the array that the group members are searched from?
Leave the value where it was and also use the latest version of the scripts.
Group Members is built by reading the full list of groups into an array, then processing the array of groups for each group, for its respective members, such that we get all of the members listed for each group.
The proper example of a group members CSV file:
"name","members.add"
"TEST_GROUP_01","test-addr-range-01-062.128.100.000-062.128.101.255"
"TEST_GROUP_01","test-addr-range-02-010.128.100.000-010.128.101.255"
"TEST_GROUP_01","test-addr-range-03-010.128.102.000-010.128.103.255"
"TEST_GROUP_01","test-addr-range-04-010.128.104.000-010.128.105.255"
"TEST_GROUP_01","test-addr-range-05-010.128.106.000-010.128.107.255"
"TEST_GROUP_02","TEST-HOST-01"
"TEST_GROUP_02","TEST-HOST-02"
"TEST_GROUP_02","TEST-HOST-03"
"TEST_GROUP_02","TEST-HOST-04"
"TEST_GROUP_02","TEST-HOST-05"
"TEST_GROUP_03","TEST-HOST-02"
"TEST_GROUP_03","TEST-HOST-04"
"TEST_GROUP_04","netw_test_LAN_01"
"TEST_GROUP_04","netw_test_LAN_02"
"TEST_GROUP_04","netw_test_LAN_03"
"TEST_GROUP_04","netw_test_LAN_04"
"TEST_GROUP_04","netw_test_LAN_05"
"TEST_GROUP_05",".1.test.domain.local"
"TEST_GROUP_05",".2.test.domain.local"
"TEST_GROUP_05",".3.test.domain.local"
For best results in determining if and what is going wrong, provide a executing of the script with the “-v” (verbose mode) and a full copy of the script execution from the CLI. I recommend not using the “—password” option, but entering that at the prompt when the mgmt_cli login executes, since I don’t need to know any passwords.
That should indicate if the processing is happening as expected.
Latest version is updated on GitHub: https://github.com/mybasementcloud/R8x-export-import-api-scripts, right now that is version 00.25.01
Versions prior to 00.23.00 do not execute operations for more than either 500 (API limitation) or the set APICLIObjectLimit value objects, and version 00.24.00 and higher handle group members and host interfaces for more than 500 groups or hosts, respectively.
BR
Eric Beasley
Security Engineer, North America Channel Sales
Check Point Software Technologies Ltd.
Cell: +1 708-224-7724 E-mail: ericb@checkpoint.com
Additional Contact Informaton:
Support : +1 (888) 361 5030 or +1 (972) 444 6600 Technical Assistance Center (TAC)
Partner Sales Assistance: (866) 488-6689 or cpp@checkpoint.com
Account Services: +1 (972) 444-6600, option 5 or AccountServices@checkpoint.com
Licensing Advocacy: +1 (800) 429-4391 or advocacy@checkpoint.com
Inside Sales: +1 (650) 628-2225
Check Point Incident Response Team: (866) 923-0907 > initial consultation is free
emergency-response@checkpoint.com<mailto:emergency-response@checkpoint.com> > you do not need to be an existing Check Point customer
Hi Eric,
first of all thank you for the great scripts!
I wanted to use the Script cli_api_export_object_group-members_to_csv.sh, too. But I have the same Problems like Zbynek. To get full information I run the script in Verbose Mode and i noticed, that the script stop exporting exactly after 1000 Elements. Perhaps there is a Limit of 1000 anywhere?
Best regards
Michael
I am modifying the APICLIObjectLimit because the default value (500) crashes the API java server with our DB. And we do not have any fix from CP yet. So the only chance for me is to decrease the value to 100 now, which seems to be safe.
Regards
Zbynek
Hi Eric. I must admit I am no coder, I just do engineering. I'm trying to export R80.10 objects and move it to another 80.10 setup. When I run your script cli_api_export_objects_to_csv.sh v00.26.01 I get the following error. Any help would be appreciated:
Done dumping hosts
Use array of hosts to generate host interfaces CSV
!!!! Error data file missing : ./dump/2017-11-06-1111UTC/csv/wip/hosts_full_csv.csv.data
Terminating!
Eric_Beasley
This issue was identified and fixed in 00.26.07 and again tested in 00.27.05.
Joshua_Hatter
Gotta say kudos just for the documentation and write up. Great work!
Great stuff man, what is ETA for services exports?
I successfully produced an export and then trimmed down the csv (hosts_full_csv.csv) output to just one host as an experiment. I ran the import script and the single host was not imported. It output a file called "dump_add_host_json" that contained the text
Line 2: {
"code" : "generic_err_invalid_parameter",
"message" : "Parameter [auto-rule] value is not valid (full path: nat-settings.auto-rule)"
}
{ "response" : []}
What am I doing wrong? Is partial import not supported?
Also, if I do a full import, what is the overwrite setting for existing objects?
Thanks!
Thanks for these awesome scripts! They have come in super helpful in a conversion project!
I do have one request:
Would it be possible to get the script updated to also export any service objects and service groups? We need to compare these between rulebases and think it will most easily be done in a csv file. I can export services through SmartConsole, but the groups are what don't come through correctly.
Thanks again for all of your hard work!
I started using this script with not much knowledge of how to use the API, and was able to make it work despite that. Awesome! I had a couple questions going into this that I have now come up with answers to, so I thought I'd add them. Apologies in advance for any dimness.
Can I use this script to just import selected objects?
Basically, no. But you can edit down an outputted csv from "cli_api_import_objects_from_csv.sh" and then import it manually with a pretty simple command. I dove into the script and found the basic behavior.
mgmt_cli add [type] --batch [path to modified csv file for that type from the export script] set-if-exists true ignore-warnings true ignore-errors true --ignore-errors true -f json -s [path to session file]
Values for [type]:
host | network | group | group-with-exclusion | address-range | dns-domain | security-zone | dynamic-object | application-site | application-site-category | application-site-group | tag | simple-gateway | time | time-group | access-role | opsec-application | service-tcp | service-udp | service-icmp | service-icmp6 | service-sctp | service-other | service-dce-rpc | service-rpc | service-group
Does this script overwrite existing objects?
"set-if-exists true" is declared in the script, and will modify existing objects to match the imported form. Changing the line
export MgmtCLI_Add_OpParms="set-if-exists true $MgmtCLI_IgnoreErr_OpParms $MgmtCLI_Base_OpParms"
to
export MgmtCLI_Add_OpParms="set-if-exists false $MgmtCLI_IgnoreErr_OpParms $MgmtCLI_Base_OpParms"
should leave existing objects alone.
Hope this isn't misleading anyone.
Hi,
just tested those tools and exporting importing seems to work pretty well.
I am just wondering that not all objects have been imported from dumped csv files.
Csv files have exactly the number of objects of the original smartcenter, so exporting was fine.
I was searching around but did not see in the console output nor in logfile why some objects were not imported.
870 of 877 host objects have been imported and 256 of 300 nets.
Would be nice to know if there any issues in the data to be imported so i could check and modify the objects in the csv prior to import.
Any hints would be appreciated
cheers
Vincent
Hi all,
did anybody already test it on R80.40 ?
I assume it should work, right?
br
Vincent
This script is so good, appreciate the effort taken and sharing with all. Do we have any update on getting the services plus services groups also?
Excellent scripts! Thank you!
Just a small one, expanding groups with space in its name during the cli_api_export_objects_to_csv script throws an error (target R80.40):
Group Franking Machines number of members = 2
Error: The parameters of show-group command should be provided in pairs (key and value). You have provided an odd number of parameters which suggests that you are probably missing a parameter.
Thanks Jamie
The export, import, delete using CSV files scripts in this post, currently version 00.33.00 and later, dated 2019-01-19 and later, are intended to allow operations on an existing R80, R80.10, R80.20[.Mx], R80.30[.Mx] Check Point management server (SMS or MDM) from bash on the management server or a management server able authenticate and reach the target management server.
These scripts show examples of:
About CheckMates
Learn Check Point
Advanced Learning
WELCOME TO THE FUTURE OF CYBER SECURITY