topic Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.3 in API / CLI Discussion

CLI API Example for exporting, importing, and deleting different objects using CSV files

Eric_Beasley — Thu, 16 Mar 2023 15:59:40 GMT

Check out GITHUB for latest version v00.60.12.100.750 as of 2023-03-14:

GitHub - mybasementcloud/R8x-export-import-api-scripts: Check Point R8x Export, Import, [and more] API scripts for bash …

https://github.com/mybasementcloud/R8x-export-import-api-scripts/releases

Available documentation is provided in the in the GitHub Repository, please check the latest releases document and the README.MD file.

Version v00.60.12.100.750 is the last planned release under that GitHub repository, and all new development is going here:
mybasementcloud/R8X_mgmt_cli_API_bash_scripts

The post below is not manageable and won't be updated any more, all specifics and improved documentation is provided on GitHub.

Eric @Eric_Beasley

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Suleyman_B_T — Mon, 17 Jul 2017 09:02:51 GMT

Hi,

Thank you for export import script.

I have a question about script.

Export object limit 500.

export APICLIObjectLimit=500

I increased the value through script.(vi editor)

Script did not work after editing

How can I increase the export&import object limit?

Thank you

Best Regards

Suleyman Toreli

SECURRENT

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Eric_Beasley — Thu, 20 Jul 2017 16:36:11 GMT

Hello,

API currently has a object limit value of 1-500, so can't go higher than 500 as far as I can tell. I'll see if I can do something to determine number of objects and slice the content into 500 objects slices to get output to csv for all the objects. Import operations via add don't apparently use the limit entry value so don't have the problem, but show operations do.

Will look into this for the next version of script, version 00.23.00 or later.

Eric

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Eric_Beasley — Sat, 22 Jul 2017 06:57:05 GMT

Hello again,

Version 00.23.00 now handles more than 500 objects in 500 object slices for json output and it was tested with ~1200 objects to csv (single file). This works for all objects except for group members, where only 500 groups are handled, but I don't have data to test whether it can handle more than 500 members for a group, since that handling is different from the limit based show "objects" operation.

Check GitHub for version 00.23.00 or later to test it out.

Also added script "cli_api_get_object_totals.sh" and "cli_api_get_object_totals_w_group_member_count.sh" that will just provide a count of objects and a log of operations to query group members. These are located under the Export Objects folder or 7z file.

Eric

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Eric_Beasley — Fri, 04 Aug 2017 04:15:21 GMT

And one more update on this question. Script version 00.24.00 now handles all processed objects for export with > 500 objects, even the tougher group members and new host interfaces settings.

Eric

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Tung_Nguyen_Son — Sun, 20 Aug 2017 02:53:15 GMT

Phuong Nguyen Le Phi Thanh Tung Good script!

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Julien_Tissot — Sun, 27 Aug 2017 10:23:50 GMT

Hello!

If there is no object of 1 category (for example no address range) the script terminates. Are you aware of that?

Otherwise it works really good, thank you!

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Eric_Beasley — Mon, 28 Aug 2017 16:01:07 GMT

Can you e-mail me an execution where it terminates like that? Also run it with the -v (verbose) parameter for more details, I need to see what the issue is.

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Eric_Beasley — Tue, 29 Aug 2017 06:25:54 GMT

OK, after some testing on a new system, identified the issue and fixed it and the initial problem on an MDS in version 00.25.00 posted to GitHub.

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

John_Tammaro1 — Thu, 31 Aug 2017 14:20:13 GMT

Fantastic work. Just what we needed.

Just a heads up for the author that there is a spelling error in the output. This does not break any functionality at all but thought I would share this.

Dump times to /home/admin/results/2017-08-31-1518BST/full/dump_times_full_2017-08-31-1518BST.json
Now processing up to next 500 objects starting with object 0 of 3 remainging!

Should be "remaining" not "remainging" . Perhaps this could be fixed upstream ?

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Eric_Beasley — Thu, 31 Aug 2017 15:26:12 GMT

Hi John,

Thanks for that, I’ll fix that in v00.25.01, typing fast can lead to great typos, and a standard text editor doesn’t spell check—so a little one like this is easy to miss. It’s more fun when you flub a command argument or variable name, that usually comes back around in testing.

Eric

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Zbynek_Michl — Fri, 01 Sep 2017 09:31:03 GMT

Hello,

I have an issues with cli_api_export_object_group-members_to_csv.sh script. If I set APICLIObjectLimit=100, then:

- Each particular pair ("name","members.add") in group-members_standard_csv.csv is listed 6x, e.g.:

MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"

- List in group-members_standard_csv.csv is not complete, it contains just a part of the all records (I know, our db is quite big). Despite the script finishes with no error.

Could you have a look on it, please? Script version is 00.25.00.

Thanks

Zbynek

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Eric_Beasley — Tue, 05 Sep 2017 16:59:59 GMT

Hi,

Why are you modifying the limit value, which identifies how many groups will be added to the array that the group members are searched from?

Leave the value where it was and also use the latest version of the scripts.

Group Members is built by reading the full list of groups into an array, then processing the array of groups for each group, for its respective members, such that we get all of the members listed for each group.

The proper example of a group members CSV file:

"name","members.add"

"TEST_GROUP_01","test-addr-range-01-062.128.100.000-062.128.101.255"

"TEST_GROUP_01","test-addr-range-02-010.128.100.000-010.128.101.255"

"TEST_GROUP_01","test-addr-range-03-010.128.102.000-010.128.103.255"

"TEST_GROUP_01","test-addr-range-04-010.128.104.000-010.128.105.255"

"TEST_GROUP_01","test-addr-range-05-010.128.106.000-010.128.107.255"

"TEST_GROUP_02","TEST-HOST-01"

"TEST_GROUP_02","TEST-HOST-02"

"TEST_GROUP_02","TEST-HOST-03"

"TEST_GROUP_02","TEST-HOST-04"

"TEST_GROUP_02","TEST-HOST-05"

"TEST_GROUP_03","TEST-HOST-02"

"TEST_GROUP_03","TEST-HOST-04"

"TEST_GROUP_04","netw_test_LAN_01"

"TEST_GROUP_04","netw_test_LAN_02"

"TEST_GROUP_04","netw_test_LAN_03"

"TEST_GROUP_04","netw_test_LAN_04"

"TEST_GROUP_04","netw_test_LAN_05"

"TEST_GROUP_05",".1.test.domain.local"

"TEST_GROUP_05",".2.test.domain.local"

"TEST_GROUP_05",".3.test.domain.local"

For best results in determining if and what is going wrong, provide a executing of the script with the “-v” (verbose mode) and a full copy of the script execution from the CLI. I recommend not using the “—password” option, but entering that at the prompt when the mgmt_cli login executes, since I don’t need to know any passwords.

That should indicate if the processing is happening as expected.

Latest version is updated on GitHub: https://github.com/mybasementcloud/R8x-export-import-api-scripts, right now that is version 00.25.01

Versions prior to 00.23.00 do not execute operations for more than either 500 (API limitation) or the set APICLIObjectLimit value objects, and version 00.24.00 and higher handle group members and host interfaces for more than 500 groups or hosts, respectively.

Eric Beasley

Security Engineer, North America Channel Sales

Check Point Software Technologies Ltd.

Cell: +1 708-224-7724 E-mail: ericb@checkpoint.com

Additional Contact Informaton:

Support : +1 (888) 361 5030 or +1 (972) 444 6600 Technical Assistance Center (TAC)

Partner Sales Assistance: (866) 488-6689 or cpp@checkpoint.com

Account Services: +1 (972) 444-6600, option 5 or AccountServices@checkpoint.com

Licensing Advocacy: +1 (800) 429-4391 or advocacy@checkpoint.com

Inside Sales: +1 (650) 628-2225

Check Point Incident Response Team: (866) 923-0907 > initial consultation is free

emergency-response@checkpoint.com<mailto:emergency-response@checkpoint.com> > you do not need to be an existing Check Point customer

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Michael_Vogler — Wed, 06 Sep 2017 11:16:58 GMT

Hi Eric,

first of all thank you for the great scripts!

I wanted to use the Script cli_api_export_object_group-members_to_csv.sh, too. But I have the same Problems like Zbynek. To get full information I run the script in Verbose Mode and i noticed, that the script stop exporting exactly after 1000 Elements. Perhaps there is a Limit of 1000 anywhere?

Best regards

Michael

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Zbynek_Michl — Wed, 27 Sep 2017 14:47:25 GMT

I am modifying the APICLIObjectLimit because the default value (500) crashes the API java server with our DB. And we do not have any fix from CP yet. So the only chance for me is to decrease the value to 100 now, which seems to be safe.

Regards

Zbynek

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Paul_Warnagiris — Mon, 06 Nov 2017 11:26:01 GMT

Hi Eric. I must admit I am no coder, I just do engineering. I'm trying to export R80.10 objects and move it to another 80.10 setup. When I run your script cli_api_export_objects_to_csv.sh v00.26.01 I get the following error. Any help would be appreciated:

Done dumping hosts

Use array of hosts to generate host interfaces CSV

!!!! Error data file missing : ./dump/2017-11-06-1111UTC/csv/wip/hosts_full_csv.csv.data
Terminating!

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Eric_Beasley — Tue, 06 Mar 2018 20:19:35 GMT

This issue was identified and fixed in 00.26.07 and again tested in 00.27.05.

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Joshua_Hatter — Thu, 08 Mar 2018 01:35:13 GMT

Gotta say kudos just for the documentation and write up. Great work!

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.33.00 and later)

Martin_Valenta — Fri, 20 Jul 2018 10:11:36 GMT

Great stuff man, what is ETA for services exports?

Re: CLI API Example for exporting, importing, and deleting different objects using CSV files (v 00.3

Brandon_Cotter — Fri, 28 Jun 2019 00:39:47 GMT

I successfully produced an export and then trimmed down the csv (hosts_full_csv.csv) output to just one host as an experiment. I ran the import script and the single host was not imported. It output a file called "dump_add_host_json" that contained the text

Line 2: {
"code" : "generic_err_invalid_parameter",
"message" : "Parameter [auto-rule] value is not valid (full path: nat-settings.auto-rule)"
}

{ "response" : []}

What am I doing wrong? Is partial import not supported?

Also, if I do a full import, what is the overwrite setting for existing objects?

Thanks!