CLI API Example for exporting, importing, and deleting different objects using CSV files

[Eric_Beasley]

Check out GITHUB for latest version v00.60.12.100.750 as of 2023-03-14:

GitHub - mybasementcloud/R8x-export-import-api-scripts: Check Point R8x Export, Import, [and more] A... 

https://github.com/mybasementcloud/R8x-export-import-api-scripts/releases

Available documentation is provided in the in the GitHub Repository, please check the latest releases document and the README.MD file.

Version v00.60.12.100.750 is the last planned release under that GitHub repository, and all new development is going here:
mybasementcloud/R8X_mgmt_cli_API_bash_scripts

The post below is not manageable and won't be updated any more, all specifics and improved documentation is provided on GitHub.

BR

Eric @Eric_Beasley 

[Suleyman_B_T]

Hi,

Thank you for export import script.

I have a question about script.

Export object limit 500.

export APICLIObjectLimit=500

I increased the value through script.(vi editor)

Script did not work after editing

How can I increase the export&import object limit?

Thank you

Best Regards

Suleyman Toreli

SECURRENT

[Eric_Beasley]

Hello,

API currently has a object limit value of 1-500, so can't go higher than 500 as far as I can tell.  I'll see if I can do something to determine number of objects and slice the content into 500 objects slices to get output to csv for all the objects.  Import operations via add don't apparently use the limit entry value so don't have the problem, but show operations do.

Will look into this for the next version of script, version 00.23.00 or later.

Eric

[Eric_Beasley]

Hello again,

Version 00.23.00 now handles more than 500 objects in 500 object slices for json output and it was tested with ~1200 objects to csv (single file).  This works for all objects except for group members, where only 500 groups are handled, but I don't have data to test whether it can handle more than 500 members for a group, since that handling is different from the limit based show "objects" operation.

Check GitHub for version 00.23.00 or later to test it out.

Also added script "cli_api_get_object_totals.sh" and "cli_api_get_object_totals_w_group_member_count.sh" that will just provide a count of objects and a log of operations to query group members.  These are located under the Export Objects folder or 7z file.

Eric

[Eric_Beasley]

And one more update on this question.  Script version 00.24.00 now handles all processed objects for export with > 500 objects, even the tougher group members and new host interfaces settings.

Eric

[Tung_Nguyen_Son]

 Phuong Nguyen Le  Phi Thanh Tung Good script!

[Julien_Tissot]

Hello!

If there is no object of 1 category (for example no address range) the script terminates. Are you aware of that?

Otherwise it works really good, thank you!

[Eric_Beasley]

Can you e-mail me an execution where it terminates like that?  Also run it with the -v (verbose) parameter for more details, I need to see what the issue is.

[Eric_Beasley]

OK, after some testing on a new system, identified the issue and fixed it and the initial problem on an MDS in version 00.25.00 posted to GitHub.

[John_Tammaro1]

Fantastic work. Just what we needed.

Just a heads up for the author that there is a spelling error in the output. This does not break any functionality at all but thought I would share this.

Dump times to /home/admin/results/2017-08-31-1518BST/full/dump_times_full_2017-08-31-1518BST.json
Now processing up to next 500 objects starting with object 0 of 3 remainging!

Should be "remaining" not "remainging" . Perhaps this could be fixed upstream ?

[Eric_Beasley]

Hi John,

Thanks for that, I’ll fix that in v00.25.01, typing fast can lead to great typos, and a standard text editor doesn’t spell check—so a little one like this is easy to miss. It’s more fun when you flub a command argument or variable name, that usually comes back around in testing.

Eric

[Zbynek_Michl]

Hello,

I have an issues with cli_api_export_object_group-members_to_csv.sh script. If I set APICLIObjectLimit=100, then:

- Each particular pair ("name","members.add") in group-members_standard_csv.csv is listed 6x, e.g.:

MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"

- List in group-members_standard_csv.csv is not complete, it contains just a part of the all records (I know, our db is quite big). Despite the script finishes with no error.

Could you have a look on it, please? Script version is 00.25.00.

Thanks

Zbynek

[Eric_Beasley]

Hi,

Why are you modifying the limit value, which identifies how many groups will be added to the array that the group members are searched from?

Leave the value where it was and also use the latest version of the scripts.

Group Members is built by reading the full list of groups into an array, then processing the array of groups for each group, for its respective members, such that we get all of the members listed for each group.

The proper example of a group members CSV file:

"name","members.add"

"TEST_GROUP_01","test-addr-range-01-062.128.100.000-062.128.101.255"

"TEST_GROUP_01","test-addr-range-02-010.128.100.000-010.128.101.255"

"TEST_GROUP_01","test-addr-range-03-010.128.102.000-010.128.103.255"

"TEST_GROUP_01","test-addr-range-04-010.128.104.000-010.128.105.255"

"TEST_GROUP_01","test-addr-range-05-010.128.106.000-010.128.107.255"

"TEST_GROUP_02","TEST-HOST-01"

"TEST_GROUP_02","TEST-HOST-02"

"TEST_GROUP_02","TEST-HOST-03"

"TEST_GROUP_02","TEST-HOST-04"

"TEST_GROUP_02","TEST-HOST-05"

"TEST_GROUP_03","TEST-HOST-02"

"TEST_GROUP_03","TEST-HOST-04"

"TEST_GROUP_04","netw_test_LAN_01"

"TEST_GROUP_04","netw_test_LAN_02"

"TEST_GROUP_04","netw_test_LAN_03"

"TEST_GROUP_04","netw_test_LAN_04"

"TEST_GROUP_04","netw_test_LAN_05"

"TEST_GROUP_05",".1.test.domain.local"

"TEST_GROUP_05",".2.test.domain.local"

"TEST_GROUP_05",".3.test.domain.local"

For best results in determining if and what is going wrong, provide a executing of the script with the “-v” (verbose mode) and a full copy of the script execution from the CLI. I recommend not using the “—password” option, but entering that at the prompt when the mgmt_cli login executes, since I don’t need to know any passwords.

That should indicate if the processing is happening as expected.

Latest version is updated on GitHub: https://github.com/mybasementcloud/R8x-export-import-api-scripts, right now that is version 00.25.01

Versions prior to 00.23.00 do not execute operations for more than either 500 (API limitation) or the set APICLIObjectLimit value objects, and version 00.24.00 and higher handle group members and host interfaces for more than 500 groups or hosts, respectively.

BR

Eric Beasley

Security Engineer, North America Channel Sales

Check Point Software Technologies Ltd.

Cell: +1 708-224-7724 E-mail: ericb@checkpoint.com

Additional Contact Informaton:

Support : +1 (888) 361 5030 or +1 (972) 444 6600 Technical Assistance Center (TAC)

Partner Sales Assistance: (866) 488-6689 or cpp@checkpoint.com

Account Services: +1 (972) 444-6600, option 5 or AccountServices@checkpoint.com

Licensing Advocacy: +1 (800) 429-4391 or advocacy@checkpoint.com

Inside Sales: +1 (650) 628-2225

Check Point Incident Response Team: (866) 923-0907 > initial consultation is free

emergency-response@checkpoint.com<mailto:emergency-response@checkpoint.com> > you do not need to be an existing Check Point customer

[Michael_Vogler]

Hi Eric, 

first of all thank you for the great scripts!

I wanted to use the Script cli_api_export_object_group-members_to_csv.sh, too. But I have the same Problems like Zbynek. To get full information I run the script in Verbose Mode and i noticed, that the script stop exporting exactly after 1000 Elements. Perhaps there is a Limit of 1000 anywhere?

Best regards

Michael

[Zbynek_Michl]

I am modifying the APICLIObjectLimit because the default value (500) crashes the API java server with our DB. And we do not have any fix from CP yet. So the only chance for me is to decrease the value to 100 now, which seems to be safe.

Regards

Zbynek

[Paul_Warnagiris]

Hi Eric.  I must admit I am no coder, I just do engineering.  I'm trying to export R80.10 objects and move it to another 80.10 setup.  When I run your script cli_api_export_objects_to_csv.sh v00.26.01  I get the following error.  Any help would be appreciated:

Done dumping hosts

Use array of hosts to generate host interfaces CSV

!!!! Error data file missing : ./dump/2017-11-06-1111UTC/csv/wip/hosts_full_csv.csv.data
Terminating!

[Eric_Beasley]

This issue was identified and fixed in 00.26.07 and again tested in 00.27.05.

[Joshua_Hatter]

Gotta say kudos just for the documentation and write up. Great work!

[Martin_Valenta]

Great stuff man, what is ETA for services exports?

[Brandon_Cotter]

I successfully produced an export and then trimmed down the csv (hosts_full_csv.csv) output to just one host as an experiment. I ran the import script and the single host was not imported. It output a file called "dump_add_host_json" that contained the text

Line 2: {
"code" : "generic_err_invalid_parameter",
"message" : "Parameter [auto-rule] value is not valid (full path: nat-settings.auto-rule)"
}

{ "response" : []}

What am I doing wrong? Is partial import not supported?

Also, if I do a full import, what is the overwrite setting for existing objects?

Thanks!

[Bob_Stevens1]

Thanks for these awesome scripts!  They have come in super helpful in a conversion project!

I do have one request:

Would it be possible to get the script updated to also export any service objects and service groups?  We need to compare these between rulebases and think it will most easily be done in a csv file.  I can export services through SmartConsole, but the groups are what don't come through correctly.

Thanks again for all of your hard work!

[Brandon_Cotter]

I started using this script with not much knowledge of how to use the API, and was able to make it work despite that. Awesome! I had a couple questions going into this that I have now come up with answers to, so I thought I'd add them. Apologies in advance for any dimness.

Can I use this script to just import selected objects?

Basically, no. But you can edit down an outputted csv from "cli_api_import_objects_from_csv.sh" and then import it manually with a pretty simple command. I dove into the script and found the basic behavior.

mgmt_cli add [type] --batch [path to modified csv file for that type from the export script] set-if-exists true ignore-warnings true ignore-errors true --ignore-errors true -f json -s [path to session file]

Values for [type]:

host | network | group | group-with-exclusion | address-range | dns-domain | security-zone | dynamic-object | application-site | application-site-category | application-site-group | tag | simple-gateway | time | time-group | access-role | opsec-application | service-tcp | service-udp | service-icmp | service-icmp6 | service-sctp | service-other | service-dce-rpc | service-rpc | service-group

Does this script overwrite existing objects?

"set-if-exists true" is declared in the script, and will modify existing objects to match the imported form. Changing the line

export MgmtCLI_Add_OpParms="set-if-exists true $MgmtCLI_IgnoreErr_OpParms $MgmtCLI_Base_OpParms"

to

export MgmtCLI_Add_OpParms="set-if-exists false $MgmtCLI_IgnoreErr_OpParms $MgmtCLI_Base_OpParms"

should leave existing objects alone.

Hope this isn't misleading anyone.

[VincentBacher]

Hi,

just tested those tools and exporting importing seems to work pretty well.

I am just wondering that not all objects have been imported from dumped csv files.

Csv files have exactly the number of objects of the original smartcenter, so exporting was fine.

I was searching around but did not see in the console output nor in logfile why some objects were not imported.
870 of 877 host objects have been imported and 256 of 300 nets.

Would be nice to know if there any issues in the data to be imported so i could check and modify the objects in the csv prior to import.
Any hints would be appreciated

cheers

Vincent

[VincentBacher]

Hi all,

did anybody already test it on R80.40 ?

I assume it should work, right?

br

Vincent

[Security_Consul]

I trying to add firewall rule via text/csv file in check with sk113078 and https://sc1.checkpoint.com/documents/latest/APIs/index.html#clish/add-access-rule~v1.5%20
But still dont understand.
So my example script:
layer,position,name,action,source,destination,service,track,install-on
Network,1,Rule1,accept,source.1Host_10.10.10.10source.2Host15.15.15.15,Host_20.20.20.20,server.1tcp-22service.2tcp-443,logs,Firewall1

[Workz]

This script is so good, appreciate the effort taken and sharing with all. Do we have any update on getting the services plus services groups also?

[stallwoodj]

Excellent scripts! Thank you!

Just a small one, expanding groups with space in its name during the cli_api_export_objects_to_csv script throws an error (target R80.40):

Group Franking Machines number of members = 2
Error: The parameters of show-group command should be provided in pairs (key and value). You have provided an odd number of parameters which suggests that you are probably missing a parameter.

Thanks Jamie

[Benjamin_Weiss]

Hi!

I see that this will work for R80.30, but don't see anything about R81.10. I'm creating a new FWMS on R81.10 and would like to move my network objects. Can I use these scripts to do so?

Thanks!

Ben

[PhoneBoy]

The API is versioned, which means they should work fine with R81.10.

[the_rock]

Yes, you can. I used it few times and works fine.

Andy