Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Eric_Beasley
Employee
Employee

CLI API Example for exporting, importing, and deleting different objects using CSV files

Check out GITHUB for latest version v00.60.12.100.750 as of 2023-03-14:

GitHub - mybasementcloud/R8x-export-import-api-scripts: Check Point R8x Export, Import, [and more] A... 

https://github.com/mybasementcloud/R8x-export-import-api-scripts/releases

 

Available documentation is provided in the in the GitHub Repository, please check the latest releases document and the README.MD file.

 

Version v00.60.12.100.750 is the last planned release under that GitHub repository, and all new development is going here:
mybasementcloud/R8X_mgmt_cli_API_bash_scripts

 

The post below is not manageable and won't be updated any more, all specifics and improved documentation is provided on GitHub.

BR

Eric @Eric_Beasley 



36 Replies
Suleyman_B_T
Participant

Hi,

Thank you for export import script.

I have a question about script.

 

Export object limit 500.

export APICLIObjectLimit=500

I increased the value through script.(vi editor)

Script did not work after editing

 

How can I increase the export&import object limit?

 

Thank you

Best Regards

Suleyman Toreli

SECURRENT

0 Kudos
Eric_Beasley
Employee
Employee

Hello,

API currently has a object limit value of 1-500, so can't go higher than 500 as far as I can tell.  I'll see if I can do something to determine number of objects and slice the content into 500 objects slices to get output to csv for all the objects.  Import operations via add don't apparently use the limit entry value so don't have the problem, but show operations do.

Will look into this for the next version of script, version 00.23.00 or later.

Eric

0 Kudos
Eric_Beasley
Employee
Employee

Hello again,

Version 00.23.00 now handles more than 500 objects in 500 object slices for json output and it was tested with ~1200 objects to csv (single file).  This works for all objects except for group members, where only 500 groups are handled, but I don't have data to test whether it can handle more than 500 members for a group, since that handling is different from the limit based show "objects" operation.

Check GitHub for version 00.23.00 or later to test it out.

Also added script "cli_api_get_object_totals.sh" and "cli_api_get_object_totals_w_group_member_count.sh" that will just provide a count of objects and a log of operations to query group members.  These are located under the Export Objects folder or 7z file.

Eric

0 Kudos
Eric_Beasley
Employee
Employee

And one more update on this question.  Script version 00.24.00 now handles all processed objects for export with > 500 objects, even the tougher group members and new host interfaces settings.

Eric

0 Kudos
Tung_Nguyen_Son
Participant

 Phuong Nguyen Le  Phi Thanh Tung Good script!

Julien_Tissot
Explorer

Hello!

If there is no object of 1 category (for example no address range) the script terminates. Are you aware of that?

Otherwise it works really good, thank you!

0 Kudos
Eric_Beasley
Employee
Employee

Can you e-mail me an execution where it terminates like that?  Also run it with the -v (verbose) parameter for more details, I need to see what the issue is.

0 Kudos
Eric_Beasley
Employee
Employee

OK, after some testing on a new system, identified the issue and fixed it and the initial problem on an MDS in version 00.25.00 posted to GitHub.

John_Tammaro1
Contributor
Contributor

Fantastic work. Just what we needed.

Just a heads up for the author that there is a spelling error in the output. This does not break any functionality at all but thought I would share this.

Dump times to /home/admin/results/2017-08-31-1518BST/full/dump_times_full_2017-08-31-1518BST.json
Now processing up to next 500 objects starting with object 0 of 3 remainging!

Should be "remaining" not "remainging" . Perhaps this could be fixed upstream ?

Eric_Beasley
Employee
Employee

Hi John,

Thanks for that, I’ll fix that in v00.25.01, typing fast can lead to great typos, and a standard text editor doesn’t spell check—so a little one like this is easy to miss. It’s more fun when you flub a command argument or variable name, that usually comes back around in testing.

Eric

0 Kudos
Zbynek_Michl
Explorer

Hello,

I have an issues with cli_api_export_object_group-members_to_csv.sh script. If I set APICLIObjectLimit=100, then:

- Each particular pair ("name","members.add") in group-members_standard_csv.csv is listed 6x, e.g.:

MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"
MyGroup1,"MyMember1"

- List in group-members_standard_csv.csv is not complete, it contains just a part of the all records (I know, our db is quite big). Despite the script finishes with no error.

Could you have a look on it, please? Script version is 00.25.00.

Thanks

Zbynek

0 Kudos
Eric_Beasley
Employee
Employee

Hi,

Why are you modifying the limit value, which identifies how many groups will be added to the array that the group members are searched from?

Leave the value where it was and also use the latest version of the scripts.

Group Members is built by reading the full list of groups into an array, then processing the array of groups for each group, for its respective members, such that we get all of the members listed for each group.

The proper example of a group members CSV file:

"name","members.add"

"TEST_GROUP_01","test-addr-range-01-062.128.100.000-062.128.101.255"

"TEST_GROUP_01","test-addr-range-02-010.128.100.000-010.128.101.255"

"TEST_GROUP_01","test-addr-range-03-010.128.102.000-010.128.103.255"

"TEST_GROUP_01","test-addr-range-04-010.128.104.000-010.128.105.255"

"TEST_GROUP_01","test-addr-range-05-010.128.106.000-010.128.107.255"

"TEST_GROUP_02","TEST-HOST-01"

"TEST_GROUP_02","TEST-HOST-02"

"TEST_GROUP_02","TEST-HOST-03"

"TEST_GROUP_02","TEST-HOST-04"

"TEST_GROUP_02","TEST-HOST-05"

"TEST_GROUP_03","TEST-HOST-02"

"TEST_GROUP_03","TEST-HOST-04"

"TEST_GROUP_04","netw_test_LAN_01"

"TEST_GROUP_04","netw_test_LAN_02"

"TEST_GROUP_04","netw_test_LAN_03"

"TEST_GROUP_04","netw_test_LAN_04"

"TEST_GROUP_04","netw_test_LAN_05"

"TEST_GROUP_05",".1.test.domain.local"

"TEST_GROUP_05",".2.test.domain.local"

"TEST_GROUP_05",".3.test.domain.local"

For best results in determining if and what is going wrong, provide a executing of the script with the “-v” (verbose mode) and a full copy of the script execution from the CLI. I recommend not using the “—password” option, but entering that at the prompt when the mgmt_cli login executes, since I don’t need to know any passwords.

That should indicate if the processing is happening as expected.

Latest version is updated on GitHub: https://github.com/mybasementcloud/R8x-export-import-api-scripts, right now that is version 00.25.01

Versions prior to 00.23.00 do not execute operations for more than either 500 (API limitation) or the set APICLIObjectLimit value objects, and version 00.24.00 and higher handle group members and host interfaces for more than 500 groups or hosts, respectively.

BR

Eric Beasley

Security Engineer, North America Channel Sales

Check Point Software Technologies Ltd.

Cell: +1 708-224-7724 E-mail: ericb@checkpoint.com

Additional Contact Informaton:

Support : +1 (888) 361 5030 or +1 (972) 444 6600 Technical Assistance Center (TAC)

Partner Sales Assistance: (866) 488-6689 or cpp@checkpoint.com

Account Services: +1 (972) 444-6600, option 5 or AccountServices@checkpoint.com

Licensing Advocacy: +1 (800) 429-4391 or advocacy@checkpoint.com

Inside Sales: +1 (650) 628-2225

Check Point Incident Response Team: (866) 923-0907 > initial consultation is free

emergency-response@checkpoint.com<mailto:emergency-response@checkpoint.com> > you do not need to be an existing Check Point customer

0 Kudos
Michael_Vogler
Explorer

Hi Eric, 

first of all thank you for the great scripts!

I wanted to use the Script cli_api_export_object_group-members_to_csv.sh, too. But I have the same Problems like Zbynek. To get full information I run the script in Verbose Mode and i noticed, that the script stop exporting exactly after 1000 Elements. Perhaps there is a Limit of 1000 anywhere?

Best regards

Michael

Zbynek_Michl
Explorer

I am modifying the APICLIObjectLimit because the default value (500) crashes the API java server with our DB. And we do not have any fix from CP yet. So the only chance for me is to decrease the value to 100 now, which seems to be safe.

Regards

Zbynek

0 Kudos
Paul_Warnagiris
Advisor

Hi Eric.  I must admit I am no coder, I just do engineering.  I'm trying to export R80.10 objects and move it to another 80.10 setup.  When I run your script cli_api_export_objects_to_csv.sh v00.26.01  I get the following error.  Any help would be appreciated:

Done dumping hosts


Use array of hosts to generate host interfaces CSV


!!!! Error data file missing : ./dump/2017-11-06-1111UTC/csv/wip/hosts_full_csv.csv.data
Terminating!

0 Kudos
Eric_Beasley
Employee
Employee

This issue was identified and fixed in 00.26.07 and again tested in 00.27.05.

0 Kudos
Joshua_Hatter
Employee
Employee

Gotta say kudos just for the documentation and write up. Great work!

0 Kudos
Martin_Valenta
Advisor

Great stuff man, what is ETA for services exports?

0 Kudos
Brandon_Cotter
Contributor

I successfully produced an export and then trimmed down the csv (hosts_full_csv.csv) output to just one host as an experiment. I ran the import script and the single host was not imported. It output a file called "dump_add_host_json" that contained the text

Line 2: {
"code" : "generic_err_invalid_parameter",
"message" : "Parameter [auto-rule] value is not valid (full path: nat-settings.auto-rule)"
}

{ "response" : []}

What am I doing wrong? Is partial import not supported?

Also, if I do a full import, what is the overwrite setting for existing objects?

Thanks!

Bob_Stevens1
Explorer

Thanks for these awesome scripts!  They have come in super helpful in a conversion project!

 

I do have one request:

Would it be possible to get the script updated to also export any service objects and service groups?  We need to compare these between rulebases and think it will most easily be done in a csv file.  I can export services through SmartConsole, but the groups are what don't come through correctly.

 

Thanks again for all of your hard work!

0 Kudos
Brandon_Cotter
Contributor

I started using this script with not much knowledge of how to use the API, and was able to make it work despite that. Awesome! I had a couple questions going into this that I have now come up with answers to, so I thought I'd add them. Apologies in advance for any dimness.

Can I use this script to just import selected objects?

Basically, no. But you can edit down an outputted csv from "cli_api_import_objects_from_csv.sh" and then import it manually with a pretty simple command. I dove into the script and found the basic behavior.

mgmt_cli add [type] --batch [path to modified csv file for that type from the export script] set-if-exists true ignore-warnings true ignore-errors true --ignore-errors true -f json -s [path to session file]

Values for [type]:

host | network | group | group-with-exclusion | address-range | dns-domain | security-zone | dynamic-object | application-site | application-site-category | application-site-group | tag | simple-gateway | time | time-group | access-role | opsec-application | service-tcp | service-udp | service-icmp | service-icmp6 | service-sctp | service-other | service-dce-rpc | service-rpc | service-group

Does this script overwrite existing objects?

"set-if-exists true" is declared in the script, and will modify existing objects to match the imported form. Changing the line

export MgmtCLI_Add_OpParms="set-if-exists true $MgmtCLI_IgnoreErr_OpParms $MgmtCLI_Base_OpParms"

to

export MgmtCLI_Add_OpParms="set-if-exists false $MgmtCLI_IgnoreErr_OpParms $MgmtCLI_Base_OpParms"

should leave existing objects alone.

 

Hope this isn't misleading anyone.

VincentBacher
Participant

Hi,

 

just tested those tools and exporting importing seems to work pretty well.

I am just wondering that not all objects have been imported from dumped csv files.

Csv files have exactly the number of objects of the original smartcenter, so exporting was fine.

I was searching around but did not see in the console output nor in logfile why some objects were not imported.
870 of 877 host objects have been imported and 256 of 300 nets.

Would be nice to know if there any issues in the data to be imported so i could check and modify the objects in the csv prior to import.
Any hints would be appreciated

cheers

Vincent

0 Kudos
VincentBacher
Participant

Hi all,

 

did anybody already test it on R80.40 ?

I assume it should work, right?

br

Vincent

0 Kudos
Security_Consul
Participant

I trying to add firewall rule via text/csv file in check with sk113078 and https://sc1.checkpoint.com/documents/latest/APIs/index.html#clish/add-access-rule~v1.5%20
But still dont understand.
So my example script:
layer,position,name,action,source,destination,service,track,install-on
Network,1,Rule1,accept,source.1Host_10.10.10.10source.2Host15.15.15.15,Host_20.20.20.20,server.1tcp-22service.2tcp-443,logs,Firewall1
0 Kudos
Workz
Participant

This script is so good, appreciate the effort taken and sharing with all. Do we have any update on getting the services plus services groups also?

0 Kudos
stallwoodj
Collaborator

Excellent scripts! Thank you!

Just a small one, expanding groups with space in its name during the cli_api_export_objects_to_csv script throws an error (target R80.40):

Group Franking Machines number of members = 2
Error: The parameters of show-group command should be provided in pairs (key and value). You have provided an odd number of parameters which suggests that you are probably missing a parameter.

 

Thanks Jamie

 

0 Kudos
Benjamin_Weiss
Explorer

Hi!

I see that this will work for R80.30, but don't see anything about R81.10. I'm creating a new FWMS on R81.10 and would like to move my network objects. Can I use these scripts to do so?

Thanks!

Ben

0 Kudos
PhoneBoy
Admin
Admin

The API is versioned, which means they should work fine with R81.10.

0 Kudos
the_rock
Legend
Legend

Yes, you can. I used it few times and works fine.

Andy

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events