topic Re: Creating multiple object via CSV in API / CLI Discussion

Creating multiple object via CSV

Raj_Khatri — Tue, 07 Mar 2017 20:23:01 GMT

I'm trying to use the Management API to create multiple network objects and then populate them into a group. I have syntax working, however, when executing SmartConsole crashes and only imports ~150 networks. There are 3000+ networks to be created. Is there a limitation of number of entries that can be imported via SmartConsole CLI?

I'm trying to get the Microsoft Azure Datacenter IP Ranges added so we can create firewall rules.

Download Microsoft Azure Datacenter IP Ranges from Official Microsoft Download Center

CLI syntax: add network -d 10.x.x.x --batch /var/tmp/filename.csv

CSV sample

name	subnet4	mask-length4	color	comments
07032017-azure-europewest1	40.112.124.0	24	slate blue	Microsoft Azure Datacenter IP Ranges - March 7, 2017

Re: Creating multiple object via CSV

Eric_Beasley — Wed, 22 Mar 2017 16:49:27 GMT

You might want to break the operation up into smaller (500 max) item pieces and then verify via json output that you don't have errors.

Also don't forget command extensions like "--ignore-errors true" to ensure that warnings don't kill your operation.

I would post the additions in blocks of 250-500, publish, then proceed, and wait for visible confirmation in SmartConsole that you have the objects before proceeding with adding them to groups and such.

With large volume additions via the CLI API, I've noticed that back end processing and updating the database can take time.

You also may be taxing the management host's resources, which may need to expand to meet the real needs of the operation, so I would keep an eye on top and critical resources like memory and CPU.

Re: Creating multiple object via CSV

Uri_Bialik — Sun, 26 Mar 2017 15:48:37 GMT

Check out this post: Create objects for Azure Data-Center IP ranges - Pythonscript

I hope this is what you were looking for.

Re: Creating multiple object via CSV

Raj_Khatri — Wed, 05 Apr 2017 17:54:23 GMT

Uri, thanks for sharing that post. I haven't tried the script yet but wanted to know if the networks and groups are written to the database immediately or if I'll get the option to publish changes.

Re: Creating multiple object via CSV

Raj_Khatri — Wed, 12 Apr 2017 19:11:30 GMT

Hi Uri, I was able to get the script to execute, however, once it was completed the networks didn't get added to the groups although it said it was doing that. See screenshots. What would cause this?

Re: Creating multiple object via CSV

Eric_Beasley — Fri, 14 Jul 2017 20:03:50 GMT

I have found that making a large volume of changes requires some processing time at the back end for the management server when then having those entries ready for next steps.

Since I didn't write the original script used, I can't say what needs to be fixed, but my generic approach would be to generate the objects, but between object type, publish. Before starting a group member add/set operation, the last added member should be checked to ensure it exists and is usable, then proceed to the add/set operation when the check is successful or kill the process after a suitable timeout to ensure things aren't caught in a loop.

Processing of updates on management depends on activity and connectivity, so cloud solutions may be impacted by limited bandwidth, ergo slower to perform massive updates.

Re: Creating multiple object via CSV

Usman_Shaikh — Mon, 15 Oct 2018 14:08:11 GMT

Hi Raj...What CSV format did you use to add to the group as I cannot find a way to "create" network objects and "add" them to the group in a single CSV

Currently I am doing 2-step process of creating networks first by using 'add network' command and then 'set group' to add them to an existing group using another CSV with appropriate headers

Thanks

Usman

Re: Creating multiple object via CSV

Usman_Shaikh — Mon, 15 Oct 2018 14:13:59 GMT

I got it working by adding "groups" column to the CSV