Mgmt_cli limitation when publishing a large number of newly created objects?

[FK]

Hi CheckMates,

I have a question regarding mgmt_cli and limitation.

Is there a limit when publishing a large amount of newly created objects?

Let´s say you have a script which creates a network group for azure IPs with thousands of network objects.

Updatable objects is not an option -> 1400 appliances.

Thank you.

Greetings from Austria.

[_Val_ · Accepted solution]

You need to mind API time out error, which is 10 min by default. If you push a long list of objects, and the command takes more than 10 minutes to post, API will return an error. The best practice is to push in smaller portions, multiple times. If each POST is short enough, you should be okay.

[FK]

Hi Guys,

thank you for inputs.

I will try to seperate it into smaller groups per publish.

Greetings.

Fabian

[Henrik_Noerr1]

This complicates rollbacks and revision control however. Not sure there is a great solution with the time it takes in general to communitate with the API on larger installations / changes

[Robert_Decker1]

One solution is to increase the timeout on Apache server from default 10 minutes.

[G_W_Albrecht]

How to achieve that ? Could not find it in 

/opt/CPshrd-R81/web/Apache/conf/cp-httpd.conf

[PhoneBoy]

The more changes you try to publish at the same time, the longer the process will take.
If you’re making a bunch of changes at once, it’s generally recommended to publish every few hundred changes or so.

[JozkoMrkvicka]

Is there some easy way how to force publish to be done once number of XY changes were done in one session ?

For example, I want to automatically publish 50 changes if performed in the same session.

If I did 160 changes in the same session, the first publish will be done after first 50 changes, next publish after next 50 changes (100 changes already done). The last 10 changes will be handled by "manual" publish.

[PhoneBoy]

On the API side of things? No.
You would have to handle that in the logic of your script.
You can commit with each mgmt_cli invocation by using -r true, however that incurs a lot of extra overhead (login/do/commit/logout with each command).

[Tomer_Noy]

I would seriously consider upgrading the gateways to newer hardware (1500 appliances) in order to use the Azure Updatable Objects.

For anyone that is coming across this thread, it's a simple as choosing it in the Access policy:

Filling your management with thousands of network objects and huge network groups isn't good practice and might introduce future slowness in some scenarios beyond the challenge of the initial publish via API.

[Robert_Decker1]

100 changes per publish is recommended.

Same goes for group members.

[FK]

Thanks for this info.

[Jim_Oqvist · Accepted solution]

Hi, In addition to all good recommendations you have received in terms of publishing 100 changes and  upgrade the hardware to be able to use updateable objects, I would like to point you to add/set/delete-object-batch https://sc1.checkpoint.com/documents/latest/APIs/index.html#cli/add-objects-batch~v1.8%20 wich are API endpoints that where added in R80.40. 

Batch API Significantly increase API performance in multiple object operations (add/set/delete).

•          Object creation time reduced by up to 92% when compared to adding one-by-one*
•          Object deletion time reduced by up to 87% when compared to deleting one-by-one*

*In the tested scenario of adding and deleting 256 hosts objects.

Commands:

•          add-objects-batch
•          set-objects-batch
•          delete-objects-batch

Here is an example of adding 250 hosts objects

https://github.com/checkpointsw-devsec/chkp_api_examples/blob/master/mgmt_cli/mgmt_api_add-objects-b...

here is an example of deleting host patterns

https://community.checkpoint.com/t5/API-CLI-Discussion/mgmt-cli-to-delete-all-objects-matching-a-pat...

Please note that usually when calling mgmt_cli with credentials (without doin a explicit login) results in four different operations including Publish.

• login
• make change
• publish
• logout

Using the object-batch operation with mgmt_cli require you to do a explicit publish.

You can pobrobly be able to make more that hundred changes when using object-batch operation with out hitting any time out value as mentioned by Val, but the exact limit is dependent of your environment I suggest that you test with different number of changes in order to get to a value that suitable for your scenario.

Kind Regards

Jim