topic Re: Mgmt_cli limitation when publishing a large number of newly created objects? in API / CLI Discussion

Mgmt_cli limitation when publishing a large number of newly created objects?

FK — Fri, 11 Jun 2021 09:24:36 GMT

Hi CheckMates,

I have a question regarding mgmt_cli and limitation.

Is there a limit when publishing a large amount of newly created objects?

Let´s say you have a script which creates a network group for azure IPs with thousands of network objects.

Updatable objects is not an option -> 1400 appliances.

Thank you.

Greetings from Austria.

Re: Mgmt_cli limitation when publishing a large number of newly created objects?

_Val_ — Fri, 11 Jun 2021 12:29:28 GMT

You need to mind API time out error, which is 10 min by default. If you push a long list of objects, and the command takes more than 10 minutes to post, API will return an error. The best practice is to push in smaller portions, multiple times. If each POST is short enough, you should be okay.

Re: Mgmt_cli limitation when publishing a large number of newly created objects?

PhoneBoy — Fri, 11 Jun 2021 20:22:35 GMT

The more changes you try to publish at the same time, the longer the process will take.
If you’re making a bunch of changes at once, it’s generally recommended to publish every few hundred changes or so.

Re: Mgmt_cli limitation when publishing a large number of newly created objects?

JozkoMrkvicka — Sat, 12 Jun 2021 21:01:04 GMT

Is there some easy way how to force publish to be done once number of XY changes were done in one session ?

For example, I want to automatically publish 50 changes if performed in the same session.

If I did 160 changes in the same session, the first publish will be done after first 50 changes, next publish after next 50 changes (100 changes already done). The last 10 changes will be handled by "manual" publish.

Re: Mgmt_cli limitation when publishing a large number of newly created objects?

PhoneBoy — Sat, 12 Jun 2021 22:42:04 GMT

On the API side of things? No.
You would have to handle that in the logic of your script.
You can commit with each mgmt_cli invocation by using -r true, however that incurs a lot of extra overhead (login/do/commit/logout with each command).

Re: Mgmt_cli limitation when publishing a large number of newly created objects?

Tomer_Noy — Sun, 13 Jun 2021 14:54:05 GMT

I would seriously consider upgrading the gateways to newer hardware (1500 appliances) in order to use the Azure Updatable Objects.

For anyone that is coming across this thread, it's a simple as choosing it in the Access policy:

Filling your management with thousands of network objects and huge network groups isn't good practice and might introduce future slowness in some scenarios beyond the challenge of the initial publish via API.

Re: Mgmt_cli limitation when publishing a large number of newly created objects?

FK — Mon, 14 Jun 2021 08:15:09 GMT

Hi Guys,

thank you for inputs.

I will try to seperate it into smaller groups per publish.

Greetings.

Fabian

Re: Mgmt_cli limitation when publishing a large number of newly created objects?

Robert_Decker1 — Mon, 14 Jun 2021 15:17:11 GMT

100 changes per publish is recommended.

Same goes for group members.

Re: Mgmt_cli limitation when publishing a large number of newly created objects?

FK — Mon, 14 Jun 2021 15:21:43 GMT

Thanks for this info.

Re: Mgmt_cli limitation when publishing a large number of newly created objects?

Henrik_Noerr1 — Wed, 16 Jun 2021 10:02:06 GMT

This complicates rollbacks and revision control however. Not sure there is a great solution with the time it takes in general to communitate with the API on larger installations / changes

Re: Mgmt_cli limitation when publishing a large number of newly created objects?

Robert_Decker1 — Wed, 16 Jun 2021 10:09:31 GMT

One solution is to increase the timeout on Apache server from default 10 minutes.

Re: Mgmt_cli limitation when publishing a large number of newly created objects?

G_W_Albrecht — Fri, 15 Oct 2021 11:04:59 GMT

How to achieve that ? Could not find it in

/opt/CPshrd-R81/web/Apache/conf/cp-httpd.conf

Re: Mgmt_cli limitation when publishing a large number of newly created objects?

Jim_Oqvist — Sun, 17 Oct 2021 05:39:53 GMT

Hi, In addition to all good recommendations you have received in terms of publishing 100 changes and upgrade the hardware to be able to use updateable objects, I would like to point you to add/set/delete-object-batch https://sc1.checkpoint.com/documents/latest/APIs/index.html#cli/add-objects-batch~v1.8%20 wich are API endpoints that where added in R80.40.

Batch API Significantly increase API performance in multiple object operations (add/set/delete).

Object creation time reduced by up to 92% when compared to adding one-by-one*
Object deletion time reduced by up to 87% when compared to deleting one-by-one*

*In the tested scenario of adding and deleting 256 hosts objects.

Commands:

add-objects-batch
set-objects-batch
delete-objects-batch

Here is an example of adding 250 hosts objects

https://github.com/checkpointsw-devsec/chkp_api_examples/blob/master/mgmt_cli/mgmt_api_add-objects-batch.sh

here is an example of deleting host patterns

https://community.checkpoint.com/t5/API-CLI-Discussion/mgmt-cli-to-delete-all-objects-matching-a-pattern/td-p/75217

Please note that usually when calling mgmt_cli with credentials (without doin a explicit login) results in four different operations including Publish.

login
make change
publish
logout

Using the object-batch operation with mgmt_cli require you to do a explicit publish.

You can pobrobly be able to make more that hundred changes when using object-batch operation with out hitting any time out value as mentioned by Val, but the exact limit is dependent of your environment I suggest that you test with different number of changes in order to get to a value that suitable for your scenario.

Kind Regards

Jim