Create a Post
cancel
Showing results for 
Search instead for 
Did you mean: 
Dima_M
Employee
Employee

Heads-up: Management API Remote calls frequency limit

Hi all,

If you're working with remote scripts utilizing Management REST API, please read the below post.

To enable efficient work of Management REST API clients, starting R81 (expected release date – September 2020)  we are going to limit the allowed frequency of the login command for remote API calls only, to 3 logins per admin per domain per minute.

Any request that will exceed this limit – will by failed by the server with a new error ("Too many requests in a given amount of time").To rule out possible impact, we recommend you to test your solutions which are based REST API once R81 Public EA is available for download.

Below you can find:

  1. Guidelines for adapting your code to handle the new error
  2. General best practices for login API

Please do not hesitate to contact us for further consultation.

Handling the new error in case too many login requests in a given time

Starting  R81, login from a remote machine is limited to 3 logins per minute for each admin to a specific domain. Your scripts should catch the error below and try again later:

 

 

retries = 0

DO

    wait for (2^retries) seconds

    result = Do login operation.

    IF result.is_success = true

        retry = false

    ELSE IF result.is_success = false

       IF result.get_error_message = "Too many requests in a given amount of time"

                  retry = true

      ELSE

                  Some other error occurred, stop calling the API.

                  retry = false

    END IF

    retries = retries + 1

WHILE (retry AND (retries < MAX_RETRIES))

 

 

 

General best practices for login API:

  • Login to last published session (enter-last-published-session) when you don't need to make any changes or updates. For example, if all commands are of type "show". This is because login for reading and writing has more database overhead than a login for read-only and other publishes will not affect your session.

Example 1(mgmt_cli):

 

 

mgmt_cli login user "aa" password "aaaa" enter-last-published-session true -f json

 

 

Example 2(Web Services):

 

 

POST {{server}}/login

Content-Type: application/json



{

  "user" : "aa",

  "password" : "aaaa"

  "enter-last-published-session" : "true"

}

 

 

  • When using the mgmt_cli, reduce the number of logins and logouts to the minimum possible by using sessions, and working within one session..
    • Using one session is faster. If you do not explicitly use a session ID (sid), then each command results in this set of operations: login, action, publish, and logout. All these extra operations cause a higher management and database overhead.
    • Reducing the number of sessions helps you avoid reaching maximum allowed number of concurrent read/write sessions. The maximum is 100.


Bad Practice Example (pseudo code):
In this example, API call is being executed without an explicit session-id. This means that each time, four commands are being executed (login,add-host,publish and logout)

 

 

for i=0 to i<100 do:

         mgmt_cli -r true add-host name hosts_list[i] ip-address ip_list[i]

 

 


Good Practice Example (pseudo code):
In this example, login is done just once. All changes are made in one session, and at the end of the session there is a publish and logout. This saves the overhead of managing multiple login and logout operations on the server.

 

 

session=`mgmt_cli -r true login --format json| jq -r '.sid'` //  login once and set session id (sid) into a variable

for i=0 to i<100  do:

   mgmt_cli add-host name hosts_list[i] ip-address ip_list[i]  --session-id $session // use the session id for adding hosts in loop

mgmt_cli publish --session-id $session  // publish all changes in one session. Publish occur only once

mgmt_cli logout --session-id $session  // logout once

 

 


Note- If you have many hundreds of changes, it is best to avoid publishing once at the end of the session. Instead, publish a few times within your session. For example, publish every 100 changes. However, you only need to log in once at the beginning of the session, and log out at the end of the session.

9 Replies
nissimv
Explorer

Hi Dima,

Thank you for sharing this valuable information. 

I have R81 - Build 287 in my lab and I'm able to run API calls (using my application) with more then the above ratio you mentioned.

I made sure there is more then 3 logins per admin per domain per minute so In my lab this issue is not observed while one of my customers has the above described issue and he get's the following error

2021-03-26 11:29:23,417 ERROR com.checkpoint.management.web_api_is.exceptions.WebApiGeneralException.log:50 [qtp901620835-94] - Error message: [Too many requests in a given amount of time]

What can be the reason this happens only to my customer while according to this post this should be default REST configs for all new R81 installations ? 

 

2021-04-05 10:33:43,040 DEBUG {R81} [com.skybox.view.agent.rest.client.generic.platform.EntityLoggingFilter] (EntityLoggingFilter.java:60) <@127.0.0.1:TASK:1> https://172.x.x.x/web_api/login
2021-04-05 10:33:44,403 DEBUG {R81} [com.skybox.view.agent.rest.client.generic.platform.EntityLoggingFilter] (EntityLoggingFilter.java:60) <@127.0.0.1:TASK:1> https://172.x.x.x/web_api/login
2021-04-05 10:33:47,892 DEBUG {R81} [com.skybox.view.agent.rest.client.generic.platform.EntityLoggingFilter] (EntityLoggingFilter.java:60) <@127.0.0.1:TASK:1> https://172.x.x.x/web_api/login
2021-04-05 10:33:49,146 DEBUG {R81} [com.skybox.view.agent.rest.client.generic.platform.EntityLoggingFilter] (EntityLoggingFilter.java:60) <@127.0.0.1:TASK:1> https://172.x.x.x/web_api/login
2021-04-05 10:33:49,243 DEBUG {R81} [com.skybox.view.agent.rest.client.generic.platform.EntityLoggingFilter] (EntityLoggingFilter.java:60) <@127.0.0.1:TASK:1> https://172.x.x.x/web_api/login
2021-04-05 10:33:50,380 DEBUG {R81} [com.skybox.view.agent.rest.client.generic.platform.EntityLoggingFilter] (EntityLoggingFilter.java:60) <@127.0.0.1:TASK:1> https://172.x.x.x/web_api/login

0 Kudos
nissimv
Explorer

Hi,

Just realize that is working because I was running in MDS environment. This issue happens only with single management environment. 

0 Kudos
Alexander_Wilke
Advisor

Hello,

is this the correct command to disable api throttling?

api throttling off

 

Is this the correct file to increase the number of logins per time interval?

/opt/CPsuite-R81.10/fw1/api/conf/api-load-control.xml

 

Regards
Alex

(1)
SimonMeadows
Participant

Hi,

I have used 'api throttling off' so I can run some pretty heavy logs requests against the api but have notice that throttling is re-enabled when the management software is patched or upgraded.

Does anyone know a way to maintain the throttling setting through a patch or upgrade?

Thanks

Simon

0 Kudos
PhoneBoy
Admin
Admin

Perhaps a regular cron job that executes this command to ensure its always turned off?
That said, it's not recommended to do this.

0 Kudos
Alexander_Wilke
Advisor

Edit this file and turn it off. If you want to have it off the we do not have to care about any changes on this file in the future.
/opt/CPsuite-R81.10/fw1/api/conf/api-load-control.xml

Make the file immuteable. This prevents the system to modify the file.

chattr +i /opt/CPsuite-R81.10/fw1/api/conf/api-load-control.xml

 

0 Kudos
PhoneBoy
Admin
Admin

The main issue with doing that is that you also prevent other necessary changes from being made to this file on upgrade.
Which could create other issues down the road. 

0 Kudos
Alexander_Wilke
Advisor

Do not agree. This file is only for throttling opting. The main option on top is to turn it on or off. The other options a detailed options to throttle. So if you do not like throttling, you can disable it and other changes in the future are not relevant.

 

To be hones - the only reason for that throttling is that the MDS API is slow, is crashing and working much worse than other APIs. To circumvent these issue you (CheckPoint) just limited access and that's it.

ias_gc-dk
Contributor

I have made a python version, because we use the cpapi library to access the mgmt-api. Perhaps others can find a use for this:

 

 

from cpapi import APIClient, APIClientArgs
from time import sleep
import getpass

warning_color = '\033[91m' #red
reset_color = '\033[0m' #return to normal colour

cp_username="testadmin"
if sys.stdin.isatty():
    cp_password = getpass.getpass("Enter password: ")

apiserver="fwmgmt01.test"
cp_domain="10.20.30.40"
client_args = APIClientArgs(server=apiserver)
with APIClient(client_args) as client:
    if client.check_fingerprint() is False:
        print("Could not get the server's fingerprint - Check connectivity with the server.")
        exit(1)
    # login to server:
    cp_login_retries = 0
    cp_login_retry = True
    while cp_login_retries < 3 and cp_login_retry == True:
        login_res = client.login(cp_username, cp_password, domain=cp_domain)
        if login_res.success is True:
            cp_login_retry = False
        elif login_res.success is False:
            print(warning_color,"Login failed for domain "+cp_domain+":\n{}".format(login_res.error_message),reset_color)
            if login_res.data['code'] == "err_too_many_requests":
                print(f"Retrying {3 - cp_login_retries} time(s)")
                cp_login_retry = True
                for i in range(20, 0, -1):
                print(i, end= ' \r')
                sleep(1)
                cp_login_retries = cp_login_retries+1 
            else:
                print(warning_color,"Login failed for domain "+cp_domain+":\n{}".format(login_res.error_message),reset_color)
                cp_login_retry = False
                continue
            else:
                print("Login unsuccessful to",cp_domain)
                exit()

 

Note that this is for logging into a VSX domain. I suppose you can just omit the domain-parameter to log into a non-VSX platform.

After logging in, issue the calls to the API you want. Example:

 

show_vpn_star_response = client.api_call("show-vpn-communities-star",{})
print(show_vpn_star_response)

 

 

0 Kudos

Leaderboard

Epsum factorial non deposit quid pro quo hic escorol.

Upcoming Events

    CheckMates Events