Ansible Connection Error when playbook with large group member

[mbouri]

Hello

I have an issue when I try to create or update group with large member (>3000 member), the playbook work fine with small group, is there any option to enable on ansible config settings or checkpoint SMS side ?

my playbook :

---
- name: Deploy Grp
hosts: check_point
connection: httpapi

tasks:

- name: include
include_vars: 'var.yaml'

- name: add-group
cp_mgmt_group:
name: "gp-name"
ignore_warnings: yes
members: "{{name}}"
- name: publish
cp_mgmt_publish:

var file : 

name:

- h-name1

- h-name2

....

hosts.ini file :

[check_point]
IP_SMS
[check_point:vars]
ansible_httpapi_use_ssl=True
ansible_httpapi_validate_certs=False
ansible_user=admin
ansible_password=password
ansible_network_os=checkpoint

the error message :

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.module_utils.connection.ConnectionError: The read operation timed out
fatal: [10.1.1.100]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"/var/lib/jenkins/.ansible/tmp/ansible-local-37897pe768pwq/ansible-tmp-1590687244.472336-2766174878375/AnsiballZ_cp_mgmt_group.py\", line 102, in <module>\n _ansiballz_main()\n File \"/var/lib/jenkins/.ansible/tmp/ansible-local-37897pe768pwq/ansible-tmp-1590687244.472336-2766174878375/AnsiballZ_cp_mgmt_group.py\", line 94, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/var/lib/jenkins/.ansible/tmp/ansible-local-37897pe768pwq/ansible-tmp-1590687244.472336-2766174878375/AnsiballZ_cp_mgmt_group.py\", line 40, in invoke_module\n runpy.run_module(mod_name='ansible.modules.network.checkpoint.cp_mgmt_group', init_globals=None, run_name='__main__', alter_sys=True)\n File \"/usr/lib64/python2.7/runpy.py\", line 176, in run_module\n fname, loader, pkg_name)\n File \"/usr/lib64/python2.7/runpy.py\", line 82, in _run_module_code\n mod_name, mod_fname, mod_loader, pkg_name)\n File \"/usr/lib64/python2.7/runpy.py\", line 72, in _run_code\n exec code in run_globals\n File \"/tmp/ansible_cp_mgmt_group_payload_wCTvQX/ansible_cp_mgmt_group_payload.zip/ansible/modules/network/checkpoint/cp_mgmt_group.py\", line 140, in <module>\n File \"/tmp/ansible_cp_mgmt_group_payload_wCTvQX/ansible_cp_mgmt_group_payload.zip/ansible/modules/network/checkpoint/cp_mgmt_group.py\", line 135, in main\n File \"/tmp/ansible_cp_mgmt_group_payload_wCTvQX/ansible_cp_mgmt_group_payload.zip/ansible_collections/check_point/mgmt/plugins/module_utils/checkpoint.py\", line 256, in api_call\n File \"/tmp/ansible_cp_mgmt_group_payload_wCTvQX/ansible_cp_mgmt_group_payload.zip/ansible_collections/check_point/mgmt/plugins/module_utils/checkpoint.py\", line 63, in send_request\n File \"/tmp/ansible_cp_mgmt_group_payload_wCTvQX/ansible_cp_mgmt_group_payload.zip/ansible/module_utils/connection.py\", line 185, in __rpc__\nansible.module_utils.connection.ConnectionError: The read operation timed out\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

[PhoneBoy]

I imagine the API is timing out with this request.
Recommend getting the TAC involved.
Also tagging @Omer_Kleinstern 

[Or_Soffer]

Hey @mbouri ,

let's continue this conversation on mail.
Please contact me by mail - orso@checkpoint.com

Thanks,

Or

[PhoneBoy]

I'll provide out of band

[IgorBodrik]

Hello, guys! I have the same issue. I need to update group containing ~3000 objects. Do you have a solution for this?

[Art_Zalenekas]

Try to do a publish every 100 objects.

[IgorBodrik]

How can i do that?

The error appears in a block:

- name: add-group
cp_mgmt_group:
name: "gp-name"
ignore_warnings: yes
members: "{{name}}"

{{ name }} - is a huge array ~3000 elements. 

[Art_Zalenekas]

Take a look at an example that I have posted before. This should be very similar if you are familiar with looping in Ansible.
https://community.checkpoint.com/t5/Ansible/Ansible-Improving-Performance/m-p/141062/highlight/true#...

[IgorBodrik]

@Art_Zalenekas Thanks for your response. But i don't need to loop over group members and publish each N changes. If I did this, I would end up with a group with one element. 

[Art_Zalenekas]

True. Through the API directly you could do add a member to a group. Through Ansible you cannot do it that way. I tested adding 3,000 members into a group, and that worked. What I had to do is increase the connection timeout to 120 (2 minutes) for the httpapi plugin of a task from the Ansible perspective.
https://docs.ansible.com/ansible/latest/collections/ansible/netcommon/httpapi_connection.html
Look at the variable persistent_command_timeout
I would highly recommend that you add/set the group membership of an object when you are adding those objects. That way you don't have to care about the group itself, but its existence/creation.

Good luck!

[IgorBodrik]

Hi! I tried to tune ANSIBLE_PERSISTENT_CONNECT_TIMEOUT and ANSIBLE_PERSISTENT_COMMAND_TIMEOUT variables. It didn't help. In my case the error messaged appears after ~2 min, while timeouts are more than that.

Actually on the first run when the group is empty it worked. Sorry, I didn't mention that. Try to run such playbook second time. Maybe you will see the error.

Thank you for your advise about setting membership in object. It works great for initial objects creation. But it would be inconvenient to have infrastructure config in such format.

[Art_Zalenekas]

You might be having performance issues with your management. My testing worked. Increase the timeout to 5 minutes.