Ansible Checkpoint Modules

[Simon_Macpherso]

Hi,

I'm running a bash script to copy cpviewdb.dat files from remote gateways to central file storage.

I will use Ansible to automate this. 

Whilst running my playbook from the CLI, the run script task (check_point.mgmt.cp_mgmt_run_script) operation times out after 300 seconds and Ansible reports the task as failed. The task continues to run in he background until completion. 

Do you know which 300 second default timeout setting is causing this and where it can be modified? 

Regards,

Simon 

[PhoneBoy]

@Or_Soffer any idea?

[Art_Zalenekas]

What version of the module are you using? Ansible native Check Point module or collection? In our Ansible collection, the default value is 30 in minutes, so it sounds like you are not using the collection.
https://docs.ansible.com/ansible/latest/collections/check_point/mgmt/cp_mgmt_run_script_module.html

Please let us know which Ansible version you are running and paste the Ansible task here that does the execution of run-script (sanitized).

[Simon_Macpherso]

Hi @Art_Zalenekas 

I'm using version 1.0.6 of the collection

(https://galaxy.ansible.com/check_point/mgmt)

I'm running ansible version 2.9.10.

ansible 2.9.10
config file = /etc/ansible/ansible.cfg
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Sep 26 2019, 13:23:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

Playbook and debug output of ansible-playbook below.

- hosts: all
  connection: httpapi
  tasks:
    - name: "copy cpviewdb"
      check_point.mgmt.cp_mgmt_run_script:
        script_name: "cpviewdb export"
        script: /usr/tmp/scripts/cpviewdb/cpviewdb_exp_v2.sh
        targets:
          - "fw-1"
       async: 1000
       poll: 10

-------------------------------------------------------------------------------------------------------------------------

task path: /var/lib/awx/projects/checkpoint/cpviewdb_run.yml:7
<10.1.2.51> ESTABLISH LOCAL CONNECTION FOR USER: root
<10.1.2.51> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-local-105715JthvxB `"&& mkdir /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556 && echo ansible-tmp-1614902298.09-105833-95466278797556="` echo /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556 `" ) && sleep 0'
Using module file /root/.ansible/collections/ansible_collections/check_point/mgmt/plugins/modules/cp_mgmt_run_script.py
<10.1.2.51> PUT /root/.ansible/tmp/ansible-local-105715JthvxB/tmpnNzePL TO /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556/AnsiballZ_cp_mgmt_run_script.py
<10.1.2.51> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556/ /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556/AnsiballZ_cp_mgmt_run_script.py && sleep 0'
<10.1.2.51> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556/AnsiballZ_cp_mgmt_run_script.py && sleep 0'
<10.1.2.51> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556/ > /dev/null 2>&1 && sleep 0'
fatal: [10.1.2.51]: FAILED! => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "args": null, 
            "comments": null, 
            "script": "/usr/tmp/scripts/cpviewdb/cpviewdb_exp_v2.sh", 
            "script_name": "cpviewdb export", 
            "targets": [
                "fw-1"
            ], 
            "version": null, 
            "wait_for_task": true
        }
    }, 
    "msg": "Task fw-cil-cluster.mgsops.fw - cpviewdb export with task id c2e0722e-a694-47a3-bdb0-6bc842bbd43b failed. Look at the logs for more details"
}

PLAY RECAP *****************************************************************************************************************************************************************************************************************************
10.1.2.51                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

Was the wait_for_task_timeout default of 30 minutes introduced in version 2.0.0 or an earlier version?

Regards,

Simon

[Simon_Macpherso]

Hi,

I'm using v1.0.6 of the collection. Was the default timeout value for wait_for_task_timeout available and set to 30 minutes in this version or was it introduced in v2.0.0? 

Running Ansible  2.9.10

Playbook and debug output of ansible-playbook below.

- hosts: all

  connection: httpapi

  tasks:

    - name: "copy cpviewdb"

      check_point.mgmt.cp_mgmt_run_script:

        script_name: "cpviewdb export"

        script: /usr/tmp/scripts/cpviewdb/cpviewdb_exp_v2.sh

        targets:

          - "fw-1"

       async: 1000

       poll: 10

-------------------------------------------------------------------------------------------------------------------------

task path: /var/lib/awx/projects/checkpoint/cpviewdb_run.yml:7

<10.1.2.51> ESTABLISH LOCAL CONNECTION FOR USER: root

<10.1.2.51> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-local-105715JthvxB `"&& mkdir /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556 && echo ansible-tmp-1614902298.09-105833-95466278797556="` echo /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556 `" ) && sleep 0'

Using module file /root/.ansible/collections/ansible_collections/check_point/mgmt/plugins/modules/cp_mgmt_run_script.py

<10.1.2.51> PUT /root/.ansible/tmp/ansible-local-105715JthvxB/tmpnNzePL TO /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556/AnsiballZ_cp_mgmt_run_script.py

<10.1.2.51> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556/ /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556/AnsiballZ_cp_mgmt_run_script.py && sleep 0'

<10.1.2.51> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556/AnsiballZ_cp_mgmt_run_script.py && sleep 0'

<10.1.2.51> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-local-105715JthvxB/ansible-tmp-1614902298.09-105833-95466278797556/ > /dev/null 2>&1 && sleep 0'

fatal: [10.1.2.51]: FAILED! => {

    "changed": false, 

    "invocation": {

        "module_args": {

            "args": null, 

            "comments": null, 

            "script": "/usr/tmp/scripts/cpviewdb/cpviewdb_exp_v2.sh", 

            "script_name": "cpviewdb export", 

            "targets": [

                "fw-1"

            ], 

            "version": null, 

            "wait_for_task": true

        }

    }, 

    "msg": "Task fw-cil-cluster.mgsops.fw - cpviewdb export with task id c2e0722e-a694-47a3-bdb0-6bc842bbd43b failed. Look at the logs for more details"

}

PLAY RECAP *****************************************************************************************************************************************************************************************************************************

10.1.2.51                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

[Art_Zalenekas]

It looks like we modified the timeout values in version 2.0.0 of the collection.
https://github.com/CheckPointSW/CheckPointAnsibleMgmtCollection/releases
If you are using just one target, could you please test-out without setting the task into async mode? Also, please update to the latest collection (--force on ansible-galaxy when pulling the collection). Let us know how that goes.

[Simon_Macpherso]

Hi,

I've updated to v 2.0.0 of the collection using the --force switch. 

Ansible is still reporting the task as failed with and without async mode added. It still continues to completion in the background as before. 

You can now see "wait_for_task_timeout": 30 in the output. 

fatal: [10.1.2.51]: FAILED! => {
"changed": false,
"invocation": {
"module_args": {
"args": null,
"comments": null,
"script": "/usr/tmp/scripts/cpviewdb/cpviewdb_exp_v2.sh",
"script_name": "cpviewdb export",
"targets": [
"fw-1"
],
"version": null,
"wait_for_task": true,
"wait_for_task_timeout": 30
}
},
"msg": "Task fw-1 - cpviewdb export with task id 79bfec7f-090d-447e-9fc5-1ec59a7bd175 failed. Look at the logs for more details"
}

PLAY RECAP *****************************************************************************************************************************************************************************************************************************************************
10.1.2.51 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

The original version of the bash script makes 2 SSH connections to remote servers sequentially. I've modified the script by  splitting it out in to 2 separate scripts so each script makes 1 SSH connection. The ansible task runs successfully when referencing the modified script as the duration is approximately half the time, it does not hit the limit that is resulting in the connection timeout. 

Regards,

Simon

[Art_Zalenekas]

Please run the Ansible plabook with -vvv args. Those are verbosity level flags. 

[Simon_Macpherso]

The following is the output with -vvvv

TASK [copy cpviewdb] *******************************************************************************************************************************************************************************************************************************************
task path: /var/lib/awx/projects/checkpoint/cpviewdb_run.yml:5
<10.1.2.51> attempting to start connection
<10.1.2.51> using connection plugin httpapi
<10.1.2.51> found existing local domain socket, using it!
<10.1.2.51> updating play_context for connection
<10.1.2.51>
<10.1.2.51> local domain socket path is /root/.ansible/pc/910d3d0ce6
<10.1.2.51> ESTABLISH LOCAL CONNECTION FOR USER: root
<10.1.2.51> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-local-69114wgQDKP `"&& mkdir /root/.ansible/tmp/ansible-local-69114wgQDKP/ansible-tmp-1615253882.22-69226-97328796880237 && echo ansible-tmp-1615253882.22-69226-97328796880237="` echo /root/.ansible/tmp/ansible-local-69114wgQDKP/ansible-tmp-1615253882.22-69226-97328796880237 `" ) && sleep 0'
Using module file /root/.ansible/collections/ansible_collections/check_point/mgmt/plugins/modules/cp_mgmt_run_script.py
<10.1.2.51> PUT /root/.ansible/tmp/ansible-local-69114wgQDKP/tmpmubyAi TO /root/.ansible/tmp/ansible-local-69114wgQDKP/ansible-tmp-1615253882.22-69226-97328796880237/AnsiballZ_cp_mgmt_run_script.py
<10.1.2.51> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-local-69114wgQDKP/ansible-tmp-1615253882.22-69226-97328796880237/ /root/.ansible/tmp/ansible-local-69114wgQDKP/ansible-tmp-1615253882.22-69226-97328796880237/AnsiballZ_cp_mgmt_run_script.py && sleep 0'
<10.1.2.51> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-local-69114wgQDKP/ansible-tmp-1615253882.22-69226-97328796880237/AnsiballZ_cp_mgmt_run_script.py && sleep 0'
<10.1.2.51> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-local-69114wgQDKP/ansible-tmp-1615253882.22-69226-97328796880237/ > /dev/null 2>&1 && sleep 0'
fatal: [10.1.2.51]: FAILED! => {
"changed": false,
"invocation": {
"module_args": {
"args": null,
"comments": null,
"script": "/usr/tmp/scripts/cpviewdb/cpviewdb_exp_v2.sh",
"script_name": "cpviewdb export",
"targets": [
"fw-1"
],
"version": null,
"wait_for_task": true,
"wait_for_task_timeout": 30
}
},
"msg": "Task fw-1 - cpviewdb export with task id fa379556-3fdd-4424-865e-b748cb6c6c05 failed. Look at the logs for more details"
}

PLAY RECAP *****************************************************************************************************************************************************************************************************************************************************
10.1.2.51 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

[Art_Zalenekas]

So I just executed that on my side and it works. From CHKP Mgmt perspective it works. I think the question that becomes is, does you bash scripts works.
First, try to just execute ls -al / in the playbook under script. If you get a finished task, then it's something within the bash script or you don't have execute bit set on the bash script, or your API user does not have the permissions to execute the bash script. Start simple and work from there up. The API call and the Ansible module works on my side using latest collection.

[Simon_Macpherso]

Can you try to replicate this by running a script for which the duration of which will be 300 seconds or more? e.g. a large file copy or download. 

My script runs. I can run the combined script (with 2 SSH connections and copy on each) manually on a gateway. I can also run each split script manually on a gateway and via Ansible. API permissions are OK. 

[Art_Zalenekas]

So I was able to reproduce that issue as well and so far I can tell it's not the Ansible module, but either the run-script or show-task API call that somehow forces the 300 seconds timeout.  I am looking into this, but I would suggest you open a TAC case as well.

[Simon_Macpherso]

Great thanks for the update and your assistance on this issue to date. 

I've opened a TAC case. 

Could the following discussion be related? 

https://community.checkpoint.com/t5/API-CLI-Discussion/Web-API-timeout/td-p/52741

[Art_Zalenekas]

Simon,
at first, it looks like it is, but this issue is a bit deeper than I have thought. If you run any run-script calls, for now, I would suggest splitting them into very concrete steps to minimize the runtime to less than 5 minutes. Please DM me with the SR number you have the case open and I will also keep track of it. Thanks!

[Simon_Macpherso]

Hi, that is what I've done in the interim - modify the playbooks so they are running less tasks. This has been fine for now but will create more work going forward. The SR is 6-0002571220. 

[Simon_Macpherso]

Hi.

This is related to sk160753. It's a known limitation that has existed since R80.10.

PRHF-14607 :

"Running a one time script on a Security Gateway (that reads files or outputs of commands) using a "One Time Script" feature in SmartConsole or with API may fail after 5 minutes with the "Operation timed out" error.
The limit for reading files is 9,730 lines or 730 KB (whichever is reached first)."

TAC has asked us to open an RFE to address this limitation.

Regards,

Simon

[Art_Zalenekas]

OK, thank you for the update!

[Hugo_vd_Kooij]

I see you use cp_mgmt_run_script. But I would expect running the command directly with the build-in Shell would propably do the job just fine. Have you tried that approach?