topic VMSS R80.40 deployment -- Delay 30 minutes !!! in Cloud Firewall

VMSS R80.40 deployment -- Delay 30 minutes !!!

Abhishek_Singh1 — Fri, 28 Aug 2020 09:45:28 GMT

Hi Mates,

So recently I am involved in deploying Checkpoint VMSS R80.40 ver in Azure Cloud . I have observed that once the VM instance gets spinned up , it takes almost 30 to 40+ odd minutes for it to get reflected on the Management ( including SIC , restrictive policy install plus Dedicated policy install) ...

Why is that delay happening ?? Anyone facing similar prob ?

Here are more details -

cloudgaurd version of VMSS -

release: R80.40
take: 294
build: 640
platform: azure
license: byol
deployment_method: GW
template_name: vmss-v2
template_version: 20200808

CME version -

Build: 991000664 Take: 121

CME config -

controllers:
Azure-Pre-Prod:
class: Azure
credentials:
"client_id": xxxxx
"client_secret": "xxxxxx"
"grant_type": "client_credentials"
tenant: "xxxxx"
subscription: "xxxxxx"
delay: 30
management:
host: localhost
name: yyyyy
templates:
Pre-Prod-VMSS:
anti-bot: true
anti-virus: true
application-control: true
https-inspection: true
ips: true
one-time-password: "_xxxx/one-time-password"
policy: "VMSS_R80.40"
threat-emulation: true
url-filtering: true
version: "R80.40"

@PhoneBoy --- Can you share any pointers here ? Is it because of the above delay param I see in CME config ? I dont see any way to edit this in the CME admin guide tho .

Re: VMSS R80.40 deployment -- Delay 30 minutes !!!

PhoneBoy — Fri, 28 Aug 2020 22:58:08 GMT

Pretty sure the delay is in seconds, not minutes.
What does the /var/log/CPcme/cme.log say?

Re: VMSS R80.40 deployment -- Delay 30 minutes !!!

Abhishek_Singh1 — Sun, 30 Aug 2020 11:42:33 GMT

Yup true , I did changed the delay value to 5 by editing the autprov config json file and restarting the service -- with no luck.

In logs I see these --

2020-08-28 08:40:42,965 CME_SERVICE ERROR The take-over-session API failed: : Management server failed to execute command
2020-08-28 08:40:42,965 CME_SERVICE ERROR Resetting management session id
2020-08-28 08:40:42,967 CME_SERVICE DEBUG
Traceback (most recent call last):
File "/opt/CPcme/cp_handlers/mgmt_handler.py", line 318, in __enter__
version='v1.2')
File "/opt/CPcme/cp_handlers/mgmt_handler.py", line 202, in __call__
raise Exception('failed API call: %s%s' % (command, msg))
Exception: failed API call: take-over-session: Management server failed to execute command
2020-08-28 08:40:42,967 CME_SERVICE INFO discard uid 47c495a1-e6de-4b26-a675-5f843b3b3422: failed
2020-08-28 08:40:42,968 CME_SERVICE INFO restoring sid
2020-08-28 08:40:42,968 CME_SERVICE DEBUG Executing get-interfaces-sync management API command with body {'__no_such_parameter__': None}
2020-08-28 08:40:42,991 CME_SERVICE ERROR The get-interfaces-sync API failed: : Unknown API version: 1.7
2020-08-28 08:40:42,991 CME_SERVICE DEBUG Executing get-interfaces-sync management API command with body {'__no_such_parameter__': None}

/////////

020-08-28 09:50:04,219 CME_SERVICE DEBUG In RequestHistory->dispatch key: Method: GET, service provider: Microsoft.Network found
2020-08-28 09:50:04,219 CME_SERVICE DEBUG List currently has 0 elements
2020-08-28 09:50:04,219 CME_SERVICE DEBUG Request count since 2020-08-28 09:47:04.218409 is: 0
2020-08-28 09:50:04,219 CME_SERVICE DEBUG In three minutes gate passed condition
2020-08-28 09:50:04,219 CME_SERVICE DEBUG In 30 min window, window size: 900
2020-08-28 09:50:04,219 CME_SERVICE DEBUG Checking thirty minutes gate condition
2020-08-28 09:50:04,220 CME_SERVICE DEBUG Current time 2020-08-28 09:50:04.220051, time to subtract 0:30:00 time range 2020-08-28 09:20:04.220051
2020-08-28 09:50:04,220 CME_SERVICE DEBUG Getting request list for key: Method: GET, service provider: Microsoft.Network
2020-08-28 09:50:04,220 CME_SERVICE DEBUG In RequestHistory->dispatch key: Method: GET, service provider: Microsoft.Network found
2020-08-28 09:50:04,220 CME_SERVICE DEBUG List currently has 0 elements
2020-08-28 09:50:04,221 CME_SERVICE DEBUG Request count since 2020-08-28 09:20:04.220051 is: 0
2020-08-28 09:50:04,221 CME_SERVICE DEBUG In thirty minutes gate passed condition
2020-08-28 09:50:04,221 CME_SERVICE DEBUG In 60 min window, window size: 12000
2020-08-28 09:50:04,221 CME_SERVICE DEBUG Checking one hour gate condition
2020-08-28 09:50:04,221 CME_SERVICE DEBUG In RequestsHistory:dispatcher->get_requests_list_by_method, returning list of READ requests
2020-08-28 09:50:04,222 CME_SERVICE DEBUG In RequestHistory->dispatch returning list for entire subscription according to method GET
2020-08-28 09:50:04,222 CME_SERVICE DEBUG Requests count in subscription XXXXXXXXXXXXXx for READ is 0
2020-08-28 09:50:04,222 CME_SERVICE DEBUG In one hour gate passed condition
2020-08-28 09:50:04,223 CME_SERVICE DEBUG Request Request method: GET

//////////

2020-08-28 08:56:40,236 CME_SERVICE DEBUG args_no_auth:['curl_cli', '--silent', '--show-error', '--globoff', '--dump-header', '/dev/fd/2', '--noproxy', 'localhost,169.254.169.254', '--url', 'https://management.azure.com/subscriptions/XXXXXXXXXXXXXX/providers/Microsoft.Compute/virtualMachineScaleSets?api-version=2018-06-01', '--request', 'GET', '--header', 'authorization: *****
2020-08-28 08:56:40,363 CME_SERVICE DEBUG curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
2020-08-28 08:56:40,366 CME_SERVICE DEBUG
headers:"curl: (60) SSL certificate problem: self signed certificate in certificate chain\nMore details here: https://curl.haxx.se/docs/sslcerts.html\n\ncurl failed to verify the legitimacy of the server and therefore could not\nestablish a secure connection to it. To learn more about this situation and\nhow to fix it, please visit the web page mentioned above.\n"
2020-08-28 08:56:40,366 CME_SERVICE DEBUG
response:""
2020-08-28 08:56:40,366 CME_SERVICE ERROR Error during gateways synchronization
2020-08-28 08:56:40,375 CME_SERVICE ERROR Error details: Traceback (most recent call last):
File "/opt/CPcme/service/cme_service.py", line 341, in loop
sync(c, management, gateways)
File "/opt/CPcme/service/cme_service.py", line 242, in sync
if controller.filter_instances():
File "/opt/CPcme/cloud_handlers/cloud_common_handler.py", line 643, in filter_instances
for i in self.get_instances():
File "/opt/CPcme/cloud_handlers/azure_handler.py", line 648, in get_instances
instances += self.get_vmss(subnets)
File "/opt/CPcme/cloud_handlers/azure_handler.py", line 469, in get_vmss
vmsss = list(self.retrieve_vmsss().values())
File "/opt/CPcme/cloud_handlers/azure_handler.py", line 98, in retrieve_vmsss
'?api-version=2018-06-01' % self.sub)
File "/opt/CPcme/cloud_connectors/azure.py", line 602, in arm
max_time=self.max_time)
File "/opt/CPcme/cloud_connectors/azure.py", line 155, in request
max_time)
File "/opt/CPcme/cloud_connectors/azure.py", line 264, in request_curl
raise CurlException(headers, args_no_auth, rc)
cloud_connectors.azure.CurlException: b'curl: (60) SSL certificate problem: self signed certificate in certificate chain\nMore details here: https://curl.haxx.se/docs/sslcerts.html\n\ncurl failed to verify the legitimacy of the server and therefore could not\nestablish a secure connection to it. To learn more about this situation and\nhow to fix it, please visit the web page mentioned above.\n'
2020-08-28 08:56:40,376 CME_SERVICE INFO

---

The connector SSL error is tho received only once as I have bypassed HTTPS inspection to "management.azure.com" already .

Also , to be precise I have noticed the total time taken is exact 40 mins every time a new instance spins up .

Re: VMSS R80.40 deployment -- Delay 30 minutes !!!

PhoneBoy — Sun, 30 Aug 2020 21:35:51 GMT

What version of management is running here?
This error message looks suspicious:

2020-08-28 08:40:42,991 CME_SERVICE ERROR The get-interfaces-sync API failed: : Unknown API version: 1.7

API v1.7 is R81, which is not GA yet.
If you’re not using R81 it might be an issue with the version of CME.
Might be a good idea to get the TAC involved.

Re: VMSS R80.40 deployment -- Delay 30 minutes !!!

Noy_Azulay — Mon, 31 Aug 2020 12:57:21 GMT

Hi,
Could you please share more details about your environment:
What is your MGMT GAIA version?
MGMT locate on could/your on-premise network?
API is up and running? (run api status)

also please share output of 'service cme test'

Thanks,

Noy

Re: VMSS R80.40 deployment -- Delay 30 minutes !!!

Ayoub_Bou — Tue, 08 Sep 2020 14:20:55 GMT

mgmt server in R80.40

[Expert@vmck:0]# api status

API Settings:
---------------------
Accessibility: Require all granted
Automatic Start: Enabled

Processes:

Name State PID More Information
-------------------------------------------------
API Started 28970
CPM Started 8235 Check Point Security Management Server is running and ready
FWM Started 7915
APACHE Started 7205

Port Details:
-------------------
JETTY Internal Port: 50276
APACHE Gaia Port: 443

Profile:
------------
Machine profile: Large SMC env resources profile without SME
CPM heap size: 1024m
API heap size: 256m

--------------------------------------------
Overall API Status: Started
--------------------------------------------

API readiness test SUCCESSFUL. The server is up and ready to receive connections

Notes:
------------
To collect troubleshooting data, please run 'api status -s <comment>'

service cme test

}
New management session: e0b2b30b-9504-4a3e-afbf-220ab7a256ff
Discarding management session: 6f87b2cd-35f6-4410-b94b-75a796780935
The get-interfaces-sync API failed: : Unknown API version: 1.7

**********
All tests passed successfully

Re: VMSS R80.40 deployment -- Delay 30 minutes !!!

Ayoub_Bou — Tue, 08 Sep 2020 14:16:17 GMT

same issue

Re: VMSS R80.40 deployment -- Delay 30 minutes !!!

Chris_Atkinson — Wed, 09 Sep 2020 12:28:56 GMT

Was a TAC case raised when you encountered this?