CloudGuard - Failed to fetch objects from the data centers

[eltonsimoes]

Hi CheckMates!

This is my post here, so I apologize in advance if I make any mistakes.

I'm analyzing a problem from a customer regarding the failure to import objects from vCenter through SmartConsole, as shown in the attached image.

When analyzing the logs of cpm.elg I noticed the following error:

23/11/22 10:13:03,238 ERROR services.internal.CloudSvcImpl [qtp-26099087-628035]: Connection failed - Got error:UNKNOWN_PROBLEM
23/11/22 10:13:03,239 ERROR services.internal.CloudSvcImpl [qtp-26099087-628051]: Connection failed - Got error:UNKNOWN_PROBLEM
23/11/22 10:13:03,239 ERROR services.internal.CloudSvcImpl [qtp-26099087-628051]: Connection to vcenter.localdomain.com.br failed
23/11/22 10:13:03,239 ERROR cloud.connection.GetCmsNodeChildrenCodeQueryHandler [qtp-26099087-628051]: failed to execute cloud command. (search command). got internal error. error:Failed to connect due to an unknown problem.

My environment consists of the following items:

SMS - version R81.10 take JHF 66;
CME - version 219 build 991592199;
VMWare vCenter - version 7.0.3 build 20150588;

Has any colleague had this problem?

[_Val_ · Accepted solution]

Looks like a connectivity issue to me. Start with this assumption, and if it is not the case, come back with more info.

[eltonsimoes]

Thanks for the reply. Following your suggestion of a connectivity-related issue, I redid the creation of the Data Centers object in SmartConsole. After that I was able to import objects from vCenter. Looking at the log information, reporting an unknown error, I have a concern that the problem might reoccur. Would it be the case to open a ticket with the TAC?

[the_rock]

Can you attach a screenshot?

[_Val_]

Yes, by all means, take this to TAC

[eltonsimoes]

Thanks for the reply. Following your suggestion of a connectivity-related issue, I redid the creation of the Data Centers object in SmartConsole. After that I was able to import objects from vCenter. 

[lolith]

Hi All,

We are also facing the exact same issue but with Azure.

13/09/23 10:30:35,932 ERROR datacenter.scanner.DcScanner [scanner-Azure-xxx]: Error during scan - attempting to reconnect for scanner Azure [Application id xxx, directory id xxx] com.checkpoint.datacenter.util.exception.UnknownProblemException: Failed querying Azure, unknown problem at com.checkpoint.datacenter.scanner.azure.AzureDeployment.getAzureResponse(AzureDeployment.java:260) at com.checkpoint.datacenter.scanner.azure.AzureScanner.innerRun(AzureScanner.java:135) at com.checkpoint.datacenter.scanner.DcScanner.run(DcScanner.java:120) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522) at java.util.concurrent.FutureTask.run(FutureTask.java:277) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.lang.Thread.run(Thread.java:820)

We did try to create a new DC object, but that did not help. I just want to know when you said you re-created a new DC object, did you mean you removed the old one and created new? If yes, what did you do with the values associated with old DC object.

To solve this issue, we tried a lot of things, but restarting vsec process, installing latest JHF, rebooting the mgmt. The DC object itself shows connected when running test towards that. Wondering what it could be?

cpstat vsec output shows connected. But at time it also shows unknown problem.

vSEC Controller Status: on
Number of disconnected Data Centers: 0
Number of Data Centers: 1
Number of imported Data Center objects: 24
Number of gateways enforcing Data Center objects: 7

Data Centers
--------------------------------------------------------------------------------------------------------
|Name |Data Center type|Connection status|Imported Data Center objects|CloudGuard Controller updates|
--------------------------------------------------------------------------------------------------------
|AzureDev|Microsoft Azure |Connected | 24| 0|
--------------------------------------------------------------------------------------------------------

cpstat vsec

vSEC Controller Status:              on

Number of disconnected Data Centers:       2

Number of Data Centers:              2

Number of imported Data Center objects:      24

Number of gateways enforcing Data Center objects: 7

Data Centers

|Name   |Data Center type|Connection status               |Imported Data Center objects|CloudGuard Controller updates|

|AzureDev |Microsoft Azure |Unknown problem, since Wed 2023-09-13 11:05:36|             24|              0|

|AzureDev2|Microsoft Azure |Unknown problem, since Wed 2023-09-13 11:05:36|              0|              0|

--------------------------------------------------------------------------------------------------------------------------------------

FYI, have a raised a TAC case also - 6-0003721994 in the meantime. Any lead will help us a lot.

Thanks,

Lolith

[Gil_Sudai]

What do you get when you run this command on the mgmt:

AZURE_CREDENTIALS='{"client_id": "<client id>","client_secret": "<client secret>","grant_type": "client_credentials","tenant": "<tenant>"}' $FWDIR/Python/bin/python3 $VSECDIR/scripts/azure/vsec.p> out.json 2>&1

Do you get a valid JSON file?

[lolith]

Hi,

Thanks for the response.

Nothing happens when I run the below command. Just sit on the next time, without any output.

AZURE_CREDENTIALS='{"client_id": "xxxxx","client_secret": "xxxx","grant_type": "client_credentials","tenant": "xxx"}' $FWDIR/Python/bin/python3 $VSECDIR/scripts/azure/vsec.py> out.json 2>&1

But when i ran the above command with out.json,

Its gives this error:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 131, in _arm
raise ARMException(problem)
ARMException: ConnectivityProblem
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 695, in main
sub = get_sub(sub_obj, args.asg_pep)
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 479, in get_sub
vms = vms_ar.get()
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 92, in get
self.headers, self.body = self.ar.get((1 << 31) - 1)
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
ARMException: ConnectivityProblem
failed to retrieve subscription: 'AZF-DCExt-Spoke1-Prod' subscriptionId: xxxxx 1/5
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 110, in _arm
return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 502 Bad Gateway
b'"{\\"error\\":{\\"code\\":\\"ProviderError\\",\\"message\\":\\"Resource provider \'Microsoft.Compute\' failed to return collection response for type \'virtualMachines\'.\\"}}"'
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 110, in _arm
return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 502 Bad Gateway
b'"{\\"error\\":{\\"code\\":\\"ProviderError\\",\\"message\\":\\"Resource provider \'Microsoft.Compute\' failed to return collection response for type \'virtualMachines\'.\\"}}"'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 131, in _arm
raise ARMException(problem)
ARMException: ConnectivityProblem
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 695, in main
sub = get_sub(sub_obj, args.asg_pep)
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 479, in get_sub
vms = vms_ar.get()
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 92, in get
self.headers, self.body = self.ar.get((1 << 31) - 1)
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
ARMException: ConnectivityProblem
failed to retrieve subscription: 'AZF-DCExt-Spoke1-Prod' subscriptionId: xxxxx 2/5

It shows connectivity problem, but how to check that? All seems to right towards Azure. We have three env/sub, 2 having issue, and one works well.

[Gil_Sudai]

Is there a proxy or such device between the mgmt server and the cloud?

Do you get connectivity errors for this command?:

curl_cli --verbose https://management.azure.com --cacert $CPDIR/conf/ca-bundle-public-cloud.crt

The above will not return 200/OK. But I want to see if it will also report connectivity errors.

[lolith]

No proxy involved.

This connects fine:

curl_cli --verbose https://management.azure.com --cacert $CPDIR/conf/ca-bundle-public-cloud.crt
* Rebuilt URL to: https://management.azure.com/
* Trying 51.120.46.144...
* TCP_NODELAY set
* Connected to management.azure.com (51.120.46.144) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /opt/CPshrd-R81.10/conf/ca-bundle-public-cloud.crt
CApath: none
* *** Current date is: Wed Sep 13 14:36:31 2023
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* err is -1, detail is 2
* *** Current date is: Wed Sep 13 14:36:31 2023
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* err is -1, detail is 2
* *** Current date is: Wed Sep 13 14:36:31 2023
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* servercert: Activated
* servercert: crl_download_timeout: 10
* servercert: crl_weak_validation: 1
* servercert: Calling cp_verify_certificate
* servercert: cp_verify_certificate returned: CURLE_OK
* Server certificate:
* subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=management.azure.com
* start date: Aug 30 18:56:32 2023 GMT
* expire date: Jun 27 23:59:59 2024 GMT
* verifyhost: No 'trusted CN' provided.
* subjectAltName: host "management.azure.com" matched cert's "management.azure.com"
* verifyhost: Alternative name 'management.azure.com' matches the destination hostname.
* issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 01
* SSL certificate verify ok.
* servercert: Finished
< HTTP/1.1 400 Bad Request
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: application/json; charset=utf-8
< Expires: -1
< x-ms-failure-cause: gateway
< x-ms-request-id: 1293d467-026e-4766-86b1-f2cc647c56d7
< x-ms-correlation-request-id: 1293d467-026e-4766-86b1-f2cc647c56d7
< x-ms-routing-request-id: NORWAYEAST:20230913T123632Z:1293d467-026e-4766-86b1-f2cc647c56d7
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< X-Content-Type-Options: nosniff
< Date: Wed, 13 Sep 2023 12:36:31 GMT
< Content-Length: 137
<
* Connection #0 to host management.azure.com left intact
{"error":{"code":"MissingApiVersionParameter","message":"The api-version query parameter (?api-version=) is required for all

[Gil_Sudai]

The response that we get from Azure is 

HTTP/1.1 502 Bad Gateway
b'"{\\"error\\":{\\"code\\":\\"ProviderError\\",\\"message\\":\\"Resource provider \'Microsoft.Compute\' failed to return collection response for type \'virtualMachines\'.\\"}}"'

I never saw it before. Do you have a way to ask Azure why they return it?  I think that you have the ID of the request in the log, Azure can use it to track their own side. 

[lolith]

Yes, interesting. Will check and keep digging further. Thanks!

Update - 14.09.23

Yesterday night around 10PM, it started working normally again. It shows connected to DC and fetching tag information as well. So, something at Azure end which might have caused this issue. Will keep digging for root cause. Thanks for your support.

Regards,

Lolith