topic Re: CloudGuard - Failed to fetch objects from the data centers in Cloud Firewall

CloudGuard - Failed to fetch objects from the data centers

eltonsimoes — Wed, 23 Nov 2022 14:59:30 GMT

Hi CheckMates!

This is my post here, so I apologize in advance if I make any mistakes.

I'm analyzing a problem from a customer regarding the failure to import objects from vCenter through SmartConsole, as shown in the attached image.

When analyzing the logs of cpm.elg I noticed the following error:

23/11/22 10:13:03,238 ERROR services.internal.CloudSvcImpl [qtp-26099087-628035]: Connection failed - Got error:UNKNOWN_PROBLEM
23/11/22 10:13:03,239 ERROR services.internal.CloudSvcImpl [qtp-26099087-628051]: Connection failed - Got error:UNKNOWN_PROBLEM
23/11/22 10:13:03,239 ERROR services.internal.CloudSvcImpl [qtp-26099087-628051]: Connection to vcenter.localdomain.com.br failed
23/11/22 10:13:03,239 ERROR cloud.connection.GetCmsNodeChildrenCodeQueryHandler [qtp-26099087-628051]: failed to execute cloud command. (search command). got internal error. error:Failed to connect due to an unknown problem.

My environment consists of the following items:

SMS - version R81.10 take JHF 66;
CME - version 219 build 991592199;
VMWare vCenter - version 7.0.3 build 20150588;

Has any colleague had this problem?

Re: CloudGuard - Failed to fetch objects from the data centers

_Val_ — Wed, 23 Nov 2022 15:01:17 GMT

Looks like a connectivity issue to me. Start with this assumption, and if it is not the case, come back with more info.

Re: CloudGuard - Failed to fetch objects from the data centers

eltonsimoes — Wed, 23 Nov 2022 20:38:14 GMT

Thanks for the reply. Following your suggestion of a connectivity-related issue, I redid the creation of the Data Centers object in SmartConsole. After that I was able to import objects from vCenter. Looking at the log information, reporting an unknown error, I have a concern that the problem might reoccur. Would it be the case to open a ticket with the TAC?

Re: CloudGuard - Failed to fetch objects from the data centers

the_rock — Wed, 23 Nov 2022 20:56:22 GMT

Can you attach a screenshot?

Re: CloudGuard - Failed to fetch objects from the data centers

_Val_ — Thu, 24 Nov 2022 07:37:02 GMT

Yes, by all means, take this to TAC

Re: CloudGuard - Failed to fetch objects from the data centers

eltonsimoes — Mon, 28 Aug 2023 04:12:28 GMT

Thanks for the reply. Following your suggestion of a connectivity-related issue, I redid the creation of the Data Centers object in SmartConsole. After that I was able to import objects from vCenter.

Re: CloudGuard - Failed to fetch objects from the data centers

lolith — Wed, 13 Sep 2023 10:37:41 GMT

Hi All,

We are also facing the exact same issue but with Azure.

13/09/23 10:30:35,932 ERROR datacenter.scanner.DcScanner [scanner-Azure-xxx]: Error during scan - attempting to reconnect for scanner Azure [Application id xxx, directory id xxx] com.checkpoint.datacenter.util.exception.UnknownProblemException: Failed querying Azure, unknown problem at com.checkpoint.datacenter.scanner.azure.AzureDeployment.getAzureResponse(AzureDeployment.java:260) at com.checkpoint.datacenter.scanner.azure.AzureScanner.innerRun(AzureScanner.java:135) at com.checkpoint.datacenter.scanner.DcScanner.run(DcScanner.java:120) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522) at java.util.concurrent.FutureTask.run(FutureTask.java:277) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.lang.Thread.run(Thread.java:820)

We did try to create a new DC object, but that did not help. I just want to know when you said you re-created a new DC object, did you mean you removed the old one and created new? If yes, what did you do with the values associated with old DC object.

To solve this issue, we tried a lot of things, but restarting vsec process, installing latest JHF, rebooting the mgmt. The DC object itself shows connected when running test towards that. Wondering what it could be?

cpstat vsec output shows connected. But at time it also shows unknown problem.

vSEC Controller Status: on
Number of disconnected Data Centers: 0
Number of Data Centers: 1
Number of imported Data Center objects: 24
Number of gateways enforcing Data Center objects: 7

cpstat vsec

vSEC Controller Status: on

Number of disconnected Data Centers: 2

Number of Data Centers: 2

Number of imported Data Center objects: 24

Number of gateways enforcing Data Center objects: 7

Data Centers

--------------------------------------------------------------------------------------------------------------------------------------

FYI, have a raised a TAC case also - 6-0003721994 in the meantime. Any lead will help us a lot.

Thanks,

Lolith

Re: CloudGuard - Failed to fetch objects from the data centers

Gil_Sudai — Wed, 13 Sep 2023 10:43:43 GMT

What do you get when you run this command on the mgmt:

AZURE_CREDENTIALS='{"client_id": "<client id>","client_secret": "<client secret>","grant_type": "client_credentials","tenant": "<tenant>"}' $FWDIR/Python/bin/python3 $VSECDIR/scripts/azure/vsec.p> out.json 2>&1

Do you get a valid JSON file?

Re: CloudGuard - Failed to fetch objects from the data centers

lolith — Wed, 13 Sep 2023 11:08:41 GMT

Hi,

Thanks for the response.

Nothing happens when I run the below command. Just sit on the next time, without any output.

AZURE_CREDENTIALS='{"client_id": "xxxxx","client_secret": "xxxx","grant_type": "client_credentials","tenant": "xxx"}' $FWDIR/Python/bin/python3 $VSECDIR/scripts/azure/vsec.py> out.json 2>&1

But when i ran the above command with out.json,

Its gives this error:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 131, in _arm
raise ARMException(problem)
ARMException: ConnectivityProblem
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 695, in main
sub = get_sub(sub_obj, args.asg_pep)
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 479, in get_sub
vms = vms_ar.get()
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 92, in get
self.headers, self.body = self.ar.get((1 << 31) - 1)
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
ARMException: ConnectivityProblem
failed to retrieve subscription: 'AZF-DCExt-Spoke1-Prod' subscriptionId: xxxxx 1/5
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 110, in _arm
return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 502 Bad Gateway
b'"{\\"error\\":{\\"code\\":\\"ProviderError\\",\\"message\\":\\"Resource provider \'Microsoft.Compute\' failed to return collection response for type \'virtualMachines\'.\\"}}"'
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 110, in _arm
return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 502 Bad Gateway
b'"{\\"error\\":{\\"code\\":\\"ProviderError\\",\\"message\\":\\"Resource provider \'Microsoft.Compute\' failed to return collection response for type \'virtualMachines\'.\\"}}"'

During handling of the above exception, another exception occurred:

The above exception was the direct cause of the following exception:

Re: CloudGuard - Failed to fetch objects from the data centers

Gil_Sudai — Wed, 13 Sep 2023 12:13:33 GMT

Is there a proxy or such device between the mgmt server and the cloud?

Do you get connectivity errors for this command?:

curl_cli --verbose https://management.azure.com --cacert $CPDIR/conf/ca-bundle-public-cloud.crt

The above will not return 200/OK. But I want to see if it will also report connectivity errors.

Re: CloudGuard - Failed to fetch objects from the data centers

lolith — Wed, 13 Sep 2023 12:39:24 GMT

No proxy involved.

This connects fine:

curl_cli --verbose https://management.azure.com --cacert $CPDIR/conf/ca-bundle-public-cloud.crt
* Rebuilt URL to: https://management.azure.com/
* Trying 51.120.46.144...
* TCP_NODELAY set
* Connected to management.azure.com (51.120.46.144) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /opt/CPshrd-R81.10/conf/ca-bundle-public-cloud.crt
CApath: none
* *** Current date is: Wed Sep 13 14:36:31 2023
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* err is -1, detail is 2
* *** Current date is: Wed Sep 13 14:36:31 2023
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* err is -1, detail is 2
* *** Current date is: Wed Sep 13 14:36:31 2023
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* servercert: Activated
* servercert: crl_download_timeout: 10
* servercert: crl_weak_validation: 1
* servercert: Calling cp_verify_certificate
* servercert: cp_verify_certificate returned: CURLE_OK
* Server certificate:
* subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=management.azure.com
* start date: Aug 30 18:56:32 2023 GMT
* expire date: Jun 27 23:59:59 2024 GMT
* verifyhost: No 'trusted CN' provided.
* subjectAltName: host "management.azure.com" matched cert's "management.azure.com"
* verifyhost: Alternative name 'management.azure.com' matches the destination hostname.
* issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 01
* SSL certificate verify ok.
* servercert: Finished
< HTTP/1.1 400 Bad Request
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: application/json; charset=utf-8
< Expires: -1
< x-ms-failure-cause: gateway
< x-ms-request-id: 1293d467-026e-4766-86b1-f2cc647c56d7
< x-ms-correlation-request-id: 1293d467-026e-4766-86b1-f2cc647c56d7
< x-ms-routing-request-id: NORWAYEAST:20230913T123632Z:1293d467-026e-4766-86b1-f2cc647c56d7
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< X-Content-Type-Options: nosniff
< Date: Wed, 13 Sep 2023 12:36:31 GMT
< Content-Length: 137
<
* Connection #0 to host management.azure.com left intact
{"error":{"code":"MissingApiVersionParameter","message":"The api-version query parameter (?api-version=) is required for all

Re: CloudGuard - Failed to fetch objects from the data centers

Gil_Sudai — Wed, 13 Sep 2023 13:05:23 GMT

The response that we get from Azure is

HTTP/1.1 502 Bad Gateway
b'"{\\"error\\":{\\"code\\":\\"ProviderError\\",\\"message\\":\\"Resource provider \'Microsoft.Compute\' failed to return collection response for type \'virtualMachines\'.\\"}}"'

I never saw it before. Do you have a way to ask Azure why they return it? I think that you have the ID of the request in the log, Azure can use it to track their own side.

Re: CloudGuard - Failed to fetch objects from the data centers

lolith — Thu, 14 Sep 2023 07:03:31 GMT

Yes, interesting. Will check and keep digging further. Thanks!

Update - 14.09.23

Yesterday night around 10PM, it started working normally again. It shows connected to DC and fetching tag information as well. So, something at Azure end which might have caused this issue. Will keep digging for root cause. Thanks for your support.

Regards,

Lolith