Vsec controller fetch dynamic object error

Kim_Moberg — Mon, 06 Nov 2023 19:33:11 GMT

Does the vsec controller have an issue with Azure VMSS scale set orchestrator mode when set to “flexible”?

I have an issue with vsec controller aka Cloudguard Controller that fetches dynamic objects from Azure AD.

When monitoring cloud_proxy.elg it should an error before it starts important objects. The error is a newly generated Azure VMSS Scale Set.

When we remove the scale set the vsec controller imports Azure AD objects when we deploy it again it stops working.

Therefore the focus are at the Azure VMSS Scale Set.

I see the error exception points to the new standard when to generated a new Scale Set and is related to Orchestration Mode. Newly Scale set the Orchestration Mode are “flexible” previously its “Uniform”.

Looking forward to be hearing from you.

Re: Vsec controller fetch dynamic object error

Kim_Moberg — Mon, 06 Nov 2023 19:53:44 GMT

Out from cloud_proxy.elg

——

06/11/23 14:42:11,058 ERROR scanner.azure.AzureDeployment [scanner-Azure-952355775]: Command failed:
ProcessOutputData{
ProcessBuilder= [/opt/CPsuite-R81.10/fw1/Python/bin/python3, /opt/CPvsec-R81.10/scripts/azure/vsec.py, --max-time, 5000]
errCode= 1
processTimeoutSeconds= 100000
processDurationSeconds= 409
Process Output String=
Process Error String= Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 409
b'{\r\n "error": {\r\n    "code": "OperationNotAllowed",\r\n    "message": "Operation \'VirtualMachineScaleSets.virtualMachines.GET\' is not allowed on Virtual Machine Scale Set \'test-scale-set\'."\r\n }\r\n}'
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 409
b'{\r\n "error": {\r\n    "code": "OperationNotAllowed",\r\n    "message": "Operation \'VirtualMachineScaleSets.virtualMachines.GET\' is not allowed on Virtual Machine Scale Set \'Test-Scale-set\'."\r\n }\r\n}'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 128, in _arm
raise ARMException(problem)
ARMException: ConnectivityProblem
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 679, in main
    sub = get_sub(sub_obj, args.asg_pep)
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 524, in get_sub
    vmss_obj[VIRTUAL_MACHINES] = vmss_obj[VIRTUAL_MACHINES].get()
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 91, in get
    self.headers, self.body = self.ar.get((1 << 31) - 1)
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
ARMException: ConnectivityProblem
failed to retrieve subscription: 'subscription01' subscriptionId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx 1/5
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 404 Not Found
b'{\r\n "error": {\r\n    "code": "NotFound",\r\n    "message": "Resource /subscriptions/xxxxxxxxx-xxxx-xxxx-xxxxxxxxx/resourceGroups/test-rg/providers/Microsoft.Compute/virtualMachineScaleSets/test-scale-set not found.",\r\n    "details": []\r\n }\r\n}'
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 404 Not Found
b'{\r\n "error": {\r\n    "code": "NotFound",\r\n    "message": "Resource /subscriptions/xxxxxxxxx-xxxx-xxxx-xxxxxxxx/resourceGroups/test-rg/providers/Microsoft.Compute/virtualMachineScaleSets/test-scale-set not found.",\r\n    "details": []\r\n }\r\n}'
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 409
b'{\r\n "error": {\r\n    "code": "OperationNotAllowed",\r\n    "message": "Operation \'VirtualMachineScaleSets.virtualMachines.GET\' is not allowed on Virtual Machine Scale Set \'test-scale-set\'."\r\n }\r\n}'
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 409
b'{\r\n "error": {\r\n    "code": "OperationNotAllowed",\r\n    "message": "Operation \'VirtualMachineScaleSets.virtualMachines.GET\' is not allowed on Virtual Machine Scale Set \'test-scale-set\'."\r\n }\r\n}'

During handling of the above exception, another exception occurred:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 679, in main
    sub = get_sub(sub_obj, args.asg_pep)
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 524, in get_sub
    vmss_obj[VIRTUAL_MACHINES] = vmss_obj[VIRTUAL_MACHINES].get()
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 91, in get
    self.headers, self.body = self.ar.get((1 << 31) - 1)
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
ARMException: ConnectivityProblem
failed to retrieve subscription: 'subscription01' subscriptionId: xxxxxx-xxxx-xxxxx-xxxx-xxxxxxxxxx 2/5
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 404 Not Found
b'{\r\n "error": {\r\n    "code": "NotFound",\r\n    "message": "Resource /subscriptions/xxxxxx-xxxx-xxxxx-xxxxx-xxxxxxxxx/resourceGroups/test-rg/providers/Microsoft.Compute/virtualMachineScaleSets/test-scale-set not found.",\r\n    "details": []\r\n }\r\n}'
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 404 Not Found
b'{\r\n "error": {\r\n    "code": "NotFound",\r\n    "message": "Resource /subscriptions/xxxxxxxx-xxxxx-xxxx-xxxx-xxxxxxxxx/resourceGroups/test-rg/providers/Microsoft.Compute/virtualMachineScaleSets/test-scale-set not found.",\r\n    "details": []\r\n }\r\n}'
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 409
b'{\r\n "error": {\r\n    "code": "OperationNotAllowed",\r\n    "message": "Operation \'VirtualMachineScaleSets.virtualMachines.GET\' is not allowed on Virtual Machine Scale Set \'test-scale-ser\'."\r\n }\r\n}'
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 409
b'{\r\n "error": {\r\n    "code": "OperationNotAllowed",\r\n    "message": "Operation \'VirtualMachineScaleSets.virtualMachines.GET\' is not allowed on Virtual Machine Scale Set \'test-scale-ser\'."\r\n }\r\n}'

During handling of the above exception, another exception occurred:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 679, in main
    sub = get_sub(sub_obj, args.asg_pep)
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 524, in get_sub
    vmss_obj[VIRTUAL_MACHINES] = vmss_obj[VIRTUAL_MACHINES].get()
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 91, in get
    self.headers, self.body = self.ar.get((1 << 31) - 1)
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
ARMException: ConnectivityProblem
failed to retrieve subscription: 'subscription01' subscriptionId: xxxxxx-xxxx-xxxxx-xxxx-xxxxxxxxx 3/5
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 404 Not Found
b'{\r\n "error": {\r\n    "code": "NotFound",\r\n    "message": "Resource /subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxxxx/resourceGroups/test-rg/providers/Microsoft.Compute/virtualMachineScaleSets/test-scale-set not found.",\r\n    "details": []\r\n }\r\n}'
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 404 Not Found
b'{\r\n "error": {\r\n    "code": "NotFound",\r\n    "message": "Resource /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx/resourceGroups/test-rg/providers/Microsoft.Compute/virtualMachineScaleSets/test-scale-setnot found.",\r\n    "details": []\r\n }\r\n}'
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 409
b'{\r\n "error": {\r\n    "code": "OperationNotAllowed",\r\n    "message": "Operation \'VirtualMachineScaleSets.virtualMachines.GET\' is not allowed on Virtual Machine Scale Set \'test-scale-set\'."\r\n }\r\n}'
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 409
b'{\r\n "error": {\r\n    "code": "OperationNotAllowed",\r\n    "message": "Operation \'VirtualMachineScaleSets.virtualMachines.GET\' is not allowed on Virtual Machine Scale Set \'test-scale-set\'."\r\n }\r\n}'

During handling of the above exception, another exception occurred:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 679, in main
    sub = get_sub(sub_obj, args.asg_pep)
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 524, in get_sub
    vmss_obj[VIRTUAL_MACHINES] = vmss_obj[VIRTUAL_MACHINES].get()
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 91, in get
    self.headers, self.body = self.ar.get((1 << 31) - 1)
File "/opt/CPsuite-R81.10/fw1/Python/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
ARMException: ConnectivityProblem
failed to retrieve subscription: 'Subscription01' subscriptionId: xxxxxxxx-xxxx-xxxx- xxxx-xxxxxxxxxx 4/5
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 404 Not Found
b'{\r\n "error": {\r\n    "code": "NotFound",\r\n    "message": "Resource /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx/resourceGroups/test-rg/providers/Microsoft.Compute/virtualMachineScaleSets/test-scale-setnot found.",\r\n    "details": []\r\n }\r\n}'
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 404 Not Found
b'{\r\n "error": {\r\n    "code": "NotFound",\r\n    "message": "Resource /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx/resourceGroups/test-rg/providers/Microsoft.Compute/virtualMachineScaleSets/test-scale-set not found.",\r\n    "details": []\r\n }\r\n}'
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 409
b'{\r\n "error": {\r\n    "code": "OperationNotAllowed",\r\n    "message": "Operation \'VirtualMachineScaleSets.virtualMachines.GET\' is not allowed on Virtual Machine Scale Set \'test-scale-set\'."\r\n }\r\n}'
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/CPvsec-R81.10/scripts/azure/vsec.py", line 109, in _arm
    return rest.arm(*args, **kwargs)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 536, in arm
    max_time=self.max_time)
File "/opt/CPvsec-R81.10/scripts/azure/rest.py", line 153, in request
    headers['proto'], headers['code'], headers['reason'], response)
rest.RequestException: HTTP/1.1 409
b'{\r\n "error": {\r\n    "code": "OperationNotAllowed",\r\n    "message": "Operation \'VirtualMachineScaleSets.virtualMachines.GET\' is not allowed on Virtual Machine Scale Set \'test-scale-set\'."\r\n }\r\n}'

During handling of the above exception, another exception occurred:

The above exception was the direct cause of the following exception:

During handling of the above exception, another exception occurred:

The above exception was the direct cause of the following exception:

}

——-

Re: Vsec controller fetch dynamic object error

Kim_Moberg — Mon, 06 Nov 2023 19:59:13 GMT

The output above, the script is trying to create ‘VirtualMachineScaleSets.virtualMachines.GET' to the Azure VMSS scale set, which is not supported in "Flexible Orchestration mode". Thats why it throw an http 409 (conflict) error

Other VMSS scale set we have are set Orchestration mode to “Uniform”.

https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/orchestration-modes-api-comparison

Re: Vsec controller fetch dynamic object error

amoss — Tue, 07 Nov 2023 07:56:01 GMT

Hi @Kim_Moberg ,

We have a fix for Azure VMSS flexible in the upcoming Jumbo, they can ask for a port fix.

Best regards,

Amos Shmuel.

Re: Vsec controller fetch dynamic object error

Kim_Moberg — Tue, 12 Dec 2023 12:17:42 GMT

Hi @amoss

We have tried to follow TAC suggestion to upgrade our existing MDS installation on behalf of below info.

"The fix is done on top T95 and ID is PRHF-29780 OR else upgraded to Take 110 as it is integrated into."

We ended up last week to upgrade from JHF94 to JHF 110 as recommended because the custom fix was fully implemented into the recommended release.

Now after the upgrade to JHF 110 we are still having the issue.

Now questions a raise.
Are there other dependencies other than having the MDS server upgraded to JHF 110?

The Cloudguard Controller running on the MDS Server are fetching objects from Azure.

Does the VSX Cluster Gateway HA run any services that which are dependent for fetching dynamic objects from Azure?

E.g., Would the VSX Cluster also need to be running JHF 110 or can that one run with a different version?

Just asking because the Cloudguard Controller are only running from the MDS server and that is the only part that has been upgraded to JHF 110.

I cannot find any information to the ARTG: Cloudguard Controller (sk115657) guide on the support page telling how it works.

topic Re: Vsec controller fetch dynamic object error in Cloud Firewall

Vsec controller fetch dynamic object error

Re: Vsec controller fetch dynamic object error

Re: Vsec controller fetch dynamic object error

Re: Vsec controller fetch dynamic object error

Re: Vsec controller fetch dynamic object error