CloudGuard Controller Azure API outage

[Duane_Toler]

Hey all, 

One of my customers endured an outage across their firewalls when using the Azure API dynamic objects.  Best we can tell, Azure had an "issue" that they have yet to admit or explain.

I had a Critical TAC case yesterday and a group call with Check Point TAC, Microsoft Azure support (Sev A case), and more than enough customer managers, group directors, and team leaders.  

When "it" happened, the management server got a poll response from Azure API that said "you have no resources", and a mass "delete-identity" IA API command was sent to all CloudGuard gateways.. zapping hundreds upon hundreds of mapped identities.  "oops".  Even in SmartConsole, the Data Center browser wasn't showing the Azure subscriptions!  I eventually restarted the vSEC controller and they all came pouring in again, and the identities were added back to the gateways!  Yet, 3 minutes later, they all were stripped out.  However, this second time, the identities weren't deleted from the gateways; they just weren't visible in SmartConsole anymore.

Later, in the late afternoon, a second vSEC controller restart was done this time with debugging enabled, and everything has been stable.

We poured over the cloud_proxy.elg debugs during all of this.  After stripping out the Bearer token strings, we uploaded this debug to Microsoft Azure support who will relay it to their API people.

This morning, Check Point TAC came back saying they had multiple cases for this issue from other customers, but I haven't gotten any concrete info on what happened.  Best estimates at this point are "Azure API people did something".

Good luck to everyone and I hope you all were mostly spared!

[Duane_Toler]

The briefest TAC update ever, but still pending more details:

I have been provided the following information:
The symptoms as a result of a glitch on Microsoft Azure side.

 TBD....

[noyerez]

Hi @Duane_Toler,

We are aware of the issue and are currently investigating it as a top priority. My team is working with our Microsoft Azure peers to identify the root cause.

We will keep you updated on our progress.