Solved: Servers behind Azure Cloudguard HA do not have internet access

[RickyDan]

Basic network diagram as shown in image. Cluster configuration below:

1. In network management, eth1 (backend) leads to server subnet.
2. In firewall policy, server subnet has internet access.
3. In NAT policy, server subnet is SNAT'd to frontend private cluster IP address for internet traffic.
4. In Gaia OS, static route to server subnet via backend subnet (azure default gateway IP address).

The firewall cluster itself has internet access - I can ping and curl public IP addresses and websites. The logs also show that server traffic is hitting the firewall and being accepted and NAT'd.

Edit: This is version R81.10 template and tcpdump on eth0 shows SNAT'd traffic leaving the interface but no return traffic.

Solved: This was actually a HA issue. Firewall2 was the active when the internet was not working. I booted up only Firewall1 this morning (I shutdown all VMs overnight) and the internet was working for the servers. 

I gave contributor permissions to the automatically created managed identities on the VNET where the firewalls are located and also on the resource group for my IP prefixes ($FWDIR/scripts/azure_ha_test.py complained about this).

[PhoneBoy]

Version/JHF?
What do you see in logs?
What do you see in tcpdump or fw monitor?

[RickyDan]

This is the R81.10 template and The logs also show that server traffic is hitting the firewall and being accepted and NAT'd. When I did a tcpdump on eth0, I saw SNAT'd traffic going towards the internet but no return traffic. I did not do a fw monitor.