Multiple Azure site regions with Cloudgaard

[Ping_Choi]

Hi,

I was wondering if anyone worked with setting up Cloudgard and interconnecting multiple Azure regions with Expressroute?

If so, how would you solve the issue of multiple Azure regions not being able to completely communicate with each other?

We have an on-prem R80.30 Cluster that can connect to our Azure East region (running Cloudgard) and Azure West region (also running Cloudgard), however the 2 regions are only able to partially talk to each other.

Basic cmds like ICMP and TRACEROUTE from Azure region to Azure region work, but other protocols like RDP cannot. Even though in the Rulebase, I have configured an allow ANY btw both regions.   

Thanks in advance!

[PhoneBoy]

What do the logs on your gateway say?
Have you done a tcpdump to verify packets are reaching the other gateway?

[Ping_Choi]

And here is the log output going in the other direction. This is a machine is the WestUS trying to RDP to a machine in the EastUS region.

192.x.1.x = machine residing in default WestUS region
192.x.2.x = machine residing in remote EastUS region
192.x.GWWEST.x = firewall ip address on internal LB of WestUS region

[CGGWWEST:0]# tcpdump -i eth1 tcp and host 192.x.2.x
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
19:30:05.861584 IP 192.x.1.x.61639 > 192.x.2.x.ms-wbt-server: Flags [S ], seq 2636769634, win 8192, options [mss 1418,nop,wscale 8,nop,nop,sackOK], len gth 0
19:30:05.862027 IP 192.x.GWWEST.x.34545 > 192.x.2.x.ms-wbt-server: Flags [S] , seq 2636769634, win 8192, options [mss 1418,nop,wscale 8,nop,nop,sackOK], leng th 0
19:30:05.941854 IP 192.x.2.x.ms-wbt-server > 192.x.GWWEST.x.34545: Flags [S. ], seq 2293385564, ack 2636769635, win 64000, options [mss 1376,nop,wscale 0,nop ,nop,sackOK], length 0
19:30:05.941908 IP 192.x.2.x.ms-wbt-server > 192.x.1.x.61639: Flags [S .], seq 2293385564, ack 2636769635, win 64000, options [mss 1376,nop,wscale 0,no p,nop,sackOK], length 0
19:30:05.942880 IP 192.x.1.x.61639 > 192.x.2.x.ms-wbt-server: Flags [. ], ack 1, win 1026, length 0
19:30:05.942988 IP 192.x.GWWEST.x.34545 > 192.x.2.x.ms-wbt-server: Flags [.] , ack 1, win 1026, length 0
19:30:05.943749 IP 192.x.1.x.61639 > 192.x.2.x.ms-wbt-server: Flags [P .], seq 1:20, ack 1, win 1026, length 19
19:30:05.944024 IP 192.x.GWWEST.x.34545 > 192.x.2.x.ms-wbt-server: Flags [P. ], seq 1:20, ack 1, win 1026, length 19
19:30:06.243602 IP 192.x.1.x.61639 > 192.x.2.x.ms-wbt-server: Flags [P .], seq 1:20, ack 1, win 1026, length 19
19:30:06.243817 IP 192.x.GWWEST.x.34545 > 192.x.2.x.ms-wbt-server: Flags [P. ], seq 1:20, ack 1, win 1026, length 19
19:30:06.543174 IP 192.x.1.x.61639 > 192.x.2.x.ms-wbt-server: Flags [P .], seq 1:20, ack 1, win 1026, length 19
19:30:06.543311 IP 192.x.GWWEST.x.34545 > 192.x.2.x.ms-wbt-server: Flags [P. ], seq 1:20, ack 1, win 1026, length 19
19:30:07.144209 IP 192.x.1.x.61639 > 192.x.2.x.ms-wbt-server: Flags [P .], seq 1:20, ack 1, win 1026, length 19
19:30:07.144359 IP 192.x.GWWEST.x.34545 > 192.x.2.x.ms-wbt-server: Flags [P. ], seq 1:20, ack 1, win 1026, length 19
19:30:08.344270 IP 192.x.1.x.61639 > 192.x.2.x.ms-wbt-server: Flags [P .], seq 1:20, ack 1, win 1026, length 19
19:30:08.344436 IP 192.x.GWWEST.x.34545 > 192.x.2.x.ms-wbt-server: Flags [P. ], seq 1:20, ack 1, win 1026, length 19
19:30:08.943030 IP 192.x.2.x.ms-wbt-server > 192.x.GWWEST.x.34545: Flags [S. ], seq 2293385564, ack 2636769635, win 64000, options [mss 1376,nop,wscale 0,nop ,nop,sackOK], length 0
19:30:08.943133 IP 192.x.2.x.ms-wbt-server > 192.x.1.x.61639: Flags [S .], seq 2293385564, ack 2636769635, win 64000, options [mss 1376,nop,wscale 0,no p,nop,sackOK], length 0
19:30:08.944038 IP 192.x.1.x.61639 > 192.x.2.x.ms-wbt-server: Flags [. ], ack 1, win 1026, options [nop,nop,sack 1 {0:1}], length 0
19:30:08.944170 IP 192.x.GWWEST.x.34545 > 192.x.2.x.ms-wbt-server: Flags [.] , ack 1, win 1026, options [nop,nop,sack 1 {0:1}], length 0
19:30:10.745440 IP 192.x.1.x.61639 > 192.x.2.x.ms-wbt-server: Flags [P .], seq 1:20, ack 1, win 1026, length 19
19:30:10.745581 IP 192.x.GWWEST.x.34545 > 192.x.2.x.ms-wbt-server: Flags [P. ], seq 1:20, ack 1, win 1026, length 19
19:30:14.942913 IP 192.x.2.x.ms-wbt-server > 192.x.GWWEST.x.34545: Flags [S. ], seq 2293385564, ack 2636769635, win 64000, options [mss 1376,nop,nop,sackOK], length 0
19:30:14.942975 IP 192.x.2.x.ms-wbt-server > 192.x.1.x.61639: Flags [S .], seq 2293385564, ack 2636769635, win 64000, options [mss 1376,nop,nop,sackOK] , length 0
19:30:14.943823 IP 192.x.1.x.61639 > 192.x.2.x.ms-wbt-server: Flags [. ], ack 1, win 1026, options [nop,nop,sack 1 {0:1}], length 0
19:30:14.943909 IP 192.x.GWWEST.x.34545 > 192.x.2.x.ms-wbt-server: Flags [.] , ack 1, win 1026, options [nop,nop,sack 1 {0:1}], length 0
19:30:15.546861 IP 192.x.1.x.61639 > 192.x.2.x.ms-wbt-server: Flags [P .], seq 1:20, ack 1, win 1026, length 19
19:30:15.547066 IP 192.x.GWWEST.x.34545 > 192.x.2.x.ms-wbt-server: Flags [P. ], seq 1:20, ack 1, win 1026, length 19
19:30:25.147228 IP 192.x.1.x.61639 > 192.x.2.x.ms-wbt-server: Flags [R .], seq 20, ack 1, win 0, length 0
19:30:25.147314 IP 192.x.GWWEST.x.34545 > 192.x.2.x.ms-wbt-server: Flags [R. ], seq 20, ack 1, win 0, length 0
19:30:26.943294 IP 192.x.2.x.ms-wbt-server > 192.x.GWWEST.x.34545: Flags [R] , seq 2293385565, win 0, length 0
19:30:26.943395 IP 192.x.2.x.ms-wbt-server > 192.x.1.x.61639: Flags [R ], seq 2293385565, win 0, length 0

[PhoneBoy]

Why am I seeing both the internal IP and the firewall IP here?
I assume this traffic is going in/out the same interface?
In any case, the issue appears to be application-specific since traffic clearly gets to the remote server and a reset happens on the other end.
What is the application-specific error message?
Anything in the firewall logs or maybe with fw ctl zdebug drop?

[Ping_Choi]

The reason you can see the internal ip and the FW ip is because I did a tcpdump to check out if the traffic from teh internal ip was reaching the firewall, which it was.

That confirmed that the internal ip is able to reach the firewall.

Here is a logfile for one of those sessions. It is able to forward the traffic-

Id: 14bdb397-0000-00c0-5f14-9f3d00000001
Marker: @A@@B@1595131202@C@2990576
Log Server Origin: 192.x.1.x
Time: 2020-07-19T19:30:05Z
Interface Direction: inbound
Interface Name: eth1
Id Generated By Indexer: false
First: true
Sequencenum: 7
Source Zone: Internal
Destination Zone: External
Service ID: Remote_Desktop_Protocol
Source: 192.x.0.x
Source Port: 61639
Destination: 192.x.1.x
Destination Port: 3389
IP Protocol: 6
Xlate (NAT) Source IP: 192.x.GW.x
Xlate (NAT) Source Port: 34545
Xlate (NAT) Destination Port:0
NAT Rule Number: 70
NAT Additional Rule Number: 0
Action: Accept
Type: Connection
Policy Name: Azure-VMSS-Policy
Policy Management: MGMT
Db Tag: {EEF6AEEE-2D29-E241-AFBF-647EB766567D}
Policy Date: 2020-07-17T19:33:37Z
Blade: Firewall
Origin: Azure-Controller--WESTUS
Service: TCP/3389
Product Family: Access
Logid: 0
Access Rule Name: On-prem to azure
Access Rule Number: 11
Policy Rule UID: aa81823f-ee39-4df6-94cc-271104fdcfea
Layer Name: Azure-VMSS-Policy Network
Interface: eth1
Description: Remote_Desktop_Protocol Traffic Accepted from West machine ip to East machine ip

[PhoneBoy]

Not sure why some.of your responses were getting flagged as spam. 😬

Right now, everything is pointing at the application closing the connection (I.e. A TCP reset).
What is providing the RDP and what information do you see in the client/server logs about the connection attempt?

[Ping_Choi]

These are Windows machines that offer out the RDP service.

The only thing in the Event logs that is obvious, is the fact that we are having an ongoing issue with the domain controllers on both sites that are not able to reach each other. This has been an ongoing issue and that is the only event that shows up in the server logs.

[PredragPetrovic · Accepted solution]

Hi,

There are multiple ways to deal with this issue. So here we go:

- Scenario 1 (recommended):

Do VNET peering between the VNET's hosting your checkpoint firewalls (mostly frontend and backend subnet). I do not recommend using one big VNET and hosting your VMs and Check Points and everything due to the VNET peering items that come with it, especially when using Express Route and subnet announcements over BGP. Let's call these networks RegionA and RegionB.

So on the Check Point in RegionA set a static route for the master subnet in the RegionB to point to the eth1 interface default gateway (Azure gateway, you get the Azure gateway on every subnet).

On the Check Point in RegionB set a static route for the master subnet in RegionA to point to the eth1 interface default gateway (Azure gateway).

Create separate route table for the backend subnets in both regions. Create a specific subnet route in RegionA route table for the RegionB master network range towards the back-end load balancer IP of the check point in regionB.

Create a specific subnet route in RegionB route table for the RegionA master network range towards the back-end load balancer of the check point in regionA.

Create the necessary firewall rules to allow communication and push the policy.

- Scenario 2 (not recommended):

Setup IPSEC VPN between RegionA and RegionB Check Points using the Internet.

Setup VTI's and configure dynamic routing protocols.

Hope this helps,

Predrag

[PredragPetrovic]

Hi,

I've attached the drawing here. Azure router is something that is native in every subnet and its the first IP address that's available in the range.

[Ping_Choi]

Thank you so much! It turned out that the application we wanted to send traffic over to the other Azure region does not support VNET peering. 

However, we worked with Azure networking and they suggested we "split" or "pipe" our Expressroute traffic so that the traffic can go to both Azure regions.

The solution was called to use Azure VNET transit gateway.