Robert_Ellis1
Contributor

Unable to connect to Azure Private Endpoint from on-premise via Cloudguard

I wonder if experts here may be able to shed any additional light on this deeply frustrating issue.

 

We have a Cloudguard cluster in our Azure tenancy. There is a domain VPN tunnel back to our on-premise Checkpoint cluster. 

The Cloudguard cluster (R80.40) is completely up-to-date, including all Takes and patches; the on-prem cluster is R80.30 (3 pending patches.)

We have been struggling to connect an on-premise server, through the tunnel, to an Azure SQL Private Endpoint network interface. Originally, we were routing traffic all the way (no NAT in the picture) but this was getting us nowhere; so now we have tried with NAT.

In the current configuration:

An on-premise network: 192.168.50.0/24. The source host is 192.168.55.251.

The cloudguard backend network subnet is 10.10.0.0/24. The backend IP address of the Cloudguard cluster members are 10.10.0.133 and 10.10.0.134. The front-end IP of the backend LB is 10.10.0.132.

I have a NAT rule on the Cloudguard which Hides the source connection behind the address range 10.10.0.200-10.10.0.204.

To take Azure network security issues out of the equation, I have placed the SQL Private Endpoint directly into the backend network with IP address 10.10.0.135. 

In the Checkpoint logs, I can see Encryption success on the on-premise cluster. And, I can see Decryption success on the Cloudguard cluster. In the same log page, I see NAT Hide working as expected:  destination IP address (10.10.0.135) remains intact and source IP address has been translated from 192.168.50.251 to 10.10.0.201.

I have added ARP entries to the Cloudguard cluster (both members) as described by sk30197; for example:  "add arp proxy ipv4-address 10.10.0.200 interface eth1 real-ipv4-address 10.10.0.133" etc.

Since Private Endpoint essentially eliminates Azure security as the culprit, let's assume this is not an Azure security issue. Does anybody have any idea where I should look next?

The only (untested) theory I have left of my own at this stage is that Azure Private Endpoint will not listen to packets carrying source IP addresses other than IP addresses which have been associated directly with VM virtual interfaces in Azure. (i.e. under this theory, ARP responses from a vNIC aren't enough - the IP address has to be actually configured and defined in Azure if the PE is to listen).  To me, this theory seems so unlikely to prove true and I feel like I'm rather clutching at straws, but equally, I have no other ideas. 

Of course, even if the above theory were correct, it would not lead to a solution, since the source addresses could only be added to either one member of the Cloudguard cluster or the other, not both; so that everything will break whenever a failover occurs in the wrong direction.

I have tried adding routing rules in the Azure backend subnet, to ensure that packets destined for any address in the Hide source range (10.10.0.200-10.10.0.204) are routed directly to the backend LB. I'm not sure these rules are needed - I suspect not - but nothing works either way. As mentioned above, a similar configuration without NAT also does not work, even with Azure routing rules in place. 

We will be raising a support ticket soon with our Support Partner, but any ideas as to where I might be able to look next would be gratefully received. 

Many thanks in advance.

 

 

 

 

0 Kudos
9 Replies
the_rock
Leader
Leader

I really wish I could give you good advice, but not really Azure guru, sorry : (. But, on firewall side, if you do regular tcpdumps, fw monitor, zdebug etc, ip route get command, do you see expected behavior?

0 Kudos
Robert_Ellis1
Contributor

FW monitor shows packets leaving the VM interface of the firewall, but nothing come back the other way. This is almost unbelievable: everything is on the one single subnet - no routing involved. Yet the PE seemingly won't reply, or the Checkpoint can't see the packets. It is bizarre.

 


[vs_0][ppak_0] eth0:iD[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35983
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][ppak_0] eth0:i[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35983
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][fw_0] eth0:i[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35983
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][ppak_0] eth0:i[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35983
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][fw_0] eth0:I[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35983
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][fw_0] eth1:o[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35983
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][fw_0] eth1:O[44]: 10.10.0.201 -> 10.10.0.135 (TCP) len=52 id=35983
TCP: 10417 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][ppak_0] eth0:iD[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35984
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][ppak_0] eth0:i[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35984
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][fw_0] eth0:I[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35984
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][fw_0] eth1:o[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35984
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][fw_0] eth1:O[44]: 10.10.0.201 -> 10.10.0.135 (TCP) len=52 id=35984
TCP: 10417 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][ppak_0] eth0:iD[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35985
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][ppak_0] eth0:i[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35985
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][fw_0] eth0:I[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35985
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][fw_0] eth1:o[44]: 192.168.50.251 -> 10.10.0.135 (TCP) len=52 id=35985
TCP: 49588 -> 1433 .S.... seq=200d5fcf ack=00000000
[vs_0][fw_0] eth1:O[44]: 10.10.0.201 -> 10.10.0.135 (TCP) len=52 id=35985
TCP: 10417 -> 1433 .S.... seq=200d5fcf ack=00000000

0 Kudos
Dmitry_Gorn
Employee
Employee

Hi Robert,

Please note that when working in the Azure platform there is no need to modify ARP, as it's handled by the Azure platform.

If you have opened a support request, please share the number so I can confirm a relevant support engineer is assisting.

The static routes on the cluster members should be configured so that traffic into the private link is sent via eth1. The routing table for the reply packet should use the internal load balancer frontend IP as next hop.

If I understand you use case correctly, there is no need to hide-NAT the packet, since the reply will be routed via the internal load balancer. Only the Active cluster member replies to load balancer health checks.

 

Thanks,

Dmitry

0 Kudos
Robert_Ellis1
Contributor

there is no need to hide-NAT the packet, since the reply will be routed via the internal load balancer. 

I think you've understood correctly. Originally I wouldn't have though we needed to NAT. We have tried it in desperation really. Clutching at straws.

The static routes on the cluster members should be configured so that traffic into the private link is sent via eth1. The routing table for the reply packet should use the internal load balancer frontend IP as next hop.

This is exactly the set up we have. Packets go out eth1, but nothing comes back the other way, no matter what.

I have just now raised a ticket with our Partner, and will try to update here as to any progress.

thanks

 

 

 

 

0 Kudos
Robert_Ellis1
Contributor

Private-endpoint.png

0 Kudos
Robert_Ellis1
Contributor

Just posting this image to underline how weird this situation is.

As you can see, absolutely no packets are reaching the private endpoints (first 2 rows) - and this is with the private endpoint in the same IP subnet as the Checkpoint backend interfaces. Just inexplicable.

0 Kudos
Robert_Ellis1
Contributor

hello

Our situation with the Cloudguard has worsened, and although a ticket is underway, and our partner has made a submission to Checkpoint, I need to try to problem solve in order to get things done and deliver on my deadlines.

The current summary is essentially:

1. We have a cloudguard clsuter, deployed in to Azure based on a template

2. There are two VMs in the cluster, and there is a front-end and back-end load balancer

3. I have verified that the active member of the cluster is VM #1

4. NSG at the front-end is open, as you would expect (traffic flows freely inbound)

5. Both VMs private front-end interfaces are members of the front-end LB's backend pool

So essentially everything is as you would expect with a vanilla deployment

However, traffic targeting the public IP address of the front-end LB simply is not reaching the Cluster members. I have been testing on port 8080, where we want to NAT Internet client connections to private resources behind the Checkpoint. Using fw monitor, I have verified that these packets never arrive on either member of the cluster.

Our support Partner says Checkpoint are investigating a fix where the Cluster VIP does not follow to the correct member when a failover occurs, but I don't see how this explains the symptoms I am describing, because the Cluster VIP is currently associated with the active member of the cluster

If anybody has any advice I would be grateful. <rant> I was rather a Checkpoint fan boy, but now find myself increasingly dismayed by how difficult it is for us to accomplish simple tasks. And we are paying a lot of money for something that doesn't seem to be fit for purpose at the moment </rant>

 

thanks

 

0 Kudos
Matthias_Haas
Advisor

Robert,

not sure if this helps, but  I would use loadbalancing rules instead of NAT rules on the external LB as decribed here: 

Additional-External-IP-azure 

if the NSG is allowing the traffic you should see the packet on the FW.

But it should work also with NAT rules of course, with LB rules you would not need the double port NAT (on the LB and the FW).

If that does not help, make sure the health check by the LB is answered by the firewall 

0 Kudos
Prabulingam_N1
Advisor

Hello Robert,

Try the attached doc setting on FrontLB Loadbalancing Rule and SmartConsole NAT rule

By accessing FrontLB PIP you can each internal VMs Private IP.

 

Regards, Prabu

0 Kudos