Advice needed for Site to Site VPN Link Selection settings (unusual setup, two ISP lines)

[Kryten]

Hi all,

I need some advice for an unusual setup regarding two ISP connections and Site to Site VPN Link Selection settings( or possibly other settings as well).

So far I learned that it is no problem to have an ISP connection over an interface with a private IP (public network gets routed to this IP from the next Hop). We create a dummy cluster interface with the public IP we want to use and set LinkSelection to use this IP.

I also learned that it should work having two ISP connections and set Link Selection to use the IP of the outgoing(initiating) or incoming(responding) interface. This we have not tested so far, but in Theory it should work, right?

Now we come to the problem: Our customer has a mix of these. The first ISP connection is an interface with a public IP on it where all traffic goes out. Now we got a second ISP with more bandwidth, to where we want to move the Site2Site VPNs in the future. This is connected to the gateways on an interface with a private IP.

We need to be able to have Site to Site VPNs on both interfaces, as we cannot move all at once and also there will be 3rd party Gateways(so no link probing possible). And further we also need to be sure about the Link selection settings before changing them, as we do not want to affect existing tunnels.

The Link Selection Settings we use now:

- first option for locally managed VPN peers is set to MainIP(public IP of first ISP)
- outgoing route selection is set to "reply from same interface"
- source IP settings are set to automatic

We then proceeded to set up a tunnel for testing, which on our end we configured with the dummy interfaces public IP address as the remote peer. On the customers end we set up the tunnel in the usual way and configured a static route for the remote network to point to the interface for the second ISP, so the traffic would go out that way and not per default route over the first ISP link.

The Tunnel itself builds up just fine, Phase1 and Phase2 are completed successfully and in a tcpdump we see the correct IP addresses talking to each other. When we send some ICMP requests, we see those requests arriving at the customers gateway, they get decrypted and the destination responds to these requests. But then we see that those ESP pakets leave the Gateway on the correct interface, but sadly with the private IP of this interface and not the public one from the dummy interface. Of course they never reach the other end.

Why is it that the IKE stuff on port 500 is handled with the correct IP, but the ESP traffic on port 4500 is not?

Is there anything we can do to make this work?

Any insight into this is much appreciated, as this has given me some headaches already for some time now.

Cheers,

Alex

[Wolfgang]

@Kryten we had a similar use case and can tell you we didn‘t find a solution. The problem will be the private IPs on the secondary link. You have to configure the outgoing source address to be your public IP from the secondary ISP. But this can be done only if you set the source address to „always use this address“. With these setting always your public IP from secondary ISP will be used on both links. If you set the source to „automatic“ the IP of your outgoing interface will be used. This is the private IP from your secondary ISP.

In our case the first ISP is a Connection similar yours with a private IP on the gateway. We configured a dummy interface with the public IP. The secondary link is a private encapsulate network from another ISP between our locations (it‘s like a routed MPLS line). We spent a lot of time but we did not get a working solution with link selection or others. Always one of the connections used the wrong source IP. Our solution was to set link selection source to „always use this IP“ with the public IP. Additional our secondary ISP configured host routes for the public IP of our first ISP in there network. But this works only because the secondary lines are encapsulated exclusive for us. 

With two ISPs and if you want using both links for VPN connections you have to use real public IPs on both ISP links directly connected on your gateway. Maybe another one knows a solution but I think there is now way to get this working.

[Kryten]

Thanks a lot for sharing that experience Wolfgang!
Its not what I hoped for, but better than continuing to get headaches 😄 I think we can manage to get public IPs on that end too, its just a lot more effort than originally planned.

I just wonder now if one public IP as virtual cluster IP on that interface would be enough, or if we need public IPs for the two physical interfaces as well?

Cheers,
Alex

[Wolfgang]

@Kryten if you can get three public IPs I would go with these. There will be some impact with only private IPs for the physical nodes.

[Kryten]

We now have this configured with three public IP addresses, but somehow it seems we do not have the correct Link Selection Settings for this.
Our Cluster now still responds through the correct Interface of the second ISP, but not with this interfaces IP address. It responds with the Cluster Main Address now(IP of the first ISP link).

EDIT: Never mind, I just checked the configuration and the NAT was not disabled in this community...that is pretty sure the reason for this 🙂

[Wolfgang]

@Kryten setting of the source IP address has to be configured here: