CP to Azure S2S vpn issue

[the_rock]

Hey guys,

I hope someone might be able to shed some light into this situation, as I find it very peculiar. So, customer has domain based vpn between cp and azure and tunnel works fine, BUT, here is the issue. So, azure subnet is 10.18.0.0/16 and there is one host in that subnet that no matter what we do, logs show its going through the tunnel, though random one shows it being dropped or going out clear (randomly), but the page to access it never does come up, like it should. 

All the other hosts/services work fine.

Now, customer did have Azure case, they did bunch of checks and determined its not the problem on their end. I, together with the customer, did bunch of captures, checked the logs, we even added that host IP into enc domain, reset the tunnel, set tunnel management per gateway as a test, no dice.

I dont sadly have the actual log at the moment (can get it from the client), but captures when we run them show traffic comes to internal interface and thats it, nothing else, which is super odd, because say host 10.18.0.80 or .85 are fine, but .81 never works. Now, I know logically it would indicate issue with the host, but MS support verified 100% that is not the case.

I had client do basic vpn debugs on cp side, will review them myself, but just wondering if anyone may have any insight/suggestions we could try. I cant possible think of anything else myself that we had not tested.

Thanks as always.

Andy

[G_W_Albrecht]

Did you contact CP TAC yet ?

[the_rock]

Not yet, as I want to review vpn debugs myself first.

Andy

[the_rock]

Just to update on this...customer will try change the IP of the problematic host to see if that helps, but if not, they will send me the vpn debugs and will review. Honestly, Im not sure this even really qualifies for TAC case, though logs clearly show when issue is there that traffic does NOT go through VPN tunnel and Azure support is adamant its not problem on their end.

Anywho, lets see what gives 🙂

Andy

[Timothy_Hall]

Do you have "disable NAT in VPN community" set?  Almost sounds like you have a NAT of some kind just for that .81 address which would allow the traffic to enter the tunnel but then get dropped on the other end.  If the destination IP is getting NATted that could be why the traffic seems to disappear in your capture after the inbound.

I assume there is no Windows Firewall on .81?  If the traffic can be verified to be entering the tunnel properly on your side, you may need a packet capture on the .81 host to confirm the traffic is actually getting there.  Had many a troubleshooting session where the traffic is going into the tunnel properly and the other end insists it is decrypting and reaching the endpoint on their side...but it isn't due to a VPN config/policy or routing issue.  Until you do that packet capture they will just blame you 🙂

[the_rock]

Yep, we do have nat disabled. I even had them create manual no nat rule for that IP, no luck. Funny enough, when we do captures, randomly it shows going through the tunnel, but even then page never comes up.

Let me see if their dedicated Azure expert can change the ip of that host and see what happens.

Andy

[the_rock]

Btw, just confirmed, no windows firewall. Let me review the logs and see what we can find.

Andy

[the_rock]

Just spoke with customer. They decided to install jumbo 89 on their mgmt and cluster, so will let me know this weekend if that changes anything. I secretly hope it fixes the issue, but lets see.

If not, they will open TAC case next week and will provide an update.

Thanks!

Andy

[AmirArama]

Hello,

this kind of issues needs cooperation from both sides.

from your (CP) side, you can only run:
vpn tu conn & fw monitor on the connection 5 tupple, and tcpdump on the ESP/NAT-T packets, but someone needs to run traffic capture on Azure side to see if the traffic reached the other sides, and if so, what it happening with it? does it reach the host? (if you manage this host, you can install wireshark over there and see for yourself), does the host respond or not?

[the_rock]

I totally agree. Let me follow up with customer to see if jumbo 89 made any difference.

Andy