CP to Azure S2S vpn issue

[the_rock]

Hey guys,

I hope someone might be able to shed some light into this situation, as I find it very peculiar. So, customer has domain based vpn between cp and azure and tunnel works fine, BUT, here is the issue. So, azure subnet is 10.18.0.0/16 and there is one host in that subnet that no matter what we do, logs show its going through the tunnel, though random one shows it being dropped or going out clear (randomly), but the page to access it never does come up, like it should. 

All the other hosts/services work fine.

Now, customer did have Azure case, they did bunch of checks and determined its not the problem on their end. I, together with the customer, did bunch of captures, checked the logs, we even added that host IP into enc domain, reset the tunnel, set tunnel management per gateway as a test, no dice.

I dont sadly have the actual log at the moment (can get it from the client), but captures when we run them show traffic comes to internal interface and thats it, nothing else, which is super odd, because say host 10.18.0.80 or .85 are fine, but .81 never works. Now, I know logically it would indicate issue with the host, but MS support verified 100% that is not the case.

I had client do basic vpn debugs on cp side, will review them myself, but just wondering if anyone may have any insight/suggestions we could try. I cant possible think of anything else myself that we had not tested.

Thanks as always.

Andy

[G_W_Albrecht]

Did you contact CP TAC yet ?

[the_rock]

Not yet, as I want to review vpn debugs myself first.

Andy

[the_rock]

Just to update on this...customer will try change the IP of the problematic host to see if that helps, but if not, they will send me the vpn debugs and will review. Honestly, Im not sure this even really qualifies for TAC case, though logs clearly show when issue is there that traffic does NOT go through VPN tunnel and Azure support is adamant its not problem on their end.

Anywho, lets see what gives 🙂

Andy

[Timothy_Hall]

Do you have "disable NAT in VPN community" set?  Almost sounds like you have a NAT of some kind just for that .81 address which would allow the traffic to enter the tunnel but then get dropped on the other end.  If the destination IP is getting NATted that could be why the traffic seems to disappear in your capture after the inbound.

I assume there is no Windows Firewall on .81?  If the traffic can be verified to be entering the tunnel properly on your side, you may need a packet capture on the .81 host to confirm the traffic is actually getting there.  Had many a troubleshooting session where the traffic is going into the tunnel properly and the other end insists it is decrypting and reaching the endpoint on their side...but it isn't due to a VPN config/policy or routing issue.  Until you do that packet capture they will just blame you 🙂

[the_rock]

Yep, we do have nat disabled. I even had them create manual no nat rule for that IP, no luck. Funny enough, when we do captures, randomly it shows going through the tunnel, but even then page never comes up.

Let me see if their dedicated Azure expert can change the ip of that host and see what happens.

Andy

[the_rock]

Btw, just confirmed, no windows firewall. Let me review the logs and see what we can find.

Andy

[the_rock]

Just spoke with customer. They decided to install jumbo 89 on their mgmt and cluster, so will let me know this weekend if that changes anything. I secretly hope it fixes the issue, but lets see.

If not, they will open TAC case next week and will provide an update.

Thanks!

Andy

[AmirArama]

Hello,

this kind of issues needs cooperation from both sides.

from your (CP) side, you can only run:
vpn tu conn & fw monitor on the connection 5 tupple, and tcpdump on the ESP/NAT-T packets, but someone needs to run traffic capture on Azure side to see if the traffic reached the other sides, and if so, what it happening with it? does it reach the host? (if you manage this host, you can install wireshark over there and see for yourself), does the host respond or not?

[the_rock]

I totally agree. Let me follow up with customer to see if jumbo 89 made any difference.

Andy

[the_rock]

Hey guys,

Just to give quick update on this, spoke with customer, they tried changing the IP on Azure host side, no luck. They wull install jumbo 89 this Saturday on the cluster and test. I would personally be shocked if that fixes anything, but lets hope for the best 🙂

Anyway, if no change, they will open TAC case.

Andy

[Lesley]

"but captures when we run them show traffic comes to internal interface and thats it, nothing else, which is super odd, because say host 10.18.0.80 or .85 are fine, but .81 never works."

if this capture is tcpdump it makes sense because you see the data incoming on LAN interface unecrypted. Then it would be send out on WAN interface and you will see ESP traffic between the 2 public IP's. 

For better understanding we need vpn debug while traffic is send towards the problem host. I assume you see on your side on the check point drops and unencrypted packets? Or you always see encrypted data in the logs towards the host? If you do not seen encrypted log entries it could be an indication it is an issue on your side. 

Are you using global encryption domain for tunnel? or you set it up on the community itself (would recommend this)

[the_rock]

Its actually set per community, not global. As far as packets, you see encrypted ones most of the time and then unencrypted say 5-10% of the time (randomly) and shows dropped because of ssl inspection, which I find very peculiar. 

Anyway, let me see if jumbo install makes any difference, but if not, then we may need to do some basic vpn debugs next time they can dedicate some time to this.

Thanks Lesley.

Andy

[the_rock]

Just to give an update...client installed jumbo 90, but same issue, which Im really not surprised about. Anywho, they will open TAC case next week to check this further.

I will update when we have more info.

Best,

Andy

[Lesley]

Any debugs on the way? Ike viewer would help here a lot 

[the_rock]

Not yet sadly. These guys are super busy, so may take awhile :- (

[Duane_Toler]

I worked with a customer today on a new Azure VPN setup.  Had some issues, too, band the customer had the Windows firewall policy on the Azure VM not allowing some traffic in and out.  They did have some NSGs in place, too, which had to be adjusted.

Another item was missing a subnet on the Azure side VPN “local network gateway”.  This may not be your issue tho.

I also had their vpn community tunnel management set to “one tunnel per subnet pair” rather than universal tunnels. I wasn’t sure how they had their Azure side configured.

I hope some of this helps. Good luck with it!

[the_rock]

Hey Duane, tx for responding man, always nice to hear from you! Yea, we tested all those things you mentioned, tried different tunnel mgmt options, no dice. We know setup is right, as its ONLY this one host with the issue, but based on all Azure support did, they told the customer to look elsewhere.

I know they will probably open TAC case, but you know how it goes, lots of IT issues and only few guys to deal with them, so they need to work on more pressing problems, specially before holidays.

I know their IT boss will text me, as he always does, to ask for my help on this, though I feel he does it lately to get a good travel destination advice from me 🤣🤣

Anywho, I will certainly update once I have more info.

Thanks as always again!

Andy