Troubles with dropped connections (hide nat)

[Graham]

We have a partner MPLS connection that I am having issues with. Since they are quite a bit larger in size convincing them the issues lies with them is a challenge.

Interestingly they have decided to use public IPs on a dedicated internal network; Again I have no control in that.
There is a shared network space 64.164.0.96/28 used for connectivity. Their requirements dictate that all our connections should originate from 64.164.0.103.  I have setup hide NAT, proxy arp, static routes and disabled address spoofing on outgoing the interface to achieve this.

All resources we need access to are in 64.128.0.0/24, 64.131.0.108/30 or 64.144.0.144/30, hence the reason for static routes.
For testing I have ANY/ANY rules to eliminate ACLs as an issue.  I am successfully able to connect to basic web pages, but anything that requires token based authentication fails.  Working with TAC we have discovered that the connection dies at phase 2, however I am not sure where to take from there.

fw ctl zdebug + drop results in no drops when we grep based on distention IP.
tcpdump shows network flow, but at some point the connection drops.

My only idea at this point to overide the interfaces topology, but I am not sure which would apply.
I am thinking I might set it to undefined since the routes are static and selecting DMZ network as well.

I would welcome any idea and I willing to try anything at this point.

[amdhim0004]

Hi @Graham 

One question.

Your interface IP and your NAT IP towards your destination are different right?

[Graham]

Yes correct the interface IP is .98 and the NAT IP is .103.

[amdhim0004]

Hi @Graham 

Thanks for the confirmation. Can you please do a tcpdump on source, while trying to access the resources.

++ Try below command and you will find out if your traffic is getting NAT or not++

tcpdump -Peni any host (Source IP)

[Timothy_Hall]

So basic connectivity with ping/ICMP and http web traffic works?  How about https traffic to the partner's websites, and do you have HTTPS/TLS inspection enabled on your firewall?

Can you be more specific about what "token-based authentication" exactly is?  It sounds like that specific traffic is not getting NATTed to the proper address, thus causing Phase 2 to fail when your firewall tries to negotiate a new IPSec tunnel for that traffic with the original or otherwise wrong source address/subnet.  That or there is some kind of routing issue on your end or theirs involving this specific traffic; either of these situations would of course not show any drops occurring.

[Graham1]

For some reason CheckMates woould allow me to post an update, so trying under a different name.

@Timothy_Hall 
Yes HTTPS traffic functions as expected.  HTTPS inspection is not enabled, mostly since I haven't been able to get it to work/no time.
By token based I mean an Entrust certificate stored on USB hardware that authenticates to Cisco AnyConnect server, sorry that is as much of that end that I know.

Would there be any value in testing network topology overrides on the Gateway object?

@amdhim0004 
[Expert@***:0]# tcpdump -Penni any host 64.128.0.143
tcpdump: WARNING: Promiscuous mode not supported on the "any" device
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
14:26:57.708981 eth4[in ]: 10.1.0.245.64023 > 64.128.0.143.500: isakmp: phase 1 I #34[]
14:26:57.711276 eth7[out]: arp who-has 64.128.0.143 tell 64.164.0.98
14:26:57.711757 eth7[in ]: arp reply 64.128.0.143 is-at cc:46:d6:d8:25:1e
14:26:57.711770 eth7[out]: 64.164.0.103.43338 > 64.128.0.143.500: isakmp: phase 1 ? #34[]
14:26:57.770970 eth7[in ]: 64.128.0.143.500 > 64.164.0.103.43338: isakmp: phase 1 R #34[]
14:26:57.771243 eth4[out]: 64.128.0.143.500 > 10.1.0.245.64023: isakmp: phase 1 R #34[]
14:26:57.779685 eth4[in ]: 10.1.0.245.64023 > 64.128.0.143.500: isakmp: phase 1 I #34[]
14:26:57.780063 eth7[out]: 64.164.0.103.43338 > 64.128.0.143.500: isakmp: phase 1 ? #34[]
14:26:57.869412 eth7[in ]: 64.128.0.143.500 > 64.164.0.103.43338: isakmp: phase 1 R #34[]
14:26:57.869729 eth4[out]: 64.128.0.143.500 > 10.1.0.245.64023: isakmp: phase 1 R #34[]
14:26:57.902029 eth4[in ]: 10.1.0.245.64024 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others I #35[]
14:26:57.902129 eth4[in ]: 10.1.0.245.64024 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others I #35[]
14:26:57.903039 eth7[out]: 64.164.0.103.26672 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others ? #35[]
14:26:57.903107 eth7[out]: 64.164.0.103.26672 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others ? #35[]
14:26:58.051808 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.052060 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.052306 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.052370 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.052823 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.052881 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.053309 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.053366 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.054052 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.054100 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.054556 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.054598 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.062050 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.062085 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.903695 eth4[in ]: 10.1.0.245.64024 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others I #35[]
14:26:59.903749 eth4[in ]: 10.1.0.245.64024 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others I #35[]
14:26:59.903789 eth7[out]: 64.164.0.103.26672 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others ? #35[]
14:26:59.903820 eth7[out]: 64.164.0.103.26672 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others ? #35[]
14:26:59.961990 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.962049 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.962489 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.962530 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.962988 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.963032 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.963489 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.963524 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.963987 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.964023 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.964735 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.964766 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.972235 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:59.972271 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.904825 eth4[in ]: 10.1.0.245.64024 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others I #35[]
14:27:03.904923 eth7[out]: 64.164.0.103.26672 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others ? #35[]
14:27:03.905025 eth4[in ]: 10.1.0.245.64024 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others I #35[]
14:27:03.905085 eth7[out]: 64.164.0.103.26672 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others ? #35[]
14:27:03.963884 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.963951 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.964381 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.964438 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.965129 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.965164 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.965633 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.965673 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.966131 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.966166 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.966631 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.966661 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.974378 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:03.974422 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.904654 eth4[in ]: 10.1.0.245.64024 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others I #35[]
14:27:11.904704 eth4[in ]: 10.1.0.245.64024 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others I #35[]
14:27:11.904738 eth7[out]: 64.164.0.103.26672 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others ? #35[]
14:27:11.904767 eth7[out]: 64.164.0.103.26672 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others ? #35[]
14:27:11.964202 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.964272 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.964672 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.964713 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.965252 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.965290 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.965802 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.965836 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.966373 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.966412 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.966954 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.966985 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.974465 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:11.974498 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]
14:27:17.922435 eth4[in ]: 10.1.0.245.52704 > 64.128.0.143.443: S 299105415:299105415(0) win 64240 <mss 1460,nop,wscale 8,nop,nop,sackOK>
14:27:17.923100 eth7[out]: 64.164.0.103.44131 > 64.128.0.143.443: S 299105415:299105415(0) win 64240 <mss 1460,nop,wscale 8,nop,nop,sackOK>
14:27:17.979036 eth7[in ]: 64.128.0.143.443 > 64.164.0.103.44131: S 588342429:588342429(0) ack 299105416 win 3900 <mss 1300>
14:27:17.979052 eth4[out]: 64.128.0.143.443 > 10.1.0.245.52704: S 588342429:588342429(0) ack 299105416 win 3900 <mss 1300>
14:27:17.979344 eth4[in ]: 10.1.0.245.52704 > 64.128.0.143.443: . ack 1 win 64240
14:27:17.979358 eth7[out]: 64.164.0.103.44131 > 64.128.0.143.443: . ack 1 win 64240
14:27:17.979563 eth4[in ]: 10.1.0.245.52704 > 64.128.0.143.443: P 1:209(208) ack 1 win 64240
14:27:17.979838 eth7[out]: 64.164.0.103.44131 > 64.128.0.143.443: P 1:209(208) ack 1 win 64240
14:27:18.035488 eth7[in ]: 64.128.0.143.443 > 64.164.0.103.44131: . ack 209 win 4108
14:27:18.035518 eth4[out]: 64.128.0.143.443 > 10.1.0.245.52704: . ack 209 win 4108
14:27:18.110944 eth7[in ]: 64.128.0.143.443 > 64.164.0.103.44131: P 1:1301(1300) ack 209 win 4108
14:27:18.111255 eth4[out]: 64.128.0.143.443 > 10.1.0.245.52704: P 1:1301(1300) ack 209 win 4108
14:27:18.112195 eth7[in ]: 64.128.0.143.443 > 64.164.0.103.44131: . 1301:2601(1300) ack 209 win 4108
14:27:18.112228 eth4[out]: 64.128.0.143.443 > 10.1.0.245.52704: . 1301:2601(1300) ack 209 win 4108
14:27:18.112551 eth4[in ]: 10.1.0.245.52704 > 64.128.0.143.443: . ack 2601 win 65000
14:27:18.112566 eth7[out]: 64.164.0.103.44131 > 64.128.0.143.443: . ack 2601 win 65000
14:27:18.113074 eth7[in ]: 64.128.0.143.443 > 64.164.0.103.44131: P 2601:3652(1051) ack 209 win 4108
14:27:18.113472 eth4[out]: 64.128.0.143.443 > 10.1.0.245.52704: P 2601:3652(1051) ack 209 win 4108
14:27:18.121801 eth4[in ]: 10.1.0.245.52704 > 64.128.0.143.443: P 209:407(198) ack 3652 win 63949
14:27:18.121975 eth7[out]: 64.164.0.103.44131 > 64.128.0.143.443: P 209:407(198) ack 3652 win 63949
14:27:18.177693 eth7[in ]: 64.128.0.143.443 > 64.164.0.103.44131: . ack 407 win 4306
14:27:18.177711 eth4[out]: 64.128.0.143.443 > 10.1.0.245.52704: . ack 407 win 4306
14:27:18.179691 eth7[in ]: 64.128.0.143.443 > 64.164.0.103.44131: P 3652:3711(59) ack 407 win 4306
14:27:18.179749 eth4[out]: 64.128.0.143.443 > 10.1.0.245.52704: P 3652:3711(59) ack 407 win 4306
14:27:18.180318 eth4[in ]: 10.1.0.245.52704 > 64.128.0.143.443: P 407:636(229) ack 3711 win 63890
14:27:18.180350 eth7[out]: 64.164.0.103.44131 > 64.128.0.143.443: P 407:636(229) ack 3711 win 63890
14:27:18.237658 eth7[in ]: 64.128.0.143.443 > 64.164.0.103.44131: . ack 636 win 4535
14:27:18.237675 eth4[out]: 64.128.0.143.443 > 10.1.0.245.52704: . ack 636 win 4535
14:27:18.237907 eth7[in ]: 64.164.0.143.443 > 64.164.0.103.44131: P 3711:4068(357) ack 636 win 4535
14:27:18.237915 eth7[in ]: 64.128.0.143.443 > 64.164.0.103.44131: F 4068:4068(0) ack 636 win 4535
14:27:18.237924 eth4[out]: 64.128.0.143.443 > 10.1.0.245.52704: P 3711:4068(357) ack 636 win 4535
14:27:18.237929 eth4[out]: 64.128.0.143.443 > 10.1.0.245.52704: F 4068:4068(0) ack 636 win 4535
14:27:18.238375 eth4[in ]: 10.1.0.245.52704 > 64.128.0.143.443: . ack 4069 win 65000
14:27:18.238388 eth7[out]: 64.164.0.103.44131 > 64.128.0.143.443: . ack 4069 win 65000
14:27:18.251597 eth4[in ]: 10.1.0.245.52704 > 64.128.0.143.443: F 636:636(0) ack 4069 win 65000
14:27:18.251610 eth7[out]: 64.164.0.103.44131 > 64.128.0.143.443: F 636:636(0) ack 4069 win 65000
14:27:18.308407 eth7[in ]: 64.128.0.143.443 > 64.164.0.103.44131: . ack 637 win 4535
14:27:18.308424 eth4[out]: 64.128.0.143.443 > 10.1.0.245.52704: . ack 637 win 4535

Thanks,
Graham

[amdhim0004]

Hello @Graham 

Thanks for the logs. 

Please have a look at the below logs. Traffic from SRC-10.1.0.245 is coming from eth4 and going out from eth7 with NAT IP 64.164.0.103. But when we get the reply from 64.128.0.143 on eth7 traffic is going out from eth4 of the firewall it's not getting NAT to the internal range. (eth4[out]: 64.128.0.143.4500) this should be your internal IP. 

14:26:57.902029 eth4[in ]: 10.1.0.245.64024 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others I #35[]
14:26:57.902129 eth4[in ]: 10.1.0.245.64024 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others I #35[]
14:26:57.903039 eth7[out]: 64.164.0.103.26672 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others ? #35[]
14:26:57.903107 eth7[out]: 64.164.0.103.26672 > 64.128.0.143.4500: NONESP-encap: isakmp: phase 2/others ? #35[]
14:26:58.051808 eth7[in ]: 64.128.0.143.4500 > 64.164.0.103.26672: NONESP-encap: isakmp: phase 2/others R #35[]
14:26:58.052060 eth4[out]: 64.128.0.143.4500 > 10.1.0.245.64024: NONESP-encap: isakmp: phase 2/others R #35[]