topic Re: Remote Access with multiple external interfaces - topology download in SASE and Remote Access

Remote Access with multiple external interfaces - topology download

Duane_Toler — Thu, 24 Jun 2021 19:47:53 GMT

Yep, yet another post about Remote Access VPN with multiple external interfaces. No, not ISP redundancy. Yes, I did the Link Selection; Yes, I did the checkbox in the Office Mode section; Yes, I read the VPN guide documentation; Yes, I did the SK digging, reading, Check Mates posts, and even put it in a VM test, R80.30 with JHF 236. Yes, it actually works for *IPsec and NAT-T* packets.

However, the problem is port 443 for site update and topology download. That part doesn't work. The client starts to connect, VPN debug gets all the way to Main Mode packet 4, but fails at the XAuth step. Yes I have the user defined, in the right group, and when I set the gateway default route to face back to my client, it fully connects just fine.

Topology-wise, I have eth0 as my "internal network", eth1 is my regular default gateway out, and eth2 is my "other" external interface (without the default route). I'm trying to connect to the 2nd external interface on eth2. The breakdown is when eth1 is default gateway. The firewall is writing packets with source IP of eth2 (my connecting interface, which is the interface I chose in Link Selection "use this IP from the topology table". I tried the GUIDBedit hacks, but that didn't work either.

When the client makes TCP SYN connection to firewall at eth2 on port 443, the gateway replies out eth1 (with eth2's IP as source) back to the client with TCP SYN-ACK.

So I did a nasty stupid thing, and put in a Linux VM as a Test, made it a router back to my client (yep it's stupid, and yes I got martians, but I disabled the rp_filter on it, and yes it's asymmetric routing, but packets got back to my client, albeit very circular; i don't care, it worked). Client now gets the TCP SYN-ACK, replies with TCP ACK, and topology gets downloaded, and client authenticates.

From that point onward, connections to the VPN domain are just IPsec NAT-T. THIS part works correctly; the firewall gets inbound IPsec NAT-T connection on eth2, it responds on eth2 (the fw ctl zdebug is spooky, too; it's writing packets on eth2 to the next-hop MAC address that sent the packet originally; even tho the default gateway is eth1! nice trick!).

For site-to-site IPsec VPN, everything would work normally and correctly from the start; none of the above is a problem. However, it's a problem for Remote Access VPN. Without doing ISP redundancy (ick; this config is going to be applied later to an HA cluster), I can't see how this is going to work. I tried the probing methods, too, and those didn't work.

Has anyone ever gotten this to work? The key here is the default gateway is NOT the interface on which connections would terminate. Again, the issue is with port 443 topo/xauth, not IPsec.

Re: Remote Access with multiple external interfaces - topology download

PhoneBoy — Thu, 24 Jun 2021 20:16:10 GMT

So is there a route back to your client on the gateway through eth2?
Or is it on the same subnet?

Re: Remote Access with multiple external interfaces - topology download

Duane_Toler — Thu, 24 Jun 2021 20:35:39 GMT

No, not anymore, and it normally wouldn't be (the client reach the gateway through normal routing). The gateway *could* reach the client back on eth2 (and it does for IPsec and NAT-T packets, by writing a packet with destination MAC address of the host that sent it). Normally you'd think "well, silly-goose, that's your problem" (and normally I'd agree). For a real-live gateway with multiple external interfaces (and no dynamic routing), one static default gateway (out eth1), the other external interface won't have a default route (because.... default, by definition). This 2nd external interface is where VPN connections will terminate.

But again, it works for IPsec and NAT-T. The new packet on the wire is the MAC of the ingress next-hop that sent the frame (I even see this MAC address tracked in the zdebug with "-m VPN + all"; that's really clever!)

Here, the host Hades is *my* local LAN router, not the Check Point firewall (eth2 just happens to be the same). 10.0.3.236 is my client. 10.233.31.80 is the firewall's eth2. Hades eth2 MAC addr ends in "b0:4b". Hades is between my client and the firewall.

[root@hades ~]# ifconfig eth2

eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 10.233.31.30 netmask 255.255.255.0 broadcast 10.233.31.255

ether 00:0c:29:18:b0:4b txqueuelen 1000 (Ethernet)

[root@hades ~]# tcpdump -nni eth2 host 10.0.3.236 -e

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes

15:01:27.652944 00:0c:29:18:b0:4b > 00:0c:29:7a:76:63, ethertype IPv4 (0x0800), length 118: 10.0.3.236.54508 > 10.233.31.80.4500: UDP-encap: ESP(spi=0x1120e857,seq=0x31), length 76

15:01:27.677150 00:0c:29:7a:76:63 > 00:0c:29:18:b0:4b, ethertype IPv4 (0x0800), length 118: 10.233.31.80.4500 > 10.0.3.236.54508: UDP-encap: ESP(spi=0x59f1983d,seq=0x23), length 76

When it came to topology and xauth... crickets (well, as far as "doing the right thing" is concerned): [this was earlier in the day, hence the time difference]

[det@hades ~]$ sudo tcpdump -nni eth2 host 10.0.3.236

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth2, link-type EN10MB (Ethernet), capture size 65535 bytes

10:46:08.183726 IP 10.0.3.236.49674 > 10.233.31.80.443: Flags [S], seq 3336228268, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 875756382 ecr 0,sackOK,eol], length 0

10:46:12.186856 IP 10.0.3.236.49674 > 10.233.31.80.443: Flags [S], seq 3336228268, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 875760382 ecr 0,sackOK,eol], length 0

10:46:12.957761 IP 10.0.3.236.49676 > 10.233.31.80.443: Flags [S], seq 2415593851, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 875761152 ecr 0,sackOK,eol], length 0

>>> here is where i noticed that NAT-T was actually working from the beginning:

10:46:12.960337 IP 10.0.3.236.55873 > 10.233.31.80.4500: NONESP-encap: isakmp:

10:46:12.977206 IP 10.233.31.80.4500 > 10.0.3.236.55873: NONESP-encap: isakmp:

>> but more crickets for XAuth/topology:

10:46:13.031651 IP 10.0.3.236.49676 > 10.233.31.80.443: Flags [S], seq 2415593851, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 875761223 ecr 0,sackOK,eol], length 0

10:46:13.066552 IP 10.0.3.236.49676 > 10.233.31.80.443: Flags [S], seq 2415593851, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 875761253 ecr 0,sackOK,eol], length 0

10:46:13.099089 IP 10.0.3.236.49676 > 10.233.31.80.443: Flags [S], seq 2415593851, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 875761283 ecr 0,sackOK,eol], length 0

10:46:13.128996 IP 10.0.3.236.49676 > 10.233.31.80.443: Flags [S], seq 2415593851, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 875761313 ecr 0,sackOK,eol], length 0

10:46:18.587032 IP 10.0.3.236.64964 > 10.233.31.80.4500: NONESP-encap: isakmp: child_sa #67[IVR]

10:46:18.588131 IP 10.233.31.80.4500 > 10.0.3.236.64964: NONESP-encap: isakmp:

10:46:20.250705 IP 10.0.3.236.64964 > 10.233.31.80.4500: NONESP-encap: isakmp: phase 1 I ident

10:46:20.252367 IP 10.233.31.80.4500 > 10.0.3.236.64964: NONESP-encap: isakmp: phase 1 R ident

10:46:20.252995 IP 10.0.3.236.49674 > 10.233.31.80.443: Flags [S], seq 3336228268, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 875768383 ecr 0,sackOK,eol], length 0

10:46:20.259071 IP 10.0.3.236.64964 > 10.233.31.80.4500: NONESP-encap: isakmp: phase 1 I ident

10:46:20.306897 IP 10.233.31.80.4500 > 10.0.3.236.64964: NONESP-encap: isakmp: phase 1 R ident

(of course ISAKMP couldn't get far because xauth never completed...)

Sooooo, das boog?

Re: Remote Access with multiple external interfaces - topology download

PhoneBoy — Thu, 24 Jun 2021 20:40:55 GMT

Ok, now I get it.
Might be a bug, might also be intended behavior.
Probably requires a TAC case to be sure.

Re: Remote Access with multiple external interfaces - topology download

Duane_Toler — Fri, 25 Jun 2021 16:32:20 GMT

Aww boo 😞

I opened a case for it. 😕

I read through the "-m VPN + all" debug again and I see another section where, for NAT-T, it does the force-reroute of the packet out the Link Selection non-default interface (correctly). However, there's nothing close to that for the port 443 packets. I'm guessing those packets are handed off to the FW worker instance for authentication, etc. I ran a kdebug of "-m fw" and see where it's choosing to write packets per the routing table (even though I have that option set to "reply from same interface").

So..., case open, debug info sent along with it. Now I wait. 🙂

Re: Remote Access with multiple external interfaces - topology download

the_rock — Sat, 26 Jun 2021 01:35:04 GMT

Ok, just an idea...what does it show if you run ip route get and then IP address you are want to check on the firewall itself?

Re: Remote Access with multiple external interfaces - topology download

MartinTzvetanov — Mon, 28 Jun 2021 08:44:05 GMT

Since which JHF did you start observing this issue? I'm pretty sure I hit the same issue trying to patch to 236 on clusters with multiple external interfaces, but didn't have enough time to dig into the traffic.

Re: Remote Access with multiple external interfaces - topology download

Duane_Toler — Mon, 28 Jun 2021 13:29:41 GMT

I see the addresses in the output (I presume this is what you mean):

local 10.233.31.80 dev lo src 10.233.31.80

cache <local> mtu 16436 advmss 16396 hoplimit 64

No that IP is not on the Loopback device; I believe the route entry here just means to accept and process connections to that address (and any others locally-connected).

Re: Remote Access with multiple external interfaces - topology download

Duane_Toler — Mon, 28 Jun 2021 13:35:34 GMT

I haven't tried this configuration on an earlier version. I was on R80.30 JHF 228 and it didn't work; updated to R80.30 JHF 236 and still didn't work. I have a TAC case open and I'm going to follow-up on that either this afternoon or tomorrow (time-permitting).

Re: Remote Access with multiple external interfaces - topology download

MartinTzvetanov — Mon, 28 Jun 2021 14:24:28 GMT

In my situation it was working with 191 and after patch to 228 or 236 it broke.

Re: Remote Access with multiple external interfaces - topology download

Duane_Toler — Mon, 28 Jun 2021 14:37:41 GMT

Oh wow, that's "exciting". I only had "strong belief" from earlier versions (e.g.: R77.30, i mean) that it worked "correctly", even tho I never independently verified and monitored it. I also knew it worked with ISP Redundancy enabled, but admittedly I never closely monitored the exact path. Way-back-when in R75.40, I also had it working and tested failover/outages, but that also was with ISP Redundancy. So for me, this specific configuration is wholly-untested, I admit. Good to know that you had it working in JHF 191, too.

I need to test it with R80.20 this week; it's on my short-list for the week now, so I'll find out in a few days when I fire up a VM.

Re: Remote Access with multiple external interfaces - topology download

MartinTzvetanov — Mon, 28 Jun 2021 14:43:06 GMT

I don't have ISP redundancy, but have 2 ISPs and BGP routing and total 10 external interfaces.

Re: Remote Access with multiple external interfaces - topology download

Duane_Toler — Mon, 28 Jun 2021 14:46:02 GMT

Sounds similar to what I have here, except not 10 ext. interfaces. 🙂 Only 2 here, but also no ISP redundancy involved. I'll update the thread with what I find. My TAC case is now "assigned" to someone, so I'm waiting on that person to review and write back.

Re: Remote Access with multiple external interfaces - topology download

the_rock — Mon, 28 Jun 2021 14:48:30 GMT

Yes, please let us know what they say, Im very curious. Honestly, cant say I ever seen this issue in R80.40 version.

Re: Remote Access with multiple external interfaces - topology download

Duane_Toler — Tue, 29 Jun 2021 01:55:07 GMT

I got an R80.20 VM with JHF 183 [matching a customer's install right now]. This version works as I initially described, which is "it doesn't work without asymmetric routing". 😕 I'm going to step through JHF 187 next, then 190 after that to see if either of those work.

After that, I'll look into updating to R80.30 with the JHF 191 to see if it works there, to match what you said [Martin].

As for my TAC case, today was crickets (but I also was busy myself elsewhere and did't have a good opportunity to call in). My "need" from TAC right now, tho, is to review the info and ask within about what is needed to make this capability work correctly (versus a phone call to re-state everything, which is annoying).

Re: Remote Access with multiple external interfaces - topology download

MartinTzvetanov — Tue, 29 Jun 2021 06:28:27 GMT

My customer's scenario has been working since R75.x until R80.30 JHF 236. It's obvious that it's related to the topology and something else specific but now sure if I could have the opportunity to dig in it.

Re: Remote Access with multiple external interfaces - topology download

Duane_Toler — Tue, 29 Jun 2021 14:13:17 GMT

Yeah R75-ish is when I last actively used this myself. I just had an interesting effect as I was slowly adjusting the knobs. I turned the gateway into "probing method" for everything (see screenshot). In CLISH, I had the interfaces in their natural state (main address having the default route, not the Link Selection interface). I connected the client [yeah still asymmetric routing here] to update topology. After that, the client immediately rolled over to Visitor Mode only (no IPsec, no NAT-T; just TCP 443 only), and the client changed its own configuration to be that of Main Address in the gateway properties! NOT one of the Link Selection external interfaces... whoa... wicked.

I then downed the "Main Address" interface [with the default route on it], but the client did not roll back over to the Link Selection interface. It's stubbornly only connecting to the Main Address, completely ignoring Link Selection and probing. So it still doesn't behave as it should.

*For Me*, this is what I want; to be able to "move" the clients to a new Main Address without pushing out new client configurations. Regardless, Link Selection still isn't working.

I'll keep poking before updating HFAs. I want to get an idea of how this is behaving before changing conditions. 🙂

Re: Remote Access with multiple external interfaces - topology download

Duane_Toler — Tue, 29 Jun 2021 18:04:05 GMT

Well now.... here in R80.20 JHF 187/188 is the same broken behavior as JHF 183. Jumbo 190 things changed slightly, but still not correct.

Now, when it's all probing method, the client does NAT-T to the gateway on the Main Address interface instead of Visitor Mode. That much is slightly better. Regardless, the client never rolls over to a Link Selection interface. I even went through GUIDBedit a bit, as in sk32229, but that didn't change anything. TAC also came back with that same SK, but it doesn't seem to have any effect in any way.

Other than NAT-T instead of Visitor Mode, the remaining behavior is still the same broken behavior. 😕

I'll try R80.20 JHF 202 (ongoing take) just for fun, but then I'll take it to R80.30 < JHF 191 to see how that goes.

Re: Remote Access with multiple external interfaces - topology download

Wolfgang — Tue, 29 Jun 2021 19:56:58 GMT

Hello folks,

I read this post and maybe I don‘t understand, but the described behaviour looks normal to me. Let me explain…

If a client connects to your gateway to the IP address of eth2 the answering packet will be routed back via the default route or via a specific route for the client IP. If the default route is going through eth1 then this is normal behaviour.
The solution for this is ISP redundancy, with this feature enabled an answer packet is routed back through the same incoming interface.

That‘s my understanding. Now the VPN part…. You can follow link selection for VPN and define every available interface as destination and the choice of the route back and source IP needs to be configured. Normally these apply to remote access VPN.

Sometimes you want to have different settings for site2site and remote access vpn. This can be enabled via GUIdbedit following

Link Selection for Remote Clients

Set „apply_resolving_mechanism_to_SR“ to false and with the setting of „ip_resolution_mechanism“ you can define all your needed interfaces.

Don‘t forget to set „Support connectivity enhancement for gateways with multiple external interface“ in the office mode section.

Re: Remote Access with multiple external interfaces - topology download

Duane_Toler — Tue, 29 Jun 2021 20:12:48 GMT

The problem is that the gateway isn't responding to XAUTH and topology downloads on any interface except either A) the main address (such as when probing method is used, as I just found), or B) asymmetrically with combination of a Link Selection interface and the non-LS interface with default route. NAT-T traffic is being emitted correctly on the Link Selection interface with disregard to the default route (and this is the expected and desired behavior; link-level re-writing of packets). Instead, the gateway is writing packets out the default route interface, yet with the source IP of the chosen Link Selection interface. 😕 That's worse; it fools you into thinking "it works" until you down the default-route interface (with the Main address). Then it doesn't work at all, unless you have a floating static route active.

The correct behavior is that the gateway should only respond to the connection in and out of the chosen Link Selection interface and it shouldn't need ISP redundancy to do it (although ISP Redundancy does lean on probing and the cpisp_update script to override the default route). This worked before R80; both Martin and I had it working.

That SK appears to have no effect now, either. I just went through it with several values in GUIDBedit and none of them made the client work as expected.