HTTPS Inspection over IPv6 on R82

[oa_munich]

Hello all,

As R82 was released, I tried it out on a test gateway. My goal was to try IPv6 prefix delegation, which was introduced in R82 - I successfully received and distributed a prefix.

While trying this out, I encountered a strange issue with HTTPS inspection, when the inspection occurs via IPv6. See attached screenshots.

A client (2001:a61:30b2:bb10:445a:95fe:caf:8ed5) initiates a connection to a website (2606:4700::6813:df4f).

In the screenshot, you can see that the firewall accepts the connection initiated by the firewall itself, i.e. the probe (first screenshot, lowest row). After a noticeable timeout (initial connection at 16:38:22, client connects at 16:38:38 - 15 seconds), the client is permitted to connect, and no inspection takes place.  The log shows that "The probe was unable to establish a TCP connection to the destination". I explicitly permitted the IPv6 address of the firewall to connect anywhere, cp2-ipv6-prefix (2001:a61:30b2:bb10:21c:7fff:fe88:996f).

If IPv6 is disabled, HTTPS inspection is working as expected, there is no generic configuration error, and the issue seems to be related to the way the probe initiates the connection via IPv6.

Any suggestions where to dig to understand why is this happening?

[PhoneBoy]

Do you see the gateway probe the destination via tcpdump or similar?
Guessing it's a bug and a TAC case will likely be necessary.

[oa_munich]

I ran a packet capture on an upstream router and saw probe packets. Will get TAC to take a look, thanks!

[Ilya_Yusupov]

Hi @oa_munich ,

Is it happen for all web sites or only for a specific one?

I tried in my lab and it looks fine, i am getting inspection with IPv6.

Thanks,

Ilya

[oa_munich]

It happens with a handful websites I've tested with, e.g. fast.com, whatismyip.com and a few more.

Are you assigning IPv6 address through prefix delegation?

[oa_munich]

Maybe a little more context here: I've got a bonded interface with 2 VLANs. VLAN 10 is a connection I am receiving the prefix on, VLAN 100 is the connection where I am distributing it.

Config:

set dhcp6 server enable
set dhcp6 client client-mode prefix-delegation
set dhcp6 prefix-delegation method dhcpv6
add dhcp6 prefix-delegation assign-to bond0.100
set dhcp6 prefix-delegation request-from bond0.10

The address (2001:a61:30b2:bb10:21c:7fff:fe88:996f) is the address seen on the bond0.100 interface, 

show interface bond0.100
...
ipv6-address 2001:a61:30b2:bb10:21c:7fff:fe88:996f/64
...

Clients get their IPv6 addresses on bond0.100.

I tested a little more, this happens on every website which does not get bypassed due to policy. Every "new" site - first time a client tries to open and waits for 15 seconds, then the site gets bypassed, subsequent attempts - the inspection is bypassed immediately.

[the_rock]

I find this super interesting...may test it some time this week in R82 lab.

Andy

[Ilya_Yusupov]

Hi,

the IPv6 delegation should not be related here, can you share with which browser are you testing it?

Thanks,

Ilya 

[oa_munich]

Chrome and Brave (Chromium based) on an Android phone and a Mac.

[Ilya_Yusupov]

I tried several times to replicate this in my lab and i don't see any issues

as you can see below i tried fast.com and it got inspected.

If you have time today and would like to have a session with me it will be great so we can go over together with you and see what's is the problem.

[Ilya_Yusupov · Accepted solution]

Updating the thread with the findings on this case.

First it was very interesting case so @oa_munich thanks for sharing.

Second after we build same topology in our lab, we found out that our topology is missing a route on the router side of link local address of the GW.

Explanation - In such topology we have interface external one which has only IPv6 link local address, the traffic is routed via default route to fe80 address of the router, when we initiating in our case local conn from the GW tcp conn, we are doing NAT so the traffic is flows with fe80 link local address, then the router receive the connection and doing NAT to get out to the internet, the router will get the reply then in his dst post NAT we will have fe80 of the GW hence it required to add route of the fe80 address of the GW via dev near side to the GW on the router.

The above will resolve the issue.

@oa_munich - feel free to update me here or in private if that's didn't work for you.

Thanks,

Ilya 

[oa_munich]

Thank you @Ilya_Yusupov and the team for spending time with me and ultimately getting to the bottom of this. One minor correction to the above statement: as the external facing interface gets only a link-local address, the upstream router _must_ do static NAT of the gateway link-local address to a public ipv6 address. As the downstream interface did receive a public ipv6 address from prefix delegation, I got stuck in thinking that as the gateway had received a public ipv6 address, nothing else was required.

So, step 1 would be to add translation of the link-local ipv6 address to a public ipv6 address on the upstream router.

Step 2 would be to ensure proper routing as suggested above.

[the_rock]

This is why I tell everyone how great you are! You helped me the same way with very COMPLICATED issue and it showed you truly cared and would not give up until it was solved.

Andy