Possible E84.30 issue?

[biskit]

Just had a problem with a customer which may be coincidental - but one to keep up your sleeve just in case...

Removed old VPN client (because of the 1st Jan issue).  Installed E84.30 on over 100 laptops.  Most work fine, but 5 of them wouldn't get domain suffix and DNS info correctly after installing E84.30.  Remove E84.30 and install E84.00 instead.  Everything works normally again.  Remove E84.00 and put E84.30 back on and it stop again.

We have a TAC case open already.  This is just a heads up in case you hit weird behaviour using E84.30?

[law24]

Having the same issue and above was also our solution, running with E84.00 on selected clients now

[law24]

Having same issue and above was also our solution. Selected clients now running E84.00

[HristoGrigorov]

Are these Windows 10 machines and are they running different Windows 10 version than the others ? 

[law24]

All ours were Win10 1909

[Tobias_Moritz]

Can you verify if the non-working E84.30 clients are using onboard LAN while the working ones are using WLAN/WWAN or an external adapter?

If yes, you may hit the issue mentioned in this thread:
https://community.checkpoint.com/t5/Endpoint-and-SandBlast-Agent/E84-20-6108-Remote-Access-VPN-Blade...

where Check Point says:

We are familiar with the DNS issue over LAN and have a fixed version for E84.20.

We are also currently working on a fix for E84.30.

Please create a Service Request and we'll provide the package.

---

---

[AndreiR]

@biskit ,

I personally apologize for this issue.

Unfortunately E84.20 and E84.30 are affected to the bug when Virtual Network Adapter interface gets higher metric which means lower priority. Due to this DNS requests inside encryption domain go through different interface and don't rich DNS server. As a result user can connect to VPN gateway but can't access resources behind it.

The issue has been fixed in coming E84.40. We are finishing its testing and it should be available in the middle of January 2021.

If you need to address this issue prior E84.40 release you have two options:

1. You may contact Check Point support and request CFG version for your client (either E84_20_CFG_08 or E84_30_CFG_06)

2. You may utilize sk109223 and change interface matric by yourself.

[biskit]

Hi @AndreiR ,

Thanks very much for your update.  I've searched for sk109223 and get "Ne results found".  Should mere mortals like myself be able to view this sk? 🤣

Thanks,

Matt

[law24]

Getting the same, thought I was doing something wrong so kept quiet. 😋

[mgades]

FWIW, I can share another finding, but the issue below is also present in earlier clients (like E83.20) and still is.
It seems related, but then again not, so while I have your attention, here goes:

The following conditions must be met:

• Client is Windows 10 PC
• Internet facing NIC(s) on client has an IPv6 address (besides IPv4 because it is dual-stacked)
• Client has one or more IPv6 DNS server addresses defined
• Endpoint Security VPN client connects successfully to security gateway

The symptom is that DNS requests to internal resouces fail (read: resources held by DNS zones served by the internal DNS servers defined on the virtual Checkpoint NIC).

I have observed the following:

All DNS requests are made to the external DNS server(s), besides the internal DNS servers.

While this behavior calls for a WTF?!, I found and believe this is actually by design, because of the "Smart Multi-Homed Name Resolution" introduced back in Windows 8, and refined in Windows 10. Simply explained: Windows sends every DNS request to all servers defined on connected NIC's simultaniously and uses the reply from whichever server responds fastest. Of course there are way more complicated algorithms behind, which takes a lot of factors into consideration (ie. LLMNR and NetBT) and also dual-stack preferences. Worth mentioning is that the OS always prefers IPv6 over IPv4 (because IPv6 is the standard/current IPv6 protocol on the Internet, IPv4 is legacy - let that sink in for now :))

Besides the DNS leaking (which could be a privacy issue), it causes problems resolving internal resources (but only if the physical/LAN NIC has an IPv6 address). Especially if the domain name in question is also registered in public DNS (the latter zone obviously not containing all the internal resource records only reachable by internal DNS servers behind the VPN).

I used Wireshark a lot to troubleshoot what was happening on the wires (both on the physical NIC(s) and the VPN adapter), and I could confirm the following:

Legend:

• VPN client IP: 10.6.6.6
• LAN NIC IP: 172.16.1.6 and 2001:db8:501:beef::dead:affe:12d
• Internal resource: intsrv.domain.com [10.10.10.10]
• Internal DNS server: 10.10.10.53
• External DNS server: 8.8.8.8 and 2001:4860:4860::8888

When resolving an internal resource the client sends the DNS request simultaneously to the internal DNS server and the external DNS server.
The internal DNS server replies with the correct A record (eg. 10.10.10.10).
The external DNS server replies with "No such name" (as expected, since the A record doesn't exist in the external zone). When asking a IPv6 DNS server it queries for both the A and AAAA record.
Replies are received around the same time - some times the internal DNS replies first, sometimes the external one, so it doesn't seem to be related to timing.

Here is when it differs:

1. When the A record (IPv4) for intsrv.domain.com is returned (in this case "No such name") by the external DNS server via IPv4, the client uses the A record received from the internal DNS server (read: it works, internal resource can be reached by FQDN)
2. When the A record (IPv4) for intsrv.domain.com is returned (in this case "No such name") by the external DNS server via IPv6, the client uses the "No such name" verdict, even though it also received the correct A record from the internal DNS server (read: it fails, internal resource can't be reached by FQDN).

In both scenarios, a "nslookup intsrv.domain.com" from the command prompt, will _always_ return the A record reply (from internal DNS, because the RR exists).
If I lookup "indirectly" by pinging the FQDN ("ping intsrv.domain.com") it doesn't resolve when an IPv6 address is defined on an connected/active NIC.

I can always resolve resource records (RR) which also exists in the DNS zone reached through public/external DNS servers. (eg. www.domain.com, etc).

Through the troubleshooting I also tried a few things:

I'm not even sure it is caused by this, as I tried disabling the feature (DisableSmartNameResolution) in a local Windows policy to troubleshoot, and it didn't make any difference. I think is is a corner case internally in the protocol preferences.

The obvious "fix" is to disable IPv6 completely (remove the IPv6 tick from LAN interface settings), but this is backwards, counter productive and not a solution, as we are actually in the process of implementing IPv6 on our networks and is actually something we want!

I created a workaround, causing least possible drawbacks*, and still supporting access to IPv6 resources (via the Internet facing NIC, because this is OK in this split-tunnelling setup).

If I set configure a dummy site-local (not link-local) IPv6 address on the virtual Checkpoint Endpoint Security VPN NIC (eg. fd00::dead/64), DNS resolution starts working as expected! Since IPv6 isn't supported at all in the Endpoint Security VPN, it isn't used - the interface just has an address defined. If I disable the IPv6 protocol on just the Checkpoint VPN NIC it still fails (as the physical NIC still has a valid IPv6 GUA).

It still sends the simultanious DNS requests to both internal/external DNS servers (because this is by default Windows 10 OS behavior), but now it uses the A record received from the internal DNS server in stead of failing on the "No such name" from the external DNS server. If I remove the IPv6 address again (and do a "ipconfig /flushdns") it fails again.

Please let me know if you need further information, wireshark dumps, etc, and I'll gladly provide it.

So my two questions are:

1. Is this inappropiate behaviour an Checkpoint issue, an Windows 10 issue or combined. Is it addressed by Checkpoint at all?
2. And the other really big question (which seems to always be avoided for some reason): When will Endpoint Security VPN get support for IPv6?

Getting IPv6 support in Remote Access VPN/Endpoint Security would actually solve the above problem in more than one way and IPv6 can be treated in security policies in the same way as the legacy IPv4 protocol. We already have feature parity in many blades, but this essential component lacks it (pssst: Cisco AnyConnect has supported it for ages).

*) One drawback is that DNS resolving an internal dual-stacked host return both the A+AAAA records and since Windows prefers IPv6 it will try to contact that host using IPv6 (via the LAN interface, and the internal resource cannot be contacted via VPN using IPv6). If that internal dual-stacked host was a HTTP/HTTPS resource the user wouldn't really notice, because of Happy Eyeballs doing its thing behind the scenes in the browser, and fall back to contacting the host via IPv4 over the VPN.

Thanks in advance!

Morten Gade Sørensen

[biskit]

@mgades thanks for the detailed findings!  Unfortunately now I'm too far removed from the issue to check or confirm anything else (the customer has installed E84.00 and everything works perfectly again).

[AndreiR]

Hi @mgades ,

Thanks for your article. That's true, Check Point Remote Access VPN doesn't support IPv6 yet. We are building roadmap for VPN clients for 2021-2022 and we will take your feedback into account for sure. 

[mgades]

Hi @AndreiR ,

I'm really glad to hear that you will push forward in getting it on the roadmap. IPv6 is the foundation of the modern Internet and I'm sure many customers will benefit from supporting it in the Remote Access VPN product(s).

Do you think the DNS issue I describe above is or will be fixed in the current release train? Can you reproduce it?
Or is it anchored in the Windows 10 IPv6/IPv4 protocol priorities, which favors DNS replies from DNS servers defined on interfaces with a routable IPv6 address over DNS servers definded behind VPN, despite the metric for VPN being better?

Thanks in advance!

[Soren_Kristense]

Any update on E84.40 release date?

[mgades]

Hi Søren

E84.40 is out now: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solut...

Morten

[Soren_Kristense]

Hi.

Is this an issue for split tunneling only or is it also on route all traffic through gateway?  

[biskit]

I'm not sure.  I can't find out easily at the moment I'm afraid.

[George_Sas]

We run split tunneling , so the issue is still there.