How to deal with DNS over HTTPS, DNS over TLS, QUIC and PSOM?

[Vladimir]

There is now a concerted move on part of multiple service providers to offer DNS over HTTPS. Browser vendors are doing it to differentiate their services supposedly addressing privacy issues, (i.e. Google LOL  ) and now, there is an offering of vendor-independent DNS over HTTPS from Cloudflare that could be found at https://1.1.1.1/  

Since not everyone running HTTPS inspection on their gateways or proxies, probability of evasion for categorized traffic is increasing.

Furthermore, presently the DNS group in services is limited to conventional DNS over UDP and DNS over TCP, so event if we are to inspect the HTTPS traffic, there are no guaranties that we can recognize and act on its DNS payload.

I would like to hear your thoughts on this subject as well as on inspection of the proprietary protocols such as QUIC and PSOM. 

[Daniel_Taney]

I'm also interested in hearing this topic explored / explained further. For the moment, we have just been blocking QUIC unconditionally in our Firewall Policy. But as this stuff gets more common, I have a feeling that won't be a sustainable solution!

[PhoneBoy]

Strict outbound filtering will mitigate some of this. 

This means:

• Not allowing DNS except from specific, known internal DNS servers
• Only allowing TCP port 80/443 outbound with some application filtering for good measure

Of course, stuff tunneled inside HTTPS is still something to potentially worry about.

QUIC in particular, is an interim protocol until HTTP/2.0 is ratified (as I understand it).

Hadn't heard of PSOM before. 

[Vladimir]

That's the thing: in case of DNS over HTTPS the lookup function being performed by the browser engine.

I would like to actually test this in my lab to see if HTTPS inspection is sufficient in recognizing the DNS payload, or if its completely obfuscated.

[PhoneBoy]

You can see how CloudFlare is doing it here: Making Requests - Cloudflare Resolver 

The funny thing is there is still a traditional DNS request to the site that runs the HTTPS DNS resolver.

I'm sure it can be blocked.

[Vladimir]

Nononono, they will resolve the traditional DNS calls at the same address, but if you'll read this: DNS over HTTPS - Cloudflare Resolver , they are talking about embedding the resolver into the applications, OS' and browsers. That's completely different ball game. Then there is DNS over TLS. Essentially, you are running a local proxy on each machine that intercepts calls to port 53, encrypts them and shows those to cloudflare via HTTPS.

What is interesting to see is if their daemon will play well with substituted certificate, or if it'll buck at it being different than the one it expects to see.

I'm busy for the next few days, but will try to get to setting up their daemon in my lab and running queries through the gateway with HTTPS inspection enabled.

I'll let you know how it looks. 

[PhoneBoy]

Right, but there is either going to be a traditional DNS lookup to find that https resolver endpoint embedded in the app OR it will go to a known IP like 1.1.1.1.

This will be an interesting cat and mouse game to track.

[Pedro_Espindola]

Did you mean "until HTTP/3.0 is ratified"?

[Vladimir]

DNS over HTTPS update

OK, as I have promised to do some more digging, these are my preliminary findings:

Secure DNS is beginning to take traction with multiple vendors and open source community actively working on number of products intended for this purpose.

For now, it is still feasible to monitor and control this traffic to some degree as it is being forwarded to well known or advertised IPs and is often recognizable by resource names.

So, if you are not using HTTPS inspection and depending on your company's security posture, you can either block traffic addressed to these servers completely, or implement your own Secure DNS proxies to retain the ability of blacklisting or whitelisting the sites and the categories accessible by users.

If you are using HTTPS inspection, you are marginally better off, as it allows you, in the case of a more mainstream secure dns providers, to see that the DNS query is being performed, but the payload of those queries is still invisible.

Below is the example of the query performed from the Secure DNS proxy running DNSCRYPT with Cloudflare as a resolver:

root@CLFRDNSPRXY1:~# dig https://www.hackaday.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> https://www.hackaday.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41178
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; MBZ: 0257 , udp: 1536
;; QUESTION SECTION:
;https://www.hackaday.com. IN A

;; ANSWER SECTION:
https://www.hackaday.com. 599 IN CNAME hackaday.com.
hackaday.com. 599 IN A 192.0.79.32
hackaday.com. 599 IN A 192.0.79.33

;; Query time: 175 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Apr 10 16:50:02 EDT 2018
;; MSG SIZE rcvd: 159

root@CLFRDNSPRXY1:~#

The gateway sees the request, but not the payload:

Id: c0a8071f-0100-00c0-5acd-23b700000000
Marker: @A@@B@1523332800@C@112360
Log Server Origin: 192.168.7.30
Time: 2018-04-10T20:51:03Z
Interface Direction: outbound
Interface Name: eth1
Connection Direction: Outgoing
Id Generated By Indexer: false
First: false
Sequencenum: 9
Xlate (NAT) Source IP: XX.XXX.XXX.XXX
Xlate (NAT) Source Port: 36248
Xlate (NAT) Destination Port:0
NAT Rule Number: 16
NAT Additional Rule Number: 1
Hll Key: 12784207492398244274
Context Num: 1
Source Zone: Internal
Destination Zone: External
Service ID: https
Source: 192.168.7.34
Source Port: 47494
Destination: 1.0.0.1
Destination Port: 443
IP Protocol: 6
Security Outzone: ExternalZone
Protocol: HTTPS
Sig Id: 0
Lastupdatetime: 2018-04-10T20:51:03Z
Action: Accept
Type: Connection
Policy Name: MobileAccess_for_GW8010
Policy Management: SMS8010
Db Tag: {4FD7C570-BB91-8C47-9F47-12F26CF733F0}
Policy Date: 2018-04-10T20:37:07Z
Blade: Firewall
Origin: GW8010
Service: TCP/443
Product Family: Access
Logid: 7
Action: Inspect
Access Rule Number: 6.2
Rule UID: ac190fa4-a3d5-48cb-986a-9d4a30a02e5e
Layer Name: APCL_and_URLF
Interface: eth1
Description: https Traffic Accepted from 192.168.7.34 to 1.0.0.1

Interestingly enough, even though the session is clearly identifying client type and user agent as dnscrypt-proxy, the corresponding application filter is not seeing it as such:

Time: 2018-04-10T20:10:35Z
Interface Direction: inbound
Interface Name: eth0
Connection Direction: Outgoing
Id: c0a8071f-2d16-0000-5acd-1a3b00000000
Sequencenum: 2
Hll Key: 12784207492398244274
Duration: 2460
Last Update Time: 2018-04-10T20:51:33Z
Update Count: 6
Connections: 11
Aggregated Log Count: 42
Creation Time: 2018-04-10T20:10:35Z
Source: 192.168.7.34
Destination: 1.0.0.1
Destination Port: 443
IP Protocol: 6
Client Type Os: Unknown
Client Type: Other: dnscrypt-proxy
User Agent: Other: dnscrypt-proxy
Protocol: HTTPS
Sig Id: 0
Service ID: https
Source Zone: Internal
Destination Zone: External
Application ID: 3876370621
Method: POST
Action: Accept
Type: Session
Policy Name: MobileAccess_for_GW8010
Policy Management: SMS8010
Db Tag: {4FD7C570-BB91-8C47-9F47-12F26CF733F0}
Policy Date: 2018-04-10T20:37:07Z
Blade: URL Filtering
Origin: GW8010
Service: TCP/443
Product Family: Access
Logid: 320
Action: Inspect
Application Name: cloudflare.com
Primary Category: Computers / Internet
Matched Category: Computers / Internet
Additional Categories: Computers / Internet,URL Filtering
Application Risk: Unknown
Marker: @A@@B@1523332800@C@112445
Log Server Origin: 192.168.7.30
Orig Log Server Ip: 192.168.7.30
Lastupdatetime: 1523393495000
Lastupdateseqnum: 2
Severity: Informational
Rounded Sent Bytes: 16544
Confidence Level: N/A
Rounded Bytes: 61760
Stored: true
Rounded Received Bytes: 49024
URLs: 16
Packets: 59
Total Bytes: 61780
Client Inbound Packets: 28
Client Outbound Packets:31
Server Inbound Packets: 22
Server Outbound Packets:33
Client Inbound Bytes: 3120
Client Outbound Bytes: 7503
Server Inbound Bytes: 8355
Server Outbound Bytes: 2968
Received Bytes: 49052
Sent Bytes: 16552
Access Rule Number: 6.2
Rule UID: ac190fa4-a3d5-48cb-986a-9d4a30a02e5e
Layer Name: APCL_and_URLF
Interface: eth0
Description: https Traffic Accepted from 192.168.7.34 to cloudflare.com(1.0.0.1)
Resource: https://dns.cloudflare.com/dns-query, https://dns.cloudflare.com/dns-query
Layer Uuid Rule Uuid: 9457d7fd-104e-494a-bf23-522eae8d2530_3c504c20-576b-4d87-b97b-d51aa5c7f613, 38746d3a-ecc1-459a-a373-e4cecb87e246_ac190fa4-a3d5-48cb-986a-9d4a30a02e5e
Bytes (sent\received): 60.3 KB (16.2 KB \ 47.9 KB)

And the rule defined to log it, does not (DNSCrypt is an object predefined by Check Point):

Another little bug here is the absence of hits on the rule that clearly should have some:

By my count, there are some 2,999,982 hits missing

We cannot expect that all of those working on Secure DNS solutions will be as obliging as Cloudflare and provide us with either convenient target IPs or a resource proclaiming its intent (i.e. https://dns.cloudflare.com/dns-query).

I think that there should be a dedicated category created and maintained for DNS obfuscation applications and it should be dynamically updated.

Interested in hearing your thoughts on this subject.

Cheers,

Vladimir

[Ryan_St__Germai]

I am going to bump this topic because a DOH C2 module was released for Cobalt Strike.

GitHub - SpiderLabs/DoHC2: DoHC2 allows the ExternalC2 library from Ryan Hanson (https://github.com/... 

[VCL001]

Google just made DOH GA: https://security.googleblog.com/2019/06/google-public-dns-over-https-doh.html

@Dameon - any word on us being able to see DNS tunneled through HTTPS at the application layer?

[Vladimir]

Well, it appears that my worries on this subject were well founded:

First-ever malware strain spotted abusing new DoH (DNS over HTTPS) protocol

[sanjay_palnitka]

Bringing up an older thread.  Now that Firefox has announced that it will enable DNS over htttps in the coming weeks, is there a good solution to block such requests on the gateways, especially if https inspection is enabled?

[Steve_Payne]

You could use the ACST to make a custom application that looks for DNS over HTTPS headers.

[VCL001]

We have application signatures for DNS over HTTPS (DOH) and QUIC in the AppWiki; has anyone tried using those? Please update if it works or if it doesn't.

[Ryan_St__Germai]

Not seeing it. DOH that is. 

Ryan

[Ave_Joe]

I am also not seeing it. Is the AppWiki category version specific?

[sanjay_palnitka]

I don't see it in R80.30. 

However, checkpoint's online Appwiki has it: https://appwiki.checkpoint.com/appwikisdb/public.htm

[Jens_Bauernfein]

I don't see it in the appwiki. What did I need to look for? dns, doh...I find nothing.

Greetings

[VCL001]

Quick clarification by R&D: DoH detection was indeed published. The detection is based on R80.40 capabilities (Supporting HTTP 2.0), therefore it is relevant only for this version.

So will be available in 80.40 when you upgrade. In the meanwhile consider building a custom AppID using the published ports and protocols to block TLS calls to the DNS services published e.g. block HTTPS calls to Google (8.8.4.4 / 8.8.8.8), CloudFlare (1.1.1.1), DNSCurve, DNS Crypt (I think we have a separate AppID for this) and Firefox

[Ryan_St__Germai]

Thanks for the clarification.

Ryan

[sanjay_palnitka]

Thank you for the information.

[Vladimir]

@VCL001 , thanks for the update. Please clarify how DoH calls will be interpreted by the system if HTTPS inspection is enabled (i.e. will they be treated as a composite HTTPS/DNS traffic and reflected in logs as DNS calls)?

Can we use PAT to redirect those to our DNS servers?

The suggestion for 80.30 and below is only partially effective: I am not as concerned about DoH traffic from browsers to the well-known DNS services, but from malware to unknown servers or those spun-up with DGAs.

Thank you,

Vladimir

[PhoneBoy]

Pretty sure we're not interpreting DoH yet, even with HTTPS Inspection enabled.
Even if we tried to, the browser (or whatever) could also be using Certificate Pinning as part of DoH as well, which would make any attempts at HTTPS Inspection fail.

Our Research folks are definitely seeing DNS-related shenanigans in malware campaigns.

[Nik_Bloemers]

And what about SMB appliances? I suppose this will not be supported on 1400 with the R77.20 codebase, but what about the R80 based 1500's?

[sanjay_palnitka]

For R80.30, putting in a rule to block https access to mozilla.cloudflare-dns.com, dns.google and cloudflare.dns.com seems to work.  I guess, for DoH, Firefox uses CDN's so blocking 1.1.1.1 may not work.  (Ref: https://wiki.mozilla.org/Trusted_Recursive_Resolver)

[Jens_Bauernfein]

I use this for now, a little script which feeds a dynamic_object.

8<---

#!/bin/bash -f

# edit R80.XX according to your version
source /opt/CPshrd-R80.30/tmp/.CPprofile.sh
curl_cli -sk https://download.dnscrypt.info/dnscrypt-resolvers/json/public-resolvers.json| jq '.[].addrs' | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"| sort -n| uniq > /dev/shm/temp_doh
dynamic_objects -u dyn_public_doh -r 104.16.248.249 104.16.248.249
dynamic_objects -o dyn_public_doh -r 104.16.249.249 104.16.249.249 -a
dynamic_objects -o dyn_public_doh -r 1.1.1.1 1.1.1.1 -a
dynamic_objects -o dyn_public_doh -r 8.8.4.4 8.8.4.4 -a
while read line; do dynamic_objects -o dyn_public_doh -r $line $line -a; done < /dev/shm/temp_doh

8<---

[Bryan_Adams]

Spun up an 80.40 lab to test this and still not seeing any object for DoH. I do however see it in the threat wiki.

[abihsot__]

hmm, seeing it in the wiki categorized as low. Considering what it enables, would you think it should be at least medium level?

[VCL001]

Sorry for delayed response; I do this part-time :-). Did get a response from R&D to some of the follow-up questions.

@Valdimar 

-          In the management logs, the relevant traffic that would be detected as DoH would be reflected as DoH.  Obviously, HTTPS Inspection must be turned on to correctly identify this traffic.

-          As defined in DoH relevant RFC (RFC 8484), it is compatible with HTTP features, including redirection. PAT is beyond the scope of APPI, and I am not familiar with another relevant capability.

-          This traffic would be logged as an application.  It would be further categorized as a Network Protocol.

-  have asked if we will add DOH and DOT to the service catalog and make it part of the access control layer too.

@Bryan_Adams : looks like the first release of 80.40 has some stuff missing. The AppID is included in a newer application package, but the management code still has older update package and so doesn't download the new package. I presume they will fix this with HF_1, which should be out soon. For now, if you want to, there is a bash script that fixes the issue that can be asked for from R&D.