Subject: Tsvart last call review of draft-ietf-doh-dns-over-https-13

From: Fernando Gont <[email-masked]>
Date: Sat, 11 Aug 2018 01:00:24 -0700

Reviewer: Fernando Gont
Review result: Almost Ready

Hi, all,

I've reviewed this document as part of the Transport Area Review Team's
(TSVART) ongoing effort to review key IETF documents. These comments were
written primarily for the transport area directors, but are copied to the
document's authors for their information and to allow them to address any
issues raised. When done at the time of IETF Last Call, the authors should
consider this review together with any other last-call comments they receive.
Please always CC ​[email-masked] if you reply to or forward this review.

Please resolve these comments along with any other Last Call comments you may
receive. Please wait for direction from your document shepherd or AD before
posting a new version of the draft.

Document: draft-ietf-doh-dns-over-https-13
Title: DNS Queries over HTTPS (DoH)
Reviewer: Fernando Gont
Review Date: August 11, 2018

Summary:

This document is almost ready, but requires some clarifications and, more
importantly, an analysis of the impact of switching from a connection-less
protocol (UDP) to a connection-oriented protocol (HTTPS/TCP) for DNS resolution.

** Technical comments **

* Page 5:
>  Using the GET method is friendlier to many HTTP cache
>  implementations.

Could the queries really be cached for the POST case?

* Page 15:
> It is the choice of a client to either
>    perform full DNSSEC validation of answers or to trust the DoH server
>    to do DNSSEC validation and inspect the AD (Authentic Data) bit in
>    the returned message to determine whether an answer was authentic or
>    not.

Relying on the DoH server for DNSsec validation does not seem like a good idea.
You want DNSsec to be end-to-end.

* Page 15 (Security Considerations):

DoH essentialy switches from a connection-less transport (UDP) to a
connection-oriented one (TCP). This means that now the server should take care
of all state-exhaustion attacks against TCP (e.g., take a look at:
https://www.gont.com.ar/papers/tn-03-09-security-assessment-TCP.pdf). Defending
against such attacks maybe non-trivial. This should at least be mentioned in
the security considerations.

* Page 16:>
>    HTTP [RFC7230] is a stateless application-level protocol and
>    therefore DoH implementations do not provide stateful ordering
>    guarantees between different requests.

Are you meaning that requests that are pipelined could be responded in
different order? Something else?

* Page 16:
>    A DoH server is allowed to answer queries with any valid DNS
>    response.  For example, a valid DNS response might have the TC
>    (truncation) bit set in the DNS header to indicate that the server
>    was not able to retrieve a full answer for the query but is providing
>    the best answer it could get.  A DoH server can reply to queries with
>    an HTTP error for queries that it cannot fulfill.

Why should this happen? Why not encode this information in the encoded DNS
response (in wire format)'

** Editorial comments **

Page 3:
>  Two primary uses cases were considered during this protocol's
>  development.

s/uses/use/

Page 3:
Please introduce and expand de acronym "DoH" the first time you employ it.

Page 7:

> This might mean that the
>    DoH client retries the query with the same DoH server, such as
>    authorization failures (HTTP status code 401 [RFC7235] Section 3.1).

s/such as/such as for/ ?

* Page 16:> Note that these deadlocks
>    also need to be considered for server that a DoH server might
>    redirect to.

s/for server/for servers/ ?

Thanks!
Fernando

---

From: "John Levine" <[email-masked]>
Date: 11 Aug 2018 10:16:04 -0400

>protocol (UDP) to a connection-oriented protocol (HTTPS/TCP) for DNS resolution.

But DNS resolution has always worked over TCP.  See RFC 1035, section
4.2.2.  Tbe usual case is retry on TCP when a UDP response is
truncated, but it's equally valid to do TCP in the first place.

Is there a reason that the security threats of DOH over TCP would be
any diferent from existing DNS over TCP?

R's,
John

---

From: Benjamin Kaduk <[email-masked]>
Date: Sat, 11 Aug 2018 10:05:38 -0500

[perhaps trimming to just ietf@ was overzealous?]
> any diferent from existing DNS over TCP?

Well, HTTPS pulls in the TLS crypto and its potential increased resource
consumption, but in general TLS tries to avoid DoS opportunites where a
client can make the server do lots of work without having first provided
some indication that the client is "real".

-Ben

---

From: "John R Levine" <[email-masked]>
Date: 11 Aug 2018 11:40:54 -0400

> some indication that the client is "real".

Well, OK.  The obvious next question is whether DoH is different from any 
other https client request.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Benjamin Kaduk <[email-masked]>
Date: Sat, 11 Aug 2018 10:46:24 -0500

> other https client request.

Having not read the draft recently, "I don't think so".

-Ben

---

From: "John R Levine" <[email-masked]>
Date: 11 Aug 2018 11:52:51 -0400

> Having not read the draft recently, "I don't think so".

I'm pretty sure you're right.  It's just a DNS request/response pair 
wrapped in http headers.

Regards,

---

From: Patrick McManus <[email-masked]>
Date: Sat, 11 Aug 2018 18:56:56 -0400

thanks for the review!

> Could the queries really be cached for the POST case?
sometimes. sort of. see 7231 4.3.3

> You want DNSsec to be end-to-end.
my understanding is that giving the client the choice between doing DNSSEC
validation and trusting the recursive to do so is standard DNS practice.

> connection-oriented one (TCP).

I disagree a bit. It adds a third TCP based DNS based approach to the DNS
over TCP and DNS over TLS approaches already standardized so it is not
breaking new ground in that sense.

This means that now the server should take care
> the security considerations.
DoH seeks to use HTTPS substrate best practices in order to inherit the
well known properties and defenses of HTTPS. The first sentence of the
security considerations tries to say this "Running DNS over HTTPS relies on
the security of the underlying HTTP transport." I don't think it makes
sense to have each foo-over-http consider the baseline considerations
separately.

> different order? Something else?
one of the relevant things http means when it says it is stateless (which
iirc is the first sentence of 7230) is the requests themselves can also be
reordered by intermediaries or even routed onto different connections..
some may hit various caches and not even be placed on the connection.
sometimes the 'intermediary' is the library or javascript engine you are
using - DoH is meant to be expressable at the semantic layer of bodies and
headers. This particular point was raised in conjunction with dnsop's
session signal draft which requires a stateful DNS transport.. HTTPS is
stateless message based (despite being layered on TCP) and therefore we
want to be clear DoH won't work for building a session layer. (that draft
is coordinated).

> response (in wire format)'
I'll let paul answer that - he does it more concisely than I can manage :)

> >  Two primary uses cases were considered during this protocol's
thanks!

> Please introduce and expand de acronym "DoH" the first time you employ it.
ack

Page 7:
> thanks

> thanks

---

From: Star Brilliant <[email-masked]>
Date: Sat, 11 Aug 2018 22:58:46 +0000

Hello Femando and the mail list,

Thanks for your review!

I would like to answer some of the questions based on what the contributors had discussed before.

Paul Hoffman is the core author, so if I made a mistake, please consider his answer as the authoritative.

> Could the queries really be cached for the POST case?

Yes, they can.

>From RFC 7231: "In general, safe methods that do not depend on a current or authoritative response are defined as cacheable; this specification defines GET, HEAD, and POST as cacheable, although the overwhelming majority of cache implementations only support GET and HEAD."

But GET is friendlier than POST, because of the fact most ISP's transparent proxy or cloud WAF systems do not cache POST.

> You want DNSsec to be end-to-end.

We discussed about this paragraph before. The problem depends on how you deploy DoH services.

1) For most people's needs, the users grab a list of DoH servers from the Internet and configure their systems to use the DoH servers. These public servers are untrustworthy. So the users need to validate DNSSEC at their end.

2) A web-based application using DoH with a pre-configured server maintained by the application author. Then the server has the same trust level as the JavaScript code that validates the records. If anyone does not trust the server, they should not use the web app. In addition, by checking the records on the server, we can save some bandwidth for mobile users.

3) A dedicated enterprise cloud environment with hundreds of machines, serving trillions of requests per day. The administrator would want to offload the DNSSEC check to the gateway to reduce service latency, improve cache hit-rate, reduce intra-AS bandwidth cost, and to increase service quality.

4) A resource-constrained device (some Arduino or ESP8266 or other IoT-like things). You will want to delegate the check to your PC as long as the PC can be trusted.

Conclusion: People use DoH in different ways. Sometimes it is better to "give the user more choices, but make the default choice secure".
Therefore, we have the text "It is the choice of a client to either trust or distrust".

> different order? Something else?

The DNS wire protocol is not always a single query-and-response model. Some functions (e.g. zone transfer) require multiple messages to be sent and received. As already stated in the text elsewhere, DoH does not support this kind of functions (at least not in this version).

> response (in wire format)'

If you are asking about "an HTTP error", here is the explain:

1) Some errors indicates a problem happening on HTTP layer, not DNS layer.
For example, 302 Redirection, 404 Not Found, 401 Unauthorized

2) Sometimes it is technically impossible to reply with an encoded DNS response.
For example, the web server crashed, throwing a default 500-504 error page

DoH is a double-layered protocol. It is intuitive that each layer may break, resulting with an error that represents that layer. (Additionally, since DoH runs on TCP/IP, a DoH server can even reply with an icmp-port-unreachable error.) Layered error is the nature of Internet protocols.

If you are asking about the TC bit, this happens in many situations. One cause is broken TCP on the authoritative server, which happens quite frequently with anycast servers.

---

From: Fernando Gont <[email-masked]>
Date: Sun, 12 Aug 2018 00:48:21 +0200

> truncated, but it's equally valid to do TCP in the first place.

I'm aware of that. But with traditional DNS, you start with UDP, and may
fall back to TCP. DoS'ing the TCP support doesn't DoS the whole system.
(If it were the case that 100% of UDP-based resolutions result in a
subsequnt TCP-based resolution (which I'm not saying is not the case..I
just don't know), then I wouldn't have this concern)

> any diferent from existing DNS over TCP?

What could be different is the impact. What I'm wondering is, in a way,
to what extent it might be easier to completely DoS DNS resolution for
the DoH case. -- Defending against resource-exhaustion attacks is
certainly more complex for stateful protocols.

Thanks!

Cheers,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: Fernando Gont <[email-masked]>
Date: Sun, 12 Aug 2018 01:19:09 +0200

On 08/11/2018 09:20 PM, Patrick McManus wrote:
> my understanding is that giving the client the choice between doing
> DNSSEC validation and trusting the recursive to do so is standard DNS
> practice.

What I had in mind when making this comment is this: There seem to be
anycast resolvers implementing DoH (e.g., CLoudflare's and others, if I
understand correctly). Giving them the responsibility for DNSsec
validation would seem to me as giving them way too much power (i.e.,
what if they were compromised and/or went evil?)

> I disagree a bit. It adds a third TCP based DNS based approach to the
> DNS over TCP and DNS over TLS approaches already standardized so it is
> not breaking new ground in that sense.

To put it in a different way:

* What would be the impact if an attacker were to DoS TCP-based
resolution in the traditional case? (i.e., while UDP-based resolution
still works)

* What would be the impact if an attacker were to DoS (HTTPS/)TCP-based
resolution in the DoH case?

If the impact were the same, then just say that, and no need to worry.
But my understanding is that in current deployments you normally only
fall back to TCP when the UDP response was truncated, whereas with DoH,
you rely 100% on TCP.

>     https://www.gont.com.ar/papers/tn-03-09-security-assessment-TCP.pdf
>     <https://www.gont.com.ar/papers/tn-03-09-security-assessment-TCP.pdf>).
>     Defending
> security considerations tries to say this "Running DNS over HTTPS relies
> on the security of the underlying HTTP transport." I don't think it
> makes sense to have each foo-over-http consider the baseline
> considerations separately.

I don't mean you need to elaborate on how to defend it. JUst noted that
if you completely get rid of UDP-base resolution, then the TCP one (in
this case DoH) becomes even more critical, and the impact of a DoS
against TCP-based resolution is higher. And defending a stateful
protocol like TCP against resource exhaustion attacks is non-trivial.

> one of the relevant things http means when it says it is stateless
> (which iirc is the first sentence of 7230) is the requests themselves
> can also be reordered by intermediaries or even routed onto different
> connections.. some may hit various caches and not even be placed on the
> connection. sometimes the 'intermediary' is the library or javascript
> engine you are using - DoH is meant to be expressable at the semantic
> layer of bodies and headers. This particular point was raised in
> conjunction with dnsop's session signal draft which requires a stateful
> DNS transport.. HTTPS is stateless message based (despite being layered
> on TCP) and therefore we want to be clear DoH won't work for building a
> session layer. (that draft is coordinated).

Maybe I'm a bit confused. If, on the same connection, I send a request
to resolve "A" and subsequently a request to resolve "B", the result of
the later might come before than the result of the former? (I'm trying
to understand what the "ordering guarantees" that are not provided with
DoH).

Thanks!

Cheers,

---

From: Patrick McManus <[email-masked]>
Date: Sat, 11 Aug 2018 19:33:23 -0400

> certainly more complex for stateful protocols.
HTTPS services can scale to very large amounts and are quite successful at
dos mitigation - there are many existence proofs. That's actually a strong
reason to utilize DoH - to participate in the scalig that comes with that
ecosystem.

---

From: Fernando Gont <[email-masked]>
Date: Sun, 12 Aug 2018 01:34:38 +0200

Hello,

Thanks for the response. Inline....

> 1) For most people's needs, the users grab a list of DoH servers from the Internet and configure their systems to use the DoH servers. These public servers are untrustworthy. So the users need to v…

This is the scenario I was assuming, for which it seemed a bad idea to
ahve the DOH server do DNSsec validation. Anyway to clarify this so that
others do't fall into the same assumption?

> If you are asking about the TC bit, this happens in many situations. One cause is broken TCP on the authoritative server, which happens quite frequently with anycast servers.

My understanding of the quoted text above was that DoH might signal via
HTTP error codes some events that could also be signaled via DNS
responses.  -- whereas my expectation was for HTTP response codes to
only signal stuff related to HTTP stransport.

Thanks!

Cheers,

---

From: Star Brilliant <[email-masked]>
Date: Sat, 11 Aug 2018 23:35:05 +0000

Hello Fermando and the maillist,

I just found that I forgot to address one question in my previous mail. Here is the addition.

> the security considerations.

On contrast, by switching from UDP to TCP, the server is now able to defend attacks more *easily*.

1) TCP requires 3-way handshake before establishing the connection. This prevented simple DoS attack with spoofed source address since the attacker will not receive the 2nd packet.

2) In the past, the server started to allocate resources upon receiving the first SYN packet, making it vulnerable to SYN Flood attack. Now we use SYN Cookies [RFC 4987], so the server does not allocate resources until the 3-way handshake has finished, to mitigate the attack as long as the server's Internet pipe is not full.

3) UDP is vulnerable to UDP Amplification attack, that is to send a very small request, requiring kilobytes of response. Combined with spoofed source address (1), the attacker can make request and response packets bouncing between 2 servers, producing a 2^n amount of junk traffic, preventing the server from operating. Typical amplification victims include DNS and NTP, and they are all UDP .

4) Nowadays, the majority of DDoS attack is to send TB/s or PB/s of arbitrary garbage to fill the server's 100Mbps Internet pipe to make your server offline. Generally you need a powerful hardware firewall to wash out the garbage, and as many as BGP peers with other ISPs so legitimate users can reach your server directly in a clean pipe. For this type of attack, UDP and TCP have no difference.

5) We already have many articles talking about TCP/IP security [RFC 4953, 4987, 5961, 6528, etc]. I disagree that we need to talk about everything from TCP to IP to Ethernet in this DoH document.

---

From: Fernando Gont <[email-masked]>
Date: Sun, 12 Aug 2018 01:37:46 +0200

On 08/11/2018 05:40 PM, John R Levine wrote:
> Well, OK.  The obvious next question is whether DoH is different from
> any other https client request.

The possible difference is the impact: Traditionally, and https-based
attack could DoS a web server. With DoH, it could DoS DNS resolution.

Thanks,

---

From: Patrick McManus <[email-masked]>
Date: Sat, 11 Aug 2018 19:39:34 -0400

yes - as section 10 says "

 Running DNS over HTTPS relies on the security of the underlying HTTP
   transport

"

---

From: Paul Hoffman <[email-masked]>
Date: Sat, 11 Aug 2018 23:44:30 +0000

> response (in wire format)'

There was a lot of discussion in the WG about this. The result was similar to what Star replied: let the semantics of the inner protocol (DNS) retain their exact meanings, and let the semantics of the outer layer (HTTP) retain their exact meaning. A DoH client is expected to understand DNS errors *and* be a real HTTP client that can deal with normal HTTP semantics.

> what if they were compromised and/or went evil?)

DoH clients are no different in this regard than normal DNS (port 53) or DNS-over-TLS clients. Any of the three types of DNS client can do its own DNS validation by asking for the supporting DNS records; DoH by design doesn't change this at all. But to be clear, the ratio of end clients that do their own validation is still incredibly small.

> fall back to TCP when the UDP response was truncated, whereas with DoH,

DoH servers do not need to reside on the same hosts as DNS/UDP or DNS/TCP servers. Cloudflare happens to do so, possibly due to address cuteness, but there is nothing in the protocol that says they should. The impact on DoH server on attacks is identical to the impact on attacks on normal TLS-based web services.

> protocol like TCP against resource exhaustion attacks is non-trivial.

DoH is never expected to "get rid of UDP-based resolution". The use cases for DoH are very much a subset of the use cases for general DNS resolution, and that will be UDP-based for the foreseeable future.

> to understand what the "ordering guarantees" that are not provided with

Yep! This is why the DoH WG has been such a good learning experience for DNS folks about the modern HTTP world, and for HTTP folks for the modern DNS world. We're not all holding hands and singing kumbaya yet, but at least we understand better the weirdnesses that the others take for granted.

--Paul Hoffman

---

From: "John R Levine" <[email-masked]>
Date: 11 Aug 2018 21:25:52 -0400

> just don't know), then I wouldn't have this concern)

With DNSSEC, the results are large enough that a whole lot of the traffic 
is over TCP, particularly on IPv6 where packet fragmentation doesn't work 
reliably.  And in any event, as someone else pointed out, https is hardly 
an untried protocol in hostile enviroments.

Regards,

---

From: Mark Andrews <[email-masked]>
Date: Mon, 13 Aug 2018 09:42:42 +1000

> On 12 Aug 2018, at 9:35 am, Star Brilliant <[email-masked]> wrote:
> 

Which also exists for DNS over UDP using DNS COOKIE.

> 2) In the past, the server started to allocate resources upon receiving the first SYN packet, making it vulnerable to SYN Flood attack. Now we use SYN Cookies [RFC 4987], so the server does not all…

DNS COOKIE doesn’t have any server state.

> 3) UDP is vulnerable to UDP Amplification attack, that is to send a very small request, requiring kilobytes of response. Combined with spoofed source address (1), the attacker can make request and…

Not with DNS COOKIE.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: Fernando Gont <[email-masked]>
Date: Mon, 13 Aug 2018 22:33:55 +0200

>  Running DNS over HTTPS relies on the security of the underlying HTTP

Giving that DoH means that you ow rely 100% on a stateful protocol, some
further discussion is needed.

For instance, if you need to harden HTTPS to better cope with DoS
attacks, part of the hardening happens at the layer below. You don't
need to get into those details, but I do think you need to stress that
relying on a connection oriented protocol implies taking care of a bunch
of thing, many very closely associated, so that the service is resilient
to DoS attacks.

Thanks,

---

From: Fernando Gont <[email-masked]>
Date: Mon, 13 Aug 2018 22:54:09 +0200

On 08/12/2018 03:25 AM, John R Levine wrote:
>> I'm aware of that. But with traditional DNS, you start with UDP, and
>> may fall back to TCP. DoS'ing the TCP support doesn't DoS the whole
>> system. (If it were the case that 100% of UDP-based resolutions result
>> in a subsequnt TCP-based resolution (which I'm not saying is not the
>> case..I just don't know), then I wouldn't have this concern)
> 
> With DNSSEC, the results are large enough that a whole lot of the
> traffic is over TCP, 

A whole lot != all. In any case, are there measurements about this?

> particularly on IPv6 where packet fragmentation
> doesn't work reliably.  And in any event, as someone else pointed out,
> https is hardly an untried protocol in hostile enviroments.

A lt of what you need to do to make a TCP-based protocol (such as HTTPS)
to be resilient to DoS doesn't have much to do with the app protocol itself.

Yes, there are lots of folks running TCP-based services in hostile
environments. And if they are to be resilient to Do attacks, they have
to take care of a lot of details. With UDP, you can get away with not
taking care about a lot of them (because it's stateless). That was
exactly my point.