Subject: snarls in real life

From: Michael Thomas <[email-masked]>
Date: Tue, 20 Apr 2021 18:31:50 -0700

So against my better judgement, I posted my Quic, the Elephant in the 
Room post to the Quic list. Within 24 hours one of the snarling working 
group chairs declared it off topic and demanded it stop. This is what I 
mean about grotesque fiefdoms where the working groups are so fricking 
insular that any commentary outside of their blinders is not tolerated. 
The thought that no working group business can take place in the face of 
anything other than something directly relatable working group business 
is a complete piece of idiocy and shuts down anything they don't want to 
hear. People are capable of doing all of this all at once.

Mike

---

From: Keith Moore <[email-masked]>
Date: Tue, 20 Apr 2021 21:51:23 -0400

On 4/20/21 9:31 PM, Michael Thomas wrote:

> Room post to the Quic list. Within 24 hours one of the snarling 
> working group chairs declared it off topic and demanded it stop. This 
> is what I mean about grotesque fiefdoms where the working groups are 
> so fricking insular that any commentary outside of their blinders is 
> not tolerated. The thought that no working group business can take 
> place in the face of anything other than something directly relatable 
> working group business is a complete piece of idiocy and shuts down 
> anything they don't want to hear. People are capable of doing all of 
> this all at once.

References, please.

Keith

---

From: "Bron Gondwana" <[email-masked]>
Date: Wed, 21 Apr 2021 12:46:04 +1000

> hear. People are capable of doing all of this all at once.

Oh cool - thanks for posting that.  It's a very instructive thread.

Some general thoughts:
 * it's not enough to be technically correct (disclaimer: I'm not well enough versed in this area to know if you are), it's also important to do the work to socialise your idea and persuade others so that they become evangelists for you as well.
 * if you require the assistance of others to run experiments for you, it's wise not to piss them off
 * there's a whole fallacy somewhere which I've had to address a few times already in my own working groups, but which I still commonly see, along the lines of "big companies have masses of resources and hence can easily run experiments or implement arbitrary ideas - and have an obligation to do so when requested/demanded".  They don't, you have to persuade them just as much as anyone else, plus they're slower to move and harder to persuade.
I'm no expert on QUIC or the technical merits of your proposal, but I doubt the effectiveness of the strategy you employed in that email in getting anything other than the response that was forthcoming.

Apropos of my recent response to a question from Jay over in the other thread - prodding others until they are triggered to annoyance and pointing at their responses to say "look, I've found badness" is a fine art for sure.

Keith asked for citation - I dug it up because I wanted to see the context too - so here's the link:

https://mailarchive.ietf.org/arch/msg/quic/e7ChoIKSopdnn4NcXwOAalTcdIY/

Everyone can make their own judgement.  Myself, I don't come out of that thread persuaded to become an evangelist for your approach.

Regards,

Bron.
--
  Bron Gondwana, CEO, Fastmail Pty Ltd
  [email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 21 Apr 2021 17:26:19 +1200

> I'm no expert on QUIC or the technical merits of your proposal, but I doubt the effectiveness of the strategy you employed in that email in getting anything other than the response that was forthco…

Every WG Chair, especially for a WG that has tight focus on a specific goal, meets the problem of potentially off-charter discussions that potentially hamper progress.

Personally, ideally, I would hope that I'd raise the question whether something is off-charter before I sent a cease and desist letter. If something is agreed to be off-charter, there is no doubt that the WG chairs can request people to take it elsewhere. It happens often.

I have no opinion about whether the discussion in question is on-charter. From a quick glance at the archive, that doesn't seem to have been discussed prior to the chair's message. Going straight to the cease and desist stage seems abrupt, to say the least.
(Unless there was preceding off-list mail, perhaps.)

Regards
    Brian

---

From: Michael Thomas <[email-masked]>
Date: Wed, 21 Apr 2021 06:43:00 -0700

> (Unless there was preceding off-list mail, perhaps.)

The thing that struck me is that one of the assumedly Google people said 
that signing their zone is tantamount to "boiling the ocean". That is a 
stunning admission, and not a good one.

Mike, there was nothing off list. straight to FO by the chair. and 
people wonder why nobody wants to participate

---

From: Michael Thomas <[email-masked]>
Date: Wed, 21 Apr 2021 09:31:49 -0700

On 4/20/21 7:46 PM, Bron Gondwana wrote:
>   * it's not enough to be technically correct (disclaimer: I'm not
>     well enough versed in this area to know if you are), it's also
>     important to do the work to socialise your idea and persuade
>     others so that they become evangelists for you as well.
>   * there's a whole fallacy somewhere which I've had to address a few
>     times already in my own working groups, but which I still commonly
>     see, along the lines of "big companies have masses of resources
>     and hence can easily run experiments or implement arbitrary ideas
>     - and have an obligation to do so when requested/demanded".  They
>     don't, you have to persuade them just as much as anyone else, plus
>     they're slower to move and harder to persuade.
>
I wasn't making any such assumption, just pointing out that it was well 
within the capability of a Google-like company to run an experiment. 
Instead I got told that signing their zone is apparently "boiling the 
ocean" which to me is astonishing. If you take that at face value, that 
is a stunning indictment of DNSSec.

Chrome already did the DANE work once upon a time so DNSSec is the only 
missing piece. But the very thought that the number of packets exchanged 
in a transport protocol's setup is *off topic* within 24 hours and a few 
messages back and forth speaks miles about how broken many working 
groups are and why nobody wants to participate.

Mike

---

From: Michael Thomas <[email-masked]>
Date: Wed, 21 Apr 2021 09:47:52 -0700

> Every WG Chair, especially for a WG that has tight focus on a specific goal, meets the problem of potentially off-charter discussions that potentially hamper progress.

This is an often repeated refrain, but I question its actual impact. 
People reply to things because they have an opinion and are interested 
enough in the subject to make time for it. Posing this as a zero sum 
game is a fallacy: if they aren't interested enough at the business at 
hand they simply won't reply at all. At some point things become noise, 
but a few messages back and forth is hardly that point. In fact, 
slightly off topic posts can bring useful insight to working groups, and 
can be extremely useful for outsiders to understand what is going on. I 
know of no other way to get that understanding other than simply asking 
on the working group list. Do you?

That and to be told by a working group chair *in their capacity* to go 
away (and away to where? no answer) is dysfunctional, and sends a clear 
message that interlopers will not be tolerated.

Mike

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 21 Apr 2021 13:06:39 -0400

Michael, I hav eno idea whether your work was or was not appropriate for 
the QUIC working group.

Arguing that working group chairs an not declare things out of scope 
weakens your overall argument.  Working groups have charters.  The 
chairs are expected to work within the charter.  Working group email 
lists are for discussion of topics relevant to the working group.

Now, I will grant that charters are not precise.  Topics are not 
precise.  And chairs should (and generally do) err on the side of 
allowing some latitude.
But claiming that chairs can not rule things as being out of scope does 
not match our process, and would hamper our work.  (Who has, as WG 
co-chair, suggested to more than one author team that the independent 
stream would make a better home for their work, as it does not fit the 
charter of the working group.)

Yours,
Joel

On 4/21/2021 12:47 PM, Michael Thomas wrote:
>> Every WG Chair, especially for a WG that has tight focus on a specific 
>> goal, meets the problem of potentially off-charter discussions that 
>> potentially hamper progress.
>

---

From: Christian Huitema <[email-masked]>
Date: Wed, 21 Apr 2021 10:08:30 -0700

On 4/21/2021 9:31 AM, Michael Thomas wrote:

>
> Chrome already did the DANE work once upon a time so DNSSec is the 
> only missing piece. But the very thought that the number of packets 
> exchanged in a transport protocol's setup is *off topic* within 24 
> hours and a few messages back and forth speaks miles about how broken 
> many working groups are and why nobody wants to participate.

My takeaway from these exchanges is a bit different. You are advocating 
for using Dane instead of PKI during the authentication exchange, 
because this leads to fewer packets. People provided three different 
counter arguments. The first argument was that in first order, 
performance is measured by the number of round-trips, not the number of 
packets, and that using Dane instead of PKI would not result in big 
performance gains in practice. The second argument was that the full 
authentication exchange is only used in a small fraction of connections. 
The other exchanges use session resumption, and in that case there is no 
difference between Dane and PKI. The third argument was that there is no 
specific work to do in the QUIC working group on this topic, since QUIC 
relies on TLS 1.3 for authentication and TLS 1.3 already supports Dane. 
Using Dane instead of PKI is a deployment issue, not a protocol 
development issue, and there is no concrete work for the QUIC WG.

-- Christian Huitema

---

From: Keith Moore <[email-masked]>
Date: Wed, 21 Apr 2021 13:20:17 -0400

On 4/21/21 1:06 PM, Joel M. Halpern wrote:

> Michael, I hav eno idea whether your work was or was not appropriate 
> for the QUIC working group.
>
> But claiming that chairs can not rule things as being out of scope 
> does not match our process, and would hamper our work.  (Who has, as 
> WG co-chair, suggested to more than one author team that the 
> independent stream would make a better home for their work, as it does 
> not fit the charter of the working group.) 

Concur with Joel on the above.

Also, if you believe that a WG chair is unfairly excluding contributions 
that are within the WG's charter, that seems like grounds for an appeal, 
and differences of opinion about such judgments are one of the reasons 
that we have an appeals process.

In other words, if a dispute can be handled by existing process, trying 
to let that process work might be the best thing to do. If you try the 
process and it doesn't work for whatever reason, maybe there's a bigger 
problem there.

But no matter what the process, some bad decisions, and some disputes, 
are both inevitable.   As long as there is some way to appeal such 
decisions, the fact that chairs sometimes make what seem like bad 
decisions about what's in scope for a group is not by itself an 
indication of a structural flaw.

Keith

---

From: Michael Thomas <[email-masked]>
Date: Wed, 21 Apr 2021 10:35:12 -0700

On 4/21/21 10:20 AM, Keith Moore wrote:
>> independent stream would make a better home for their work, as it 
>> does not fit the charter of the working group.) 
>
> Also, if you believe that a WG chair is unfairly excluding 
> contributions that are within the WG's charter, that seems like 
> grounds for an appeal, and differences of opinion about such judgments 
> are one of the reasons that we have an appeals process.

My overarching point here is that it was entirely predicable. To get 
shut down unceremoniously after a half dozen messages and less than a 
day... really? That impeded working group progress? As I said before, 
there are no other venues to go to to get answers to basic questions. 
Are working groups so fragile that they can't have any meta conversations?

>
> In other words, if a dispute can be handled by existing process, 
> trying to let that process work might be the best thing to do. If you 
> try the process and it doesn't work for whatever reason, maybe there's 
> a bigger problem there.

Sorry, I have no interest in "appealing" or chasing other process. The 
point here isn't to "win".

Mike

---

From: Michael Thomas <[email-masked]>
Date: Wed, 21 Apr 2021 10:49:56 -0700

On 4/21/21 10:08 AM, Christian Huitema wrote:
> My takeaway from these exchanges is a bit different. You are 
> advocating for using Dane instead of PKI during the authentication 
> exchange, because this leads to fewer packets. People provided three 
> different counter arguments. The first argument was that in first 
> order, performance is measured by the number of round-trips, not the 
> number of packets, and that using Dane instead of PKI would not result 
> in big performance gains in practice. The second argument was that the 
> full authentication exchange is only used in a small fraction of 
> connections. The other exchanges use session resumption, and in that 
> case there is no difference between Dane and PKI. The third argument 
> was that there is no specific work to do in the QUIC working group on 
> this topic, since QUIC relies on TLS 1.3 for authentication and TLS 
> 1.3 already supports Dane. Using Dane instead of PKI is a deployment 
> issue, not a protocol development issue, and there is no concrete work 
> for the QUIC WG.
>
The meta question is whether that is so off topic that it needs to be 
officially shut down with the working group chairs. The technical merits 
are what they are. What I was told in no uncertain terms is that I am 
not allowed to even ask the question. Is that appropriate?

Mike

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 21 Apr 2021 13:56:02 -0400

+1

The issue here is not just the charter. QUIC is a WG with over 100 active
participants. It has far greater need to keep focus than others.

In particular, when a WG chair asks 'why are you bringing this here when it
is really not QUIC specific', it is important to have a better answer than:

"I'm not asking this working group to do anything. Just socializing
something that generated a lot of discussion on the IETF list that might
be of interest to the Quic community."

The problem I have with your approach is that when people raised issues,
you didn't try to address them, you just told them they weren't issues.

One of the things I really like about the new RFC/ID format is that we can
now have diagrams showing the impact of a proposal. Turning a two party
protocol (initiator/responder) into a three party protocol (adding a DNS
resolver) to shave a couple of packets doesn't look like a win to me. In
fact it looks like a de-optimization if the information needed isn't
already in-cache.

Managing the number of Web servers Google has facing the world is hard. Its
the sort of infrastructure that when scoping a possible acquisition, I
would always have to budget several million dollars for a ground up
replacement simply because such infrastructures must by their very nature
rely on a great depth of local arcana that will walk out of the door the
day the people carrying it in their heads walk out the door. I would
certainly never presume to tell the operators of such systems that moving
from the WebPKI to DANE is easy.

Google already has free WebPKI certs for itself through GTS and for
everyone else through LetsEncrypt. So the value prop for DANE is limited to
shaving packets. And that requires people to sign their Zones which is
expensive because someone is going to have to crack open a manual and read
it. And that costs real money when folk are billing at $300-$500/hr.

The whole problem with DANE start to finish was that it was DNSSEC people
looking for a problem to solve. And they chose the wrong problem. The only
good reason to deploy DNSSEC is to publish security policy. And that isn't
viable unless you have

1) A means of expressing the security policy

2) A means of automating publication of the security policy to the DNS zone

3) A complete overhaul of the DNS client-resolver protocol to make THAT
efficient.

SMTP over TLS is not a useful paradigm because email is store and forward
and we don't care about latency.

The DNS world keeps shooting itself in the foot here and then complaining
nobody is taking them seriously. TLSA is too limited as a means of
describing security policy. RFC 6763 is the approach to build on as I show
in:

draft-hallambaker-web-service-discovery-05 - DNS Web Service Discovery
(ietf.org)
<https://tools.ietf.org/html/draft-hallambaker-web-service-discovery-05>

I do not have a mechanism for automating publication of the security policy
to the DNS but that is going to be one of the things the Mesh eventually
does for free as devices are connected to a personal Mesh.

DPRIV is another example of DNS world refusing to see beyond its limited
horizons and then complaining nobody is using their stuff. It was
abundantly clear that we needed to fix the broken DNS client resolver
protocol if there was going to be any takeup of DPRIV. Instead we were told
we must do this in a year so the only thing we can look at is DNS over TLS
which will require TCP fast start to be sufficiently performant.

I don't just suspect my work was sabotaged under BULLRUN, I know because
one of the people responsible apologized to me personally. I am pretty sure
that DPRIV and DANE were among the proposals deliberately sabotaged so as
to delay deployment of cryptographic systems that worked.

It is so easy for a whispering campaign to persuade people that the 'evil
CAs' are the ones only working for themselves. So ignore the proposals they
have been working on and circulating for five years now, schemes that don't
require a kernel change and instead do this thing someone just scribbled on
the back of a paper napkin in crayon.

That is all in the past now. We all have a job to do and it is to secure
the net. And we can't do that by thinking small. And we certainly can't do
that if we don't listen to and try to understand the stakeholders who have
to deploy if it is going to work.

---

From: Keith Moore <[email-masked]>
Date: Wed, 21 Apr 2021 13:56:37 -0400

On 4/21/21 1:35 PM, Michael Thomas wrote:

>> try the process and it doesn't work for whatever reason, maybe 
>> there's a bigger problem there.
>

I guess I thought the point was to make IETF less likely to discourage 
participation.   But in this case, I find myself asking, how could IETF 
be different?   There will always be some input that is distracting 
enough to a WG (however that's defined) that the chairs see a need to 
rule it out-of-scope.   You seem to be saying that the chairs don't need 
to be quite so strict. Well, perhaps, but it's always going to be a 
judgment call on the chairs' part as to what will derail a WG, just as 
it's a judgment call on the part of a person who believes they've been 
unfairly ruled out-of-scope to decide whether to appeal it.

So basically it sounds like you're asking for a culture change rather 
than a process change.   I guess in my experience most IETF WG chairs 
are willing to tolerate quite a bit of deviation from a group's charter, 
at least for a while, and some participants would probably prefer a bit 
less tolerance.   This will necessarily vary from one WG to another, but 
sometimes a tight focus is absolutely necessary.

Things I find myself wondering: Were you treated respectfully even 
though the chairs didn't want to discuss your draft in the group?   Did 
anyone suggest alternative ways of building support for your proposal?

Bottom line: Anyone who wants to promote an idea within IETF has to find 
some way of building consensus around that idea.   And any time there's 
inertia opposing that idea, building that consensus isn't likely to be 
easy, even if it's a very good idea.    It's still the job of the 
proponents of that idea to try to build consensus; nobody else is going 
to do it for them.

Keith

---

From: Keith Moore <[email-masked]>
Date: Wed, 21 Apr 2021 13:59:10 -0400

On 4/21/21 1:49 PM, Michael Thomas wrote:

>> number of packets, and that using Dane instead of PKI would not 
>> result in big performance gains in practice. The second argument was 
>> that the full authentication exchange is only used in a small 
>> fraction of connections. The other exchanges use session resumption, 
>> and in that case there is no difference between Dane and PKI. The 
>> third argument was that there is no specific work to do in the QUIC 
>> working group on this topic, since QUIC relies on TLS 1.3 for 
>> authentication and TLS 1.3 already supports Dane. Using Dane instead 
>> of PKI is a deployment issue, not a protocol development issue, and 
>> there is no concrete work for the QUIC WG.
>>
> officially shut down with the working group chairs. The technical 
> merits are what they are. What I was told in no uncertain terms is 
> that I am not allowed to even ask the question. Is that appropriate? 

Why are you asking us, when it's really the responsible ADs' job to 
determine the answer to that question?     There's not one right generic 
answer to that kind of question.   The answer to that question requires 
evaluating the technical merits of the idea with respect to the WG's 
charter, which is a lot of work for anyone who isn't already "in the 
loop" to evaluate.

Keith

---

From: Christian Huitema <[email-masked]>
Date: Wed, 21 Apr 2021 11:03:53 -0700

On 4/21/2021 10:49 AM, Michael Thomas wrote:
>>> hours and a few messages back and forth speaks miles about how 
>>> broken many working groups are and why nobody wants to participate.
>>
>>

There are a couple of topics that would be clearly appropriate for the 
QUIC working group. A document describing your experience deploying 
QUIC+DANE, for example, would be on topic. If there are issue preventing 
mutually agreeing clients and servers from using QUIC and DANE, that too 
would be very much on topic. On the other hand, your latter posts 
focused on the development of the Chrome browser, its level of support 
for DANE, and Google's willingness to deploy DNSSEC in their domains. 
That very much off topic for the  QUIC WG.

-- Christian Huitema

---

From: Michael Thomas <[email-masked]>
Date: Wed, 21 Apr 2021 11:21:00 -0700

On 4/21/21 11:03 AM, Christian Huitema wrote:
> QUIC+DANE, for example, would be on topic. If there are issue 
> preventing mutually agreeing clients and servers from using QUIC and 
> DANE, that too would be very much on topic. On the other hand, your 
> latter posts focused on the development of the Chrome browser, its 
> level of support for DANE, and Google's willingness to deploy DNSSEC 
> in their domains. That very much off topic for the  QUIC WG.
>
The reason I wrote my post is precisely because I *don't* have the 
resources to do such an experiment, and even if I had all of the code 
and signed domain my experience would be anecdotal at best. Only one of 
the big browser vendors could meaningfully run such an experiment. The 
overarching problem here though is *where* is the appropriate venue to 
ask questions or make observations? I got told to go elsewhere, well 
where is this "elsewhere" precisely?

That said, one of the interesting things coming out of this is that 
maybe there really are some fundamental issues surrounding DNSSec 
deployment. Why have none of the browser vendors signed their zones?

Mike

---

From: Michael Thomas <[email-masked]>
Date: Wed, 21 Apr 2021 11:35:41 -0700

On 4/21/21 10:56 AM, Keith Moore wrote:

> Bottom line: Anyone who wants to promote an idea within IETF has to 
> find some way of building consensus around that idea.   And any time 
> there's inertia opposing that idea, building that consensus isn't 
> likely to be easy, even if it's a very good idea.    It's still the 
> job of the proponents of that idea to try to build consensus; nobody 
> else is going to do it for them.

How can you "build consensus" when you aren't allowed to even talk about 
it? I don't go to meetings. I don't want to go to meetings. Does that 
mean I am excluded from "building consensus"? Are there other ways to 
"build consensus" that I'm not aware of? I'm not even trying to "build 
consensus", per se. I'm just trying to get people to consider whether an 
alternative might be better and if not, why not.

Does that "hamper progress"? Especially after about a half dozen 
messages and less than 24 hours?

But this list doesn't seem to be the answer because it's mostly about 
ietf process and the right technical people are definitionally going to 
be hit and miss.

Mike

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 22 Apr 2021 08:50:36 +1200

On 22-Apr-21 06:35, Michael Thomas wrote:
> mean I am excluded from "building consensus"? 

No, but (coupled with the move to on-line meetings with all their
sociological disadvantages) it really makes it much, much harder.

> Are there other ways to 

Traditionally this list right here is where you would do that. Or if you
scope it as a Transport Area problem, [email-masked] ("A mailing list
for discussions on topics pertinent to the IETF Transport and Services
Area (TSV)").

> messages and less than 24 hours?

As I said, it seemed a bit abrupt but it is within a WG Chair's remit.
> ietf process and the right technical people are definitionally going to 

That is indeed this list's current problem; hence, tsv-area might be
a better bet.

Regards
   Brian

---

From: Larry Masinter <[email-masked]>
Date: Wed, 21 Apr 2021 15:45:18 -0700

The best way to build consensus is one person at a time. 
Get one other person to agree with you.
The rest will be easier.
If you can't, it should tell you that you're in the rough.
--
https://LarryMasinter.net https://interlisp.org

---

From: "Bron Gondwana" <[email-masked]>
Date: Thu, 22 Apr 2021 08:57:12 +1000

On Thu, Apr 22, 2021, at 02:31, Michael Thomas wrote:

>> there's a whole fallacy somewhere which I've had to address a few times already in my own working groups, but which I still commonly see, along the lines of "big companies have masses of resources and hence can easily run experiments or implement arbitrary ideas - and have an obligation to do so when requested/demanded".  They don't, you have to persuade them just as much as anyone else, plus they're slower to move and harder to persuade.
> I wasn't making any such assumption, just pointing out that it was well within the capability of a Google-like company to run an experiment. 

Potato, potato.  You're assuming they have the capability based on their size, not on knowledge of how the teams are structured or what their incentives are, which is my point - why would they want to do this for you?  Why are you entitled to their time and their effort to run the experiment for you?
 * Because Google is big?
 * Because your technical argument is compelling?
 * Some other reason that I missed?

> Instead I got told that signing their zone is apparently "boiling the ocean" which to me is astonishing. If you take that at face value, that is a stunning indictment of DNSSec. 

It sounds like a conversation that's worth having with those who have tried DNSSec and given up, or decided not to try to use it.  The fact that you're astonished by their response is a good clue that maybe there's something interesting here.

Personally, I'm right in there with indictments on DNSSec in general.  I didn't write this, but I stand by it:

https://fastmail.blog/2016/12/20/dnssec-dane/

Rob wrote "DNSSEC is fragile and easy to get wrong in subtle ways."  I say that DNSSEC is operational poison - it's hard to get right, easy to get wrong, and most importantly hard to debug failures when it happens - your users aren't going to be able to report the cause.  It's theoretically good tech, but it clearly isn't getting traction and berating those who choose not to use it doesn't help.

> Chrome already did the DANE work once upon a time so DNSSec is the only missing piece. But the very thought that the number of packets exchanged in a transport protocol's setup is *off topic* within 24 hours and a few messages back and forth speaks miles about how broken many working groups are and why nobody wants to participate. 

You seem very focused on this "number of packets" idea.  Lower number of packets is nice, but only if the tradeoffs are worth it.  Discarding all the existing progress to restart from scratch with a new design is a pretty high cost and clearly those working on QUIC aren't that keen.  I didn't see anybody jump in that thread and say "hey, I'd also like to discuss this".

And if it makes everything more fragile, that's a downside too.  DNSSEC makes things fragile based on the number of big name sites that screw it up every year.  It also makes it much more expensive - DNSSEC is hard to get right and hard to keep right.

Bron.

---

From: Keith Moore <[email-masked]>
Date: Wed, 21 Apr 2021 19:44:41 -0400

On 4/21/21 6:57 PM, Bron Gondwana wrote:

> Rob wrote "DNSSEC is fragile and easy to get wrong in subtle ways."  I 
> say that DNSSEC is operational poison - it's hard to get right, easy 
> to get wrong, and most importantly hard to debug failures when it 
> happens - your users aren't going to be able to report the cause.  
> It's theoretically good tech, but it clearly isn't getting traction 
> and berating those who choose not to use it doesn't help.
> [...]
>
> And if it makes everything more fragile, that's a downside too.  
> DNSSEC makes things fragile based on the number of big name sites that 
> screw it up every year.  It also makes it much more expensive - DNSSEC 
> is hard to get right and hard to keep right.

I suspect that work invested in getting this fixed (whether that means 
tweaking DNSSEC, better tools, or both) would be a lot more useful (and 
satisfying) than a great many of the ideas that get kicked around on the 
ietf@ list.

Keith

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 21 Apr 2021 19:47:24 -0400

> and berating those who choose not to use it doesn't help.

This rehashing of stale and outdated strawman DNSSEC-bashing is neither
necessary nor productive.  Your critique of the assumptions about Google
stands on its merits, without needing to disparage the technical
specifics.

The tools for managing DNSSEC reliably have gotten substantially better
over the years.  Indeed Google has signed a number of its own domains,
including the .goog TLD and hundreds of thousands of customer domains
they're DNS hosting.  Google are having no problems running one of the
largest DNSSEC operations on the planet.  What they have not done, for
various reasons that are not relevant here is sign google.com or
gmail.com, ...  I can make potentially plausible guess as to why, but
they're not relevant.

> screw it up every year.

Microsoft just had a recent DNS failure without DNSSEC, the major cloud
services have had intermittent outages also unrelated to DNSSEC.  I've
seen no evidence that DNSSEC is particularly more fragile than other
technologies we operate.

> It also makes it much more expensive - DNSSEC is hard to get right and
> hard to keep right.

This is no longer true.  The tools for reliable automated signing and
monitoring thereof have improved substantially.  Only naive seat of
the pants deployments with no monitoring are more fragile.

My domain has been signed since 2014 without any disruptions, with just
a modest monitoring script that has alerted me to pendign expiration
(automated re-signing wasn't kicking in) a couple of times, well before
the signatures expired.  The bugs that resulted in resigning not
happening have been fixed for some time, and I don't have to expend any
energy to keep DNSSEC running, it just works.

-- 
    Viktor.

---

From: "Bron Gondwana" <[email-masked]>
Date: Thu, 22 Apr 2021 10:00:58 +1000

> energy to keep DNSSEC running, it just works.

That's you - you're an expert in this field.  Most people aren't.  And yet - as you mention, you had a bug with automated re-signing failing and had to add monitoring.

Also, I suspect that the content of your zone is managed by... you.

Extrapolating from that to assume that everyone else in the world will have the same experience... maybe the tooling has become heaps better than when we looked in 2016, but the list of DNSSEC failures hasn't exactly trickled to zero - cdc.gov in the year 2021 being a nice example case:

https://mailman.nanog.org/pipermail/nanog/2021-January/211507.html

Bron.

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 21 Apr 2021 20:23:52 -0400

> On Apr 21, 2021, at 8:00 PM, Bron Gondwana <[email-masked]> wrote:
> 
> That's you - you're an expert in this field.  Most people aren't.  And yet - as you mention,
> you had a bug with automated re-signing failing and had to add monitoring.

No, I did not have to add monitoring, the monitoring came first!
If it isn't monitored, it is not a critical service.  I never had
an outage, just had to fix some long ago resolved bugs.

> Also, I suspect that the content of your zone is managed by... you.

The zone content is largely irrelevant for signing, DNSSEC signing
just covers whatever is found in the zone.

> Extrapolating from that to assume that everyone else in the world will have
> the same experience... maybe the tooling has become heaps better than when
> we looked in 2016, but the list of DNSSEC failures hasn't exactly trickled
> to zero - cdc.gov in the year 2021 being a nice example case.

The tool has gotten heaps better, but some folks haven't upgraded, and have
poor operational discipline.  They likely have many other failures that
people just don't write about, because they're less popular punching bags
than DNSSEC.  It takes just a grain of competence and attention to detail,
as with any production technology.  What doesn't work is neglect, and that
also goes for unsigned DNS, with parent NS records going stale, replication
failing, ...

I am suggesting that Google can easily do DNSSEC for google.com, they likely
face non-trivial adoption barriers with global DNSSEC load-balancing, and
other specialised tech.  I am just saying the old excuses are tired out, we
can and should move on.

-- 
	Viktor.

---

From: "Bron Gondwana" <[email-masked]>
Date: Thu, 22 Apr 2021 10:59:43 +1000

> just covers whatever is found in the zone.

This assumes that the zone is a static, serialisable beast which doesn't have different content depending on whether you look at it funny (or from a different geoip region, and that there's no eventual consistency happening underneath)

> other specialised tech.  I am just saying the old excuses are tired out, we

I'm liking the "actually, we are the protocol police" more and more.  If people aren't compelled to implement something, then it's not offering them enough value.  But you know - maybe if they say "we don't want to because reasons", telling htem "your excuses are tired" won't persuade.  We can only really move on if we bring the world with us.

Bron.

---

From: [email-masked]
Date: Wed, 21 Apr 2021 17:10:16 -0700 (PDT)

> That's you - you're an expert in this field.  Most people aren't.  And yet -
> as you mention, you had a bug with automated re-signing failing and had to add
> monitoring.

Once the DNSSEC support in Bind got to the point where it was able to handle
most (but not all) expiration issues automatically, I decided to give DNSSEC a
try.

It's simple enough to configure it for basic domains, but I have a split-horizon
setup, and that complicated things quite a bit.

Another problem was that a long time back I took advantage of an absurdly low
promotional price from a registrar that did (and AFAIK still doesn't) support
DNSSEC. I opted to eat the cost for a couple of domains and migrated to a
registrar that did (always a royal pain). I finally migrated the last of them
mid-2020.

However, in the process I managed to make a cut-and-paste error and ended up
with one valid and one invalid DS record for one of my infrequently-used
domains. Which was noticed by exactly nothing, including the DNSSEC testing tool
I happened to use to validate my setup.

Some time later I got a problem report from IANA. It seems that IANA has DNSSEC
checking enabled on mail server, I assume because they have enabled DANE
(although this was never 100% clear), and I happened to have used this domain
for an internal list forwarding address. Unfortunately the error message IANA
was seeing was misleading, so it took a bit of time to track down. 

And I still have work to do because Bind has added a number of options that I
need to set, but before that can happen I need to upgrade Bind, and before that
can happen I need to upgrade some other stuff.

All this is for a total of 9 domains - a toy setup by any measure.

I know a little (but not a lot) about the DNS infrastructure at Oracle, and the
cost and complexity of migrating it to DNSSEC would be staggering.

				Ned

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 22 Apr 2021 00:09:48 -0400

> setup, and that complicated things quite a bit.

Sure.  It is often simpler to avoid views and just operate separate
zones, even if some IP addresses appear in both zones.  But views
do have their uses.  What versio of BIND are using?  Significant
further improvements in DNSSEC support have been made in 9.16.

> However, in the process I managed to make a cut-and-paste error and
> ended up with one valid and one invalid DS record for one of my
> infrequently-used domains. Which was noticed by exactly nothing,
> including the DNSSEC testing tool I happened to use to validate my
> setup.

Then you need better monitoring scripts, because DS records should
always match an extant DNSKEY, proper operational procedure is to first
retire the DS record, and only later the corresponding key.  With fully
automated signing, and as registries begin to roll out CDS/CDNSKEY
support, this sort of mistake will become less common.  But it is easy
to check for.

> for an internal list forwarding address.

I wouldn't expect a DANE mail server to notice a partial DS RRset
mismatch, validating resolvers continue to work so long as the
other DS record is sound.

> And I still have work to do because Bind has added a number of options
> that I need to set, but before that can happen I need to upgrade BIND,
> and before that can happen I need to upgrade some other stuff.

When you're done upgrading, you'll find BIND 9.16 quite easy to
work with.  Take a look at the new signing policy features.

> All this is for a total of 9 domains - a toy setup by any measure.

It is often easier to things at scale than for just a handful of
domains.  The effort to learn the tools is more efficiently amortised.
This is why we're seeing a rush to move IT to the cloud and make it
someone else's problem.

> I know a little (but not a lot) about the DNS infrastructure at
> Oracle, and the cost and complexity of migrating it to DNSSEC would be
> staggering.

I'm rather sceptical about "staggering", sure there'd be a bunch of man
weeks spent in meetings, and some devops folks may have to put together
some Nagios monitoring scripts, and spend a bit of time to enable
automated zone signing.  It is a non-trivial, but not a staggering cost.

Yes, the business case to expend the modest, but non-trivial resources
is often not that great at present, and complications arise if one is
also depending on just short of standards-compliant DNS load-balancers,
that don't even do regular EDNS at all well.

The main adoption barrier is IMHO really poor support for key rollover
at the registrars.  There's a bunch of recent activity to make that
better, and some new registrars are much better at this than the long
established ones, who aren't investing in keeping current, the cost
savings may make sense in the short to medium term, but long-term I
expect they're not a good strategy.

-- 
    Viktor.

---

From: Mark Andrews <[email-masked]>
Date: Thu, 22 Apr 2021 14:12:18 +1000

> On 22 Apr 2021, at 10:00, Bron Gondwana <[email-masked]> wrote:
> 

CDC just have plain incompetent DNS administrators.  Serving different (unsigned/bad) content on
ns[123].cdc.gov to that on the delegated server for akam.cdc.gov at the time was just idiotic.
It will cause issues even without DNSSEC.  If they are trying to make ns[123].cdc.gov hidden
primaries for akam.cdc.gov they did an abysmal job of it.  Put the hidden primaries on different
addresses or use TSIG to select a different view with the unsigned content.

They sent what appeared to be spoofed (signatures stripped) responses to every validating resolver
on the planet.  The validating resolvers eventually figure it out but not without exceeding client
timeouts and/or query limits.

Currently they are returning REFUSED for names ending in akam.cdc.gov which means they effectively
only have a single working nameserver for cdc.gov for anyone trying to reach their web site.

% dig dnskey akam.cdc.gov +norec +bufsize=1400 @ns1.cdc.gov

; <<>> DiG 9.15.4 <<>> dnskey akam.cdc.gov +norec +bufsize=1400 @ns1.cdc.gov
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 8482
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 4a0cdd6da4a272451d341bfd6080edd34da99a5353fecd1d (good)
;; QUESTION SECTION:
;akam.cdc.gov.			IN	DNSKEY

;; Query time: 229 msec
;; SERVER: 198.246.96.61#53(198.246.96.61)
;; WHEN: Thu Apr 22 13:30:27 AEST 2021
;; MSG SIZE  rcvd: 69

% dig dnskey akam.cdc.gov +norec +bufsize=1400 @ns2.cdc.gov

; <<>> DiG 9.15.4 <<>> dnskey akam.cdc.gov +norec +bufsize=1400 @ns2.cdc.gov
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 35055
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: b57f84015326a3acecdc0fa36080ede53418789f416c7f42 (good)
;; QUESTION SECTION:
;akam.cdc.gov.			IN	DNSKEY

;; Query time: 394 msec
;; SERVER: 198.246.96.92#53(198.246.96.92)
;; WHEN: Thu Apr 22 13:30:46 AEST 2021
;; MSG SIZE  rcvd: 69

% dig dnskey akam.cdc.gov +norec +bufsize=1400 @ns3.cdc.gov

; <<>> DiG 9.15.4 <<>> dnskey akam.cdc.gov +norec +bufsize=1400 @ns3.cdc.gov
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 9881
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 55043d8473bc59bbfd7e57206080edf744d56968c25be8bc (good)
;; QUESTION SECTION:
;akam.cdc.gov.			IN	DNSKEY

;; Query time: 303 msec
;; SERVER: 198.246.125.10#53(198.246.125.10)
;; WHEN: Thu Apr 22 13:31:04 AEST 2021
;; MSG SIZE  rcvd: 69

% dig dnskey akam.cdc.gov +norec +bufsize=1400 @auth00.ns.uu.net

; <<>> DiG 9.15.4+ <<>> dnskey akam.cdc.gov +norec +bufsize=1400 @auth00.ns.uu.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7246
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 6, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 5ac471018fb08a53284b835c6080ee0d00573362d9cb243e (good)
;; QUESTION SECTION:
;akam.cdc.gov.			IN	DNSKEY

;; AUTHORITY SECTION:
akam.cdc.gov.		86400	IN	NS	a9-64.akam.net.
akam.cdc.gov.		86400	IN	NS	a5-66.akam.net.
akam.cdc.gov.		86400	IN	NS	a28-65.akam.net.
akam.cdc.gov.		86400	IN	NS	a8-67.akam.net.
akam.cdc.gov.		86400	IN	NS	a1-43.akam.net.
akam.cdc.gov.		86400	IN	NS	a2-64.akam.net.

;; Query time: 1266 msec
;; SERVER: 198.6.1.65#53(198.6.1.65)
;; WHEN: Thu Apr 22 13:31:26 AEST 2021
;; MSG SIZE  rcvd: 198

% 

% dig www.cdc.gov @ns1.cdc.gov +norec

; <<>> DiG 9.15.4 <<>> www.cdc.gov @ns1.cdc.gov +norec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58095
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 0753cc8fb65017d8e84280f26080ef345d618fe913abd0fe (good)
;; QUESTION SECTION:
;www.cdc.gov.			IN	A

;; ANSWER SECTION:
www.cdc.gov.		300	IN	CNAME	www.akam.cdc.gov.

;; Query time: 704 msec
;; SERVER: 198.246.96.61#53(198.246.96.61)
;; WHEN: Thu Apr 22 13:36:21 AEST 2021
;; MSG SIZE  rcvd: 91

% dig www.cdc.gov @ns2.cdc.gov +norec

; <<>> DiG 9.15.4 <<>> www.cdc.gov @ns2.cdc.gov +norec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5508
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 527eb3a3f9958642ae7a63f76080ef4ff9709b05d69578e1 (good)
;; QUESTION SECTION:
;www.cdc.gov.			IN	A

;; ANSWER SECTION:
www.cdc.gov.		300	IN	CNAME	www.akam.cdc.gov.

;; Query time: 227 msec
;; SERVER: 198.246.96.92#53(198.246.96.92)
;; WHEN: Thu Apr 22 13:36:47 AEST 2021
;; MSG SIZE  rcvd: 91

% dig www.cdc.gov @ns3.cdc.gov +norec

; <<>> DiG 9.15.4 <<>> www.cdc.gov @ns3.cdc.gov +norec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14138
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: d152743ae9d43d26022fb3856080ef59995338b4117d63fe (good)
;; QUESTION SECTION:
;www.cdc.gov.			IN	A

;; ANSWER SECTION:
www.cdc.gov.		300	IN	CNAME	www.akam.cdc.gov.

;; Query time: 338 msec
;; SERVER: 198.246.125.10#53(198.246.125.10)
;; WHEN: Thu Apr 22 13:36:57 AEST 2021
;; MSG SIZE  rcvd: 91

% dig www.cdc.gov @auth00.ns.uu.net +norec

; <<>> DiG 9.15.4 <<>> www.cdc.gov @auth00.ns.uu.net +norec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11325
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 6, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 1248394757c2c358b65340b76080ef880260f2463fd1f873 (good)
;; QUESTION SECTION:
;www.cdc.gov.			IN	A

;; ANSWER SECTION:
www.cdc.gov.		300	IN	CNAME	www.akam.cdc.gov.

;; AUTHORITY SECTION:
akam.cdc.gov.		86400	IN	NS	a5-66.akam.net.
akam.cdc.gov.		86400	IN	NS	a1-43.akam.net.
akam.cdc.gov.		86400	IN	NS	a9-64.akam.net.
akam.cdc.gov.		86400	IN	NS	a2-64.akam.net.
akam.cdc.gov.		86400	IN	NS	a8-67.akam.net.
akam.cdc.gov.		86400	IN	NS	a28-65.akam.net.

;; Query time: 223 msec
;; SERVER: 198.6.1.65#53(198.6.1.65)
;; WHEN: Thu Apr 22 13:37:44 AEST 2021
;; MSG SIZE  rcvd: 220

% 

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 22 Apr 2021 01:11:00 -0400

> other specialised tech.  I am just saying the old excuses are tired out, we

P.S.  My typo (crucial skipped negation) in the above may have given
rather the wrong impression.  I meant to write:

    I am *not* suggesting that Google can easily do DNSSEC for
    google.com, ...

There are likely significant complexities in making changes at Google's
scale, given all the specialised machinery in use for DNS load-balancing.

Mind you, there could be an opportunity to revisit this once support for
DNS SVCB/HTTPS records is ubiquitous.  In a clean-slate design the
targets of these can be stable names in unsigned zones, with all the
dynamic IP responses limited to those zones:

    ; Signed largely static data:
    ;
    example.com. 1D IN HTTPS 0 www.dyndns.example.com.
    _443._tcp.example.com IN TLSA 2 1 1 ...
    ;
    www.example.com. 1D IN HTTPS 0 www.dyndns.example.com.
    _443._tcp.www.example.com IN TLSA 2 1 1 ...

    ; Unsigned, short TTL dynamic data, client-subnet dependent, ...
    ;
    www.dyndns.example.com. IN A ...
    www.dyndns.example.com. IN AAAA ...

and all the short TTLs, non-DNSSEC load-balancers, ... limited to just
such specialised zones.

-- 
	Viktor.

---

From: [email-masked]
Date: Thu, 22 Apr 2021 07:58:12 -0700 (PDT)

> further improvements in DNSSEC support have been made in 9.16.

dnssec-policy suppport is the main reason I need to upgrade.

> support, this sort of mistake will become less common.  But it is easy

I had checks in place. The problem is the reporting tool I chose to use
should have caught this, but didn't.

I think it is asking a lot of people to code all of their own checks. Most
people are going to pick a tool and use it like I did. If that tool is deficient
in some way... how was I supposed to know that?

Of course the real problem is that cut-and-paste was required to do this.
It's really really easy to mess this up when you're processing multiple
domains.

> other DS record is sound.

The one IANA used, and probably still uses, does check. I checked the RFC at the
time and I think the check IANA is doing is allowed, but definitely suboptimal
for a mail server.

But the bigger problem was the error message. If it has said "one of your
DS records is messed up" that would have been fine. But it said "invalid
domain" or something similar. So it was me at one end, report in hand
saying everything is fine, and IANA on the other with a mysterious error.

It was only when IANA thought to do a DNSSEC check using a different tool - one
that does check this - that we figured out the problem.

> work with.  Take a look at the new signing policy features.

Ack.

> automated zone signing.  It is a non-trivial, but not a staggering cost.

You seriously underestimate the difficulty. You might want to consider the fact
that even if you ignore things like Oracle having multiple separately managed
business units spread across many different countries, like most really large
companies, Oracle is composed almost entirely of acquisitions, new and old, which
for timing and political reasons have undergone varying degrees of assimilation. 

Some of those acquisitions have their own entirely separate DNS setups,
with separate management. 

Of course the goal is to get them all into the collective. But as anyone who has
gone through the process can tell you, there's always resistance, even if in the
long run it's futile...

And there's always another acquisition in the pipeline.

> expect they're not a good strategy.

Agreed. 

I don't know for sure how many registrars Oracle interacts with - in addition
to being one itself - but my estimate is "a lot".

				Ned

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 23 Apr 2021 15:50:27 +1200

I'd like to think this will lighten the mood:
https://dilbert.com/strip/2021-04-22
but there is of course some truth in it.

Regards
   Brian Carpenter