Subject: IETF Mail List Failure

From: IETF Executive Director <[email-masked]>
Date: Thu, 17 May 2018 19:50:54 -0700

Dear IETF Community -

On 17 May, from 17:48 UTC (13:48 EDT) to 23:48 UTC (18:49 EDT), outbound email from the IETF email lists failed to send. Mail sent to the lists went into the archives, but was not received by list subscribers. 

A problem in the DMARC-rewriting system caused outbound mail delivery to fail silently, without alarm or error message. The rewriting system has been taken offline, which restored outbound email service.  The lists are back up.

Unfortunately, because the mail was discarded rather than queued, if you sent mail to any IETF list today during the outage window, we ask you to please resend your message.

We apologize for this issue and are working with the Tools Team to correct the system before it is put back online again.

Alexa Morris
IETF Executive Director

---

From: Randy Bush <[email-masked]>
Date: Fri, 18 May 2018 09:50:13 +0200

> On 17 May, from 17:48 UTC (13:48 EDT) to 23:48 UTC (18:49 EDT),
> outbound email from the IETF email lists failed to send. Mail sent to
> the lists went into the archives, but was not received by list
> subscribers.
> 
> A problem in the DMARC-rewriting system caused outbound mail delivery
> to fail silently, without alarm or error message. The rewriting system
> has been taken offline, which restored outbound email service.  The
> lists are back up.
> 
> Unfortunately, because the mail was discarded rather than queued, if
> you sent mail to any IETF list today during the outage window, we ask
> you to please resend your message.
> 
> We apologize for this issue and are working with the Tools Team to
> correct the system before it is put back online again.

thanks, alexa.  these things happen, and i won't even demand a refund :)
sympathies for the pain.

< micro-management = off >

i really appreciate technical details.  there are still a few operators
left in the ietf community, and we try to learn from our oopsies.  a
deeper post mortem would even be welcome as many of us are also trying
to deal with dmarc hell.  you run a medium scale mail operation and any
lessons you share are appreciated.

randy

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 18 May 2018 09:04:29 -0500

Hi, Alexa,

> > has been taken offline, which restored outbound email service.  The

I echo Randy's thanks for technical details.

> > you to please resend your message.

Just so I do the right thing, do you folks think all mail to be forwarded
was silently discarded, or only the mail sent from addresses that were
subject to re-writing?

Spencer

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 18 May 2018 10:07:46 -0400

Thanks for the information and the effort.

But really, aren't all these issues really showing we are trying to use the
wrong tool? IETF actually has a protocol that is purpose designed for
running mailing lists and obviates the need for DMARC hell - NNTP.

NNTP is actually two separate protocols. There is the server-server
replication protocol which was really important when NNTP was designed and
is no longer needed and doesn't scale. And then there is the client-server
interaction which is actually really useful.

---

From: "Paul Hoffman" <[email-masked]>
Date: Fri, 18 May 2018 08:18:36 -0700

On 18 May 2018, at 7:07, Phillip Hallam-Baker wrote:

> wrong tool?

No.

--Paul Hoffman

---

From: Michael Richardson <[email-masked]>
Date: Fri, 18 May 2018 13:13:54 -0400

> running mailing lists and obviates the need for DMARC hell - NNTP.

yeah, I hear you, and almost agree.

    > interaction which is actually really useful.

We do have the common IMAP server, and maybe that will become more and more
used.  Certainly if you want to catch up on last night's hiccup, that would
work well.

Is there a way to create a URL (or .something file) that would configure
Thunderbird or LookOut automatically to subscribe to a WG ML via IMAP?
That would help newbies.

I would probably prefer to use NNTP to access that system, as the good old
news reader tools still beat most mail readers (for me, who still lives in
emacs), but they also don't handle HTML email. (Maybe that's a good thing,
given efail) 

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Randy Bush <[email-masked]>
Date: Fri, 18 May 2018 19:24:02 +0200

> But really, aren't all these issues really showing we are trying to use the

you seem to have missed

---

From: Alexa Morris <[email-masked]>
Date: Fri, 18 May 2018 20:03:49 -0700

I am relieved to report that the issue was repaired earlier today and DMARC rewriting is back online (and has been for several hours). I know that many of you are interested in hearing the technical details, and I will make sure that we share that information shortly. 

Regards,
Alexa

> On May 18, 2018, at 12:50 AM, Randy Bush <[email-masked]> wrote:
>

---

From: "John Levine" <[email-masked]>
Date: 19 May 2018 12:08:32 -0400

In article <8082.1526663634@localhost> you write:
>> NNTP is actually two separate protocols. ...

>We do have the common IMAP server, ...

>emacs), but they also don't handle HTML email. (Maybe that's a good thing,

Funny you should mention that.  I have a little script that copies
messages on the lists I read from the IETF IMAP server to local
newsgroups to my local NNTP server, which I then read with my
newsreader.  The groups are marked as moderated which means that
"posted" articles are mailed instead, and it sends them to the
appropriate list.  Works great.

Re HTML messages, newsreaders have evolved since you used trn twenty
years ago.  Take a look.  Thunderbird's not bad.

Setting up an IETF NNTP server with all of the lists as newsgroups
wouldn't be very hard, give or take the usual hassles of setting
up a new service and being sure it's someone's job to oversee it.

R's,
John

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 21 May 2018 08:38:19 -0500

So, keeping in mind that as a TSV AD, I've been sitting through the IESG's
discussions about what to do with DMARC and IETF mailing lists, but e-mail
in general and DMARC in particular is not remotely my area of expertise
(all e-mail looks like a bytestream to transport types, as does almost
everything else) ...

> appropriate list.  Works great.

I am intrigued by this thought, but find myself wondering whether there is
any remaining problem with DMARC from e-mailed responses.

I also wonder what threading looks like if an exchange is bouncing back and
forth between someone who's using e-mail and someone who's using a
newsreader and e-mailing responses (from the point of view of the e-mail
user, the newsreader user, and third-party observers).

But I don't think the IESG believes we've come up with the Final Ultimate
Solution to the DMARC Mailing List Problem (with apologies to
http://www.dmuth.org/fussp.html), so I'm still willing to wonder about
stuff like this ...

Spencer, who is 1/16th of the IESG on his best days, and is not the IESG
liaison to the tools team (according to https://trac.ietf.org/trac/iesg/wiki,
that's now EKR)