Subject: IETF Policy on dogfood consumption or avoidance - SMTP version

From: John C Klensin <[email-masked]>
Date: Sun, 15 Dec 2019 16:15:40 -0500

Hi.

It has long been my personal belief that, in its operation of
various of its own services on the Internet the IETF should
adhere closely to its own standards.  If we do not do so, we
lose all credibility in recommending to others that they follow
our standards.  This practice has been referred to in many
discussion threads over the years as "eating our own dog food".  

It has recently come to the attention of several of us, via an
extended discussion on the SMTP list, that the IETF email
servers are rejecting all SMTP connections whose EHLO commands
contain IP address literals.   While the text describing the
appropriateness of use of IP literal is RFC 5321 is more
complicated than it probably ought to be, the discussion in
Section 4.1.4 of that document seems quite clear that an SMTP
server MUST NOT reject a message simply because an IP address
literal (or a domain name that does not point to a host) is
used. Those interested in the niceties of that issue should
review the correspondence on the [email-masked] list and
comment there if appropriate.

A ticket ( [www.ietf.org/rt #282782] ) was generated early in
the month about the ietf.org mail servers apparently rejecting
messages with IP address literals in the EHLO field.  The
rejection is accompanied by a reply message that appears to be
inappropriate in multiple ways; again, those interested should
see the ietf-smtp list for an already-extensive discussion.  The
Secretariat responded by indicating that all such addresses were
being rejected and that the rejection was occurring under
instructions from IETF leadership, instructions that were
reaffirmed after the ticket was filed.  Whatever the problem is,
and indeed, whether there is a problem, the Secretariat is
therefore blameless.  I suggest that the IETF has a problem.

The purpose of this note is _not_ to evaluate the underlying
technical issues, what should be done about them, or whether the
text in RFC 5321 should be improved.  Those, it seems to me, are
topics for the ietf-smtp list.   They have been discussed there
at length and presumably will continue to be discussed there.
It is whether there is consensus among IETF participants that
"the leadership" (I presume whatever bodies, individuals, or
their designees are relevant) should have the authority to
instruct the Secretariat to violate an IETF standard without
consultation of appropriate experts within the community
(presumably on relevant mailing lists), evidence of IETF rough
consensus, and/or Internet Drafts that specify alterations to
the relevant standard(s).  I also don't want to cast blame about
decisions of the past, only to understand what the process is
for giving instructions to the Secretariat (or approving their
suggestions) is now and whether IETF conformance to IETF
standards is something we care about for the future.

  john

---

From: Randy Bush <[email-masked]>
Date: Sun, 15 Dec 2019 14:03:25 -0800

> standards is something we care about for the future.

excuse if i stay out of the above layer seven issue(s) you raise.
american, brit, ietf, ... politics disgust me, and discussion seems
futile.

but, as one of my hats is in ops, i gotta ask two technical questions.

  o would it be technically easy for the smtp servers to accept ip
    literals in a conforming manner?  yes, this is a question for my
    esteemed friend glen and his partner in crime, matt.

  o what would the technical and/or security exposure or other
    downside(s) be of doing so?

randy

---

From: "John Levine" <[email-masked]>
Date: 15 Dec 2019 17:29:27 -0500

In article <8EE11B75E1F8A7E7105A1573@PSB> you write:
>contain IP address literals. ...

$ telnet -4 mail.ietf.org 25
Trying 4.31.198.44...
Connected to mail.ietf.org.
Escape character is '^]'.
220 ietfa.amsl.com ESMTP
ehlo [64.57.183.53]
250-ietfa.amsl.com
250-PIPELINING
250-SIZE 67108864
250-ETRN
250-STARTTLS
250-AUTH PLAIN LOGIN
250-AUTH=PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250 8BITMIME
quit
221 2.0.0 Bye
Connection closed by foreign host.

$ telnet mail.ietf.org 25
Trying 2001:1900:3001:11::2c...
Connected to mail.ietf.org.
Escape character is '^]'.
220 ietfa.amsl.com ESMTP
ehlo [ipv6:2001:470:1f07:1126:0:43:6f73:7461]
250-ietfa.amsl.com
250-PIPELINING
250-SIZE 67108864
250-ETRN
250-STARTTLS
250-AUTH PLAIN LOGIN
250-AUTH=PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250 8BITMIME
quit
221 2.0.0 Bye
Connection closed by foreign host.

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 15 Dec 2019 17:54:32 -0500

On Sun, Dec 15, 2019 at 02:03:25PM -0800, Randy Bush wrote:

> I gotta ask two technical questions.
> 

Since the servers in question run Postfix, I feel I am well placed to
answer that question with a "yes".  Indeed Postfix accepts IP address
literals in helo names by default.  Even when Postfix is configured
to reject non-FQDN forms:

    http://www.postfix.org/postconf.5.html#reject_non_fqdn_helo_hostname

    Reject the request when the HELO or EHLO hostname is not in
    fully-qualified domain or address literal form, as required by the
    RFC.

address literals are still accepted.

It takes some effort to get Postfix to reject address literals, the
administrator would have to implement a custom regular expression table,
to accept some and reject other HELO name forms.  For example:

  main.cf:
    smtpd_helo_restrictions =
        permit_mynetworks,
        permit_sasl_authenticated,
        reject_non_fqdn_helo_hostname,
        check_helo_access pcre:${config_directory}/helo.pcre

  helo.pcre:
    /^\[/   REJECT Address literals not welcome here

>     downside(s) be of doing so?

Use of address literals is not common among carefully configured,
well-maintained Internet-facing outbound relay MTAs.  There is likely
some positive correlation between address-literal use and generally
sloppy MTA configuration, which might open said MTAs to abuse.

That said, I personally doubt that use of address literals is a
high-quality signal that distinguishes legitimate MTAs from abuse
sources.  It is merely a small indicator that the sending MTA is
not up to best-practice.

Keeping out junk is unavoidably a balancing act of trade-offs, some
legitimate traffic is liable to get blocked.  The key question with
any approach one takes is whether it is sufficiently effective (how
much junk is blocked) and whether the FP rate is low enough.

-- 
    Viktor.

P.S.

In my own Postfix configuration, I see:

    /^[\d.]+$/  550 Your software is not RFC 2821 compliant

which blocks "HELO 192.0.2.1", but not "HELO [192.0.2.1]".  It could
have been more precise to match only dotted quads.

    /^(?:\d+\.){3}\d+$/ 550 Your software is not RFC 2821 compliant

Perhaps somewhere along the way, someone came up with a variant
that does block address literals, whether deliberately or by
accident.

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Sun, 15 Dec 2019 17:58:01 -0500

> Connection closed by foreign host.

It throws the error later in the transaction:

(slice-n-mice from the original report, which had the IP obfuscated)

220 ietfa.amsl.com ESMTP
EHLO [A.B.C.D]
250-ietfa.amsl.com
250-PIPELINING
250-SIZE 67108864
250-ETRN
250-STARTTLS
250-AUTH PLAIN LOGIN
250-AUTH=PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250 8BITMIME
MAIL FROM:<>
250 2.1.0 Ok
RCPT TO:<[email-masked]>
550 5.7.1 <[A.B.C.D]>: Helo command rejected: RFC2821 violation

Personally,  my opinion is that if there's indication that a lot of spam or
other malicious mail is arriving from "address literal EHLO" sources, it's
appropriate to respond with a "550 5.7.1 Rejected due to policy reasons",
preferably in response to the EHLO, but at RCPT TO is at least acceptable if
that's where the MTA code says to do it.  I've run mail servers that I set to
throw "550 rejected due to local policy" for *lots* less significant reasons,
and if the Secretariat has been told to reject such mail, "550 5.7.1 due to
local policy" is the *exact* way to implement it.

But claiming it's an RFC2821 violation isn't acceptable.

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 15 Dec 2019 17:59:02 -0500

> Escape character is '^]'.

Postfix delays HELO checks until RCPT TO.  You'd need to go all the way
to RCPT TO:

    $ telnet mail.ietf.org 25
    220 ietfa.amsl.com ESMTP
    EHLO [100.2.39.101]
    250-ietfa.amsl.com
    250-PIPELINING
    250-SIZE 67108864
    250-ETRN
    250-STARTTLS
    250-AUTH PLAIN LOGIN
    250-AUTH=PLAIN LOGIN
    250-ENHANCEDSTATUSCODES
    250 8BITMIME
    mail from:<[email-masked]>
    250 2.1.0 Ok
    rcpt to:<[email-masked]>
    550 5.7.1 <[100.2.39.101]>: Helo command rejected: RFC2821 violation
    quit
    221 2.0.0 Bye
    Connection closed by foreign host.

-- 
    Viktor.

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 15 Dec 2019 23:00:16 +0000

John C Klensin wrote on 15/12/2019 21:15:
> the relevant standard(s).
this is rather missing the point - or perhaps accepting the point - that 
there's a good deal of difference between defining a working protocol 
and running a production system which uses that protocol. The protocol 
will be defined by people who are good at defining protocols, but the 
system will be run by someone who knows how to run systems. Sometimes 
there is consistency between the two.

Currently it's expedient to drop domain literals in EHLO commands, but 
this is a policy practice of the operators rather than an integral 
function of the protocol itself.  Codifying this in the protocol is 
roping in policy considerations and probably the line protocol is not 
the place to do this.

Nick

---

From: Keith Moore <[email-masked]>
Date: Sun, 15 Dec 2019 18:04:59 -0500

On 12/15/19 6:00 PM, Nick Hilliard wrote:

> The protocol will be defined by people who are good at defining 
> protocols, but the system will be run by someone who knows how to run 
> systems.

I do not find that this is true in general, and IMO it is not a valid 
working assumption.   There's a tremendous amount of Bad Conventional 
Practice out there, especially in both email handling and security, that 
has come to be accepted as if it were wisdom.

Keith

---

From: John C Klensin <[email-masked]>
Date: Sun, 15 Dec 2019 18:16:51 -0500

John,

It may have been fixed when we (the people participating in the
long discussion on the ietf-smtp list) were not looking, in
which case, thanks to the Secretariat.. I think (read that list
for the reason for the qualification)).

However, as I tried to say several times, that isn't the point.
The point is that, apparently, some committee representing "the
leadership" -- a committee whose existence, charge, and
composition I don't believe has been publicly announced -- gave
AMS instructions (years ago - this is not about the current
IESG) about blocking IP address literals, instructions that
violated an IETF standard.  I don't know if the IESG reviewed
those instructions before they were given, but I'm not convinced
that makes any difference at this point.  I think instructions
that tell the Secretariat to violate an IETF standard,
regardless of the technical details, are a bad idea and an
especially bad one when the community does not know what is
going on.   YMMD, of course.  If the committee (or the IESG) was
not aware that following the instructions would violate the
standard (and I suspect that was the case), then we still have a
problem because anyone looking at us from the outside doesn't
know or care whether we are not conforming to our standards
deliberately or by accident or ignorance.  

So, again, I want to understand whether we (both the leadership
and the broader IETF community) think that instructing the
Secretariat to violate IETF standards is ok.   And, if it is not
ok, what mechanisms we need to prevent similar occurrences,
possibly over less obscure issues, in the future.

    best,
     john

p.s. And, yes, I have seen the pointers to the anti-spam
policies.  But whether this case is covered at all there is open
to question and, more important, nothing in that policy
statement says "reducing spam by even a small percentage is so
important that it is ok for the IETF mail servers to reject
legitimate SMTP sessions even when such rejections are
explicitly prohibited by the standard".  Had such a statement
appeared, I assume that the discussion that has been occurring
on the ietf-smtp list for the last few weeks would have occurred
years ago.

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 15 Dec 2019 18:19:21 -0500

On Sun, Dec 15, 2019 at 06:04:59PM -0500, Keith Moore wrote:

> Practice out there, especially in both email handling and security,
> that has come to be accepted as if it were wisdom.

Yes, there is much cargo-culting of questionable practice.  But there
are also in some cases specifications that (no longer) match operational
realities.  Which is not to suggest that support for address literals is
necessarily such an example, it rather depends on metrics I don't have,
and which may even be time and place specific.

Perhaps there was once a particular stream of abuse directed at ietf.org
that the rule in question stopped.  It may still be effective, then and
could now be merely a low-volume source of false-positives.  Or it could
still be, perhaps intermittently, an effective way to distinguish spam
from ham.

If I were the operator of the MTA in question, I'd consider dropping the
rule after checking recent logs for signs of its efficacy.

-- 
    Viktor.

---

From: "Salz, Rich" <[email-masked]>
Date: Sun, 15 Dec 2019 23:25:04 +0000

Do you have any proof of this?

---

From: Keith Moore <[email-masked]>
Date: Sun, 15 Dec 2019 18:26:19 -0500

On 12/15/19 6:19 PM, Viktor Dukhovni wrote:

> realities.

Yes this does happen.   Though I doubt this is the case for IP address 
literals in EHLO.

Keith

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Sun, 15 Dec 2019 18:31:01 -0500

On Sun, 15 Dec 2019 23:00:16 +0000, Nick Hilliard said:

> function of the protocol itself.

The point is that, given a syntactically correct and acceptable EHLO,
there's a massive difference between:

mail.ietf.org says "550 5.7.1 mail rejected due to EHLO violating local policy"

mail.ietf.org says "550 5.7.1 mail rejected due to EHLO RFC2821 violation".

If the Secretariat was told to reject such mail, the first is a totally correct
way to do it, and the second isn't (if it *was* an RFC violation, a 500 or 501
should be returned rather than 550)..

Whether a given site should block address literal EHLO's is a totally different
question entirely - I've run mail servers where anything that EHLO'd with
an address literal was almost guaranteed spam/malware, and I've run mail
systems whose entire purpose in life was to accept and forward mail for
appliances that could only EHLO with address literals.  As a result, I'll have
to defer to people who have actual stats for ietf.org mail for how much
spam it blocks versus how much ham mail is rejected, and the level of
tolerance for rejection of sketchy-looking but legitimate mail....

---

From: "John R Levine" <[email-masked]>
Date: 15 Dec 2019 18:32:29 -0500

> appropriate to respond with a "550 5.7.1 Rejected due to policy reasons",

I agree we should fix the message.  It has been a very long time since I 
saw any legit mail from a host that identified itself with an IP other 
than someone trying to make a point that I don't understand.

R's,
John

---

From: Randy Bush <[email-masked]>
Date: Sun, 15 Dec 2019 15:57:35 -0800

> Personally, my opinion is that if there's indication that a lot of
> spam or other malicious mail is arriving from "address literal EHLO"
> sources

that is not reasonably filtered by other means.

randy

---

From: John C Klensin <[email-masked]>
Date: Sun, 15 Dec 2019 23:12:21 -0500

--On Sunday, December 15, 2019 18:32 -0500 John R Levine
<[email-masked]> wrote:

>> Personally,  my opinion is that if there's indication that a
>> lot of spam or other malicious mail is arriving from "address
>> literal EHLO" sources, it's appropriate to respond with a
>> "550 5.7.1 Rejected due to policy reasons",
> 
> I agree we should fix the message.  It has been a very long
> time since I saw any legit mail from a host that identified
> itself with an IP other than someone trying to make a point
> that I don't understand.

John, Valdis, Subramanian, and maybe a few others,

First of all, I read John's message too quickly and had
forgotten that the rejection message attributing the problem to
EHLO does not appear until after a RCPT command is sent.  So
SM's test is the important one; John's just didn't go far
enough.  There are also a number of issues raised by the error
message.  The problem condition should not be attributed to RFC
2821; it probably shouldn't refer to the "Helo command" when
EHLO was used; it is, as suggested above, a policy matter and
not a requirement of any IETF protocol specification; and, while
the specification arguably permits it, accepting the EHLO and
then rejecting the mail transaction only after processing the
MAIL and at least one RCPT command is probably at least in bad
taste.  Some of those issues suggest changes or clarification
that should be applied when and if RFC 5321 is updated and
replaced.  Most of them have been discussed, at length, on the
ietf-smtp list; I assume the others soon will be.

Interpretation is up to the SAA team but, given that the
[email-masked] list exists and a productive discussion of
the technical parts of this topic has already occurred there, I
hope we can avoid either repeating that discussion on this list
or forking it and expecting people to follow and integrate both.

None of the above has much of anything to do with the key issue
that I thought 
was worth raising on this list.  According to the response to
the trouble ticket, the Secretariat was told to block mail
transactions that used IP address literals in the EHLO command.
No matter how pragmatically desirable that may be (see the
ietf-smtp list for a discussion on that topic too), at least as
I read it (and wrote it) RFC 5321 prohibits that rejection,
making the IETF mail servers non-conforming to the standard.
So, again, the first question for this list is whether we are
comfortable with servers run in the IETF's name being
non-conforming to IETF standards.  I see objections to that on
ethical and credibility ground, but others may disagree.  If we
do believe that our not -conforming to our standards may be
appropriate in some circumstances, should there be a mechanism
for at least informing, and possibly consulting, the community
before that is done or are we comfortable not knowing about it
until someone stumbles over the problem.

I'm not going silent on the second topic because I want to see
what others have to say.  And, because I believe the technical
issues and tradeoffs belong on the ietf-smtp list, I'm not going
to comment further on them here until the SAA team or some AD
requests that I do otherwise.

best,
  john

---

From: Keith Moore <[email-masked]>
Date: Mon, 16 Dec 2019 00:07:15 -0500

On 12/15/19 11:12 PM, John C Klensin wrote:

> ethical and credibility ground, but others may disagree.

In general, not specific to SMTP: If it could be established that there 
was a substantial and legitimate operational reason to deviate from a 
standard, IMO the correct response is to file an erratum with the RFC 
Editor and correct the standard in the next revision.   We should not be 
stubborn about sticking to standards that promote bad advice.

However, I seriously doubt that there is a legitimate reason to block 
mail that is sent using HELO/EHLO with an address literal. It is not a 
reliable indicator of spam, at least not over the long term.   If there 
really are spammers sending mail that way, there is almost certain to be 
some clue in the content of the message that could be used to filter 
with similar effectiveness.

I have been of the impression for around 30 years that the HELO (later 
EHLO) argument is pretty much meaningless and really should be ignored.

Also, bogus spam filters appear to be at least as detrimental to the 
reliability of email as spam itself.    IMO, IETF should not encourage 
such practices either in its standards or its practices, as they are 
harmful to the Internet.  Heuristics will likely only work in the short 
term and shouldn't be promoted as good practice.

Keith

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 16 Dec 2019 11:27:39 +0000

> mail.ietf.org says "550 5.7.1 mail rejected due to EHLO RFC2821 violation".

there's no requirement for the text from a 550 reply code to be parsed 
by the MTA, so the difference is in the eye of the person receiving the 
bounce message.  If it's semantically inaccurate, then maybe someone can 
change the text and we can move on?

Nick

---

From: S Moonesamy <[email-masked]>
Date: Mon, 16 Dec 2019 03:30:18 -0800

Hi John,
>ietf-smtp list; I assume the others soon will be.

The rejection is for policy reasons.  The error text stating that it 
is because of a "RFC violation" is incorrect.

>non-conforming to IETF standards.  I see objections to that on

It does not look good when a SDO is unable to explain [1] why it is 
ignoring its own standard.

Regards,
S. Moonesamy

1. There is an initiative of the Dutch government: 
http://r.elandsys.com/r/94862  At first glance, it looks like there 
is something wrong with the server which was tested.  It takes a 
little more effort to interpret the results.

---

From: Glen <[email-masked]>
Date: Mon, 16 Dec 2019 08:11:11 -0800

All -

While I am not normally on the IETF list, I was summoned by my *very*
esteemed friend Randy :-) and I will answer his questions below.  I
have read some of the other messages, and know that this has been
partially addressed already, but as the IT director of the vendor
providing IT operations services to the IETF, and the person who is
most hands-on with this, I felt that I should personally answer the
technical questions he posed.

As Randy wisely implies, the politics of this are beyond me, so please
do not consider my silence elsewhere - anywhere - as apathy; rather,
prudence.  I am always happy to answer questions about our operations.
I cannot ever speak to anything else beyond that.

On Mon, Dec 16, 2019 at 7:43 AM Randy Bush <[email-masked]> wrote:
>     esteemed friend glen

Extremely easy.  The statements already made about Postfix are
correct.  There is a configuration file, with two lines in it:

/^[0-9.]+$/             550 RFC2821 violation
/^\[[0-9.]+\]$/         550 RFC2821 violation

In just seconds, I can easily change the messages, or remove the
rules, either with complete ease.

>     downside(s) be of doing so?

These rules have been in place for roughly 10-ish years, as has
already been explained by John.  They are in essence gateway checks,
which occur before other measures like Postconfirm or Spamassassin see
the messages.  On a given day, there are between 700 and 1000 incoming
messages rejected by this rule.

Changing the messages would have no technical exposure or downsides
that I can see.  Changing the messages may have a positive or negative
security exposure in that it might either (a) send the message that we
(the IETF) are watching and know what we're doing and scare attackers
off, or (b) might cause attackers to abandon this channel (which at
the moment could be a honeypot-esque bit bucket) and focus on other
methods of attack.  But I think both of those things are extremely
small side-effects.

Removing the rules would increase the load on Spamassassin and - for
that subset of those 1000 messages per day that pass through
Spamassassin's upper threshhold - cause us to send out challenge
emails to the (potentially forged) senders of all of those emails.
This could possibly cause increases in greylisting threshholds or
other automated checks used by others to evaluate email - and in
potential delays in IETF email delivery.  It could make us become (or
be perceived as) a spam source, or (incorrectly, but, it's all about
perception) an open relay.  It could also potentially cause some hosts
or ISPs to block or blacklist us, requiring the users of those hosts
to either appeal to their ISPs, or change ISPs, to continue
participating with us.  There may be other downsides I am not aware
of.

I trust the answers are helpful.  If there are other technical
questions to which answers are desired, please copy me directly, as I
do not normally subscribe to the IETF list.

Thank you!
Glen
--
Glen Barney
IT Director
AMS (IETF Secretariat)

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 16 Dec 2019 16:18:14 +0000

> rules, either with complete ease.

s/RFC2821 violation/policy violation/

+ let's move on.

Nick

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 16 Dec 2019 11:32:24 -0500

> s/RFC2821 violation/policy violation/

+1

Try as we may, a 40 year old protocol that was designed before we had any
of the technology required to mitigate abuse is not going to meet every
modern need.

What concerns me is not that the old protocols are failing but that the new
protocols that are being deployed are closed. Sure, the Signal spec is
open. But it is a closed system. You cannot connect to the Signal world
from the outside.

Millions of people are using non-SMTP and non-XMPP messaging systems every
day. Perhaps deployment isn't quite as impossible as some people imagine.

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 16 Dec 2019 12:04:59 -0500

> /^\[[0-9.]+\]$/         550 RFC2821 violation

While the patterns look similar, the first one rejects non-compliant
"EHLO 192.0.2.1" and similar dotted quads (or more generally some
mixture of digits and dots), the second rejects RFC-compliant address
literals.  So at least the second message should probably be different,
if the rule is retained.

> Changing the messages may have a positive or negative security
> exposure in that it might either (a) send the message that we (the
> IETF) are watching and know what we're doing and scare attackers off,
> or (b) might cause attackers to abandon this channel (which at the
> moment could be a honeypot-esque bit bucket) and focus on other

In this case, Postfix already augments the message with sufficient
context to make the custom message text immaterial to everyone other
than RFC subject-matter experts.  The sender can determine that it is
the EHLO name that is being rejected.

    550 5.7.1 <[192.0.2.1]>: Helo command rejected: RFC2821 violation

But this being a community of RFC subject-matter experts, the text is
subject to further scrutiny.  It can be changed with no impact other
than addressing its correctness viz. the RFC.

> messages rejected by this rule.

The name "postconfirm" suggests a component that asks (purported)
senders of borderline messages to confirm their existence/intent to send
the message.  One should indeed strive to use that sparingly.

Or is this just Postfix sender address verification (SAV), which merely
checks that the MX hosts of the sender domain don't reject the sender
address?

> emails to the (potentially forged) senders of all of those emails.

Here we get to the question of whether the second rule should be kept or
dropped.  Do you know which of the two accounts for the bulk of the
rejected traffic?  The rules could be moved below any RBL or other IP
reputation checks, below any RHSBL or other sender domain reputation
checks, but still above rules that would trigger "challenge messages".
Then their marginal utility would be more easily discerned.

If the challenge messages are generated after a message is accepted,
(with the message sitting in a quarantine queue waiting for the
verdict), then the HELO checks can be used quite late, at any point
before the message is accepted, which would again help to determine
whether other measures already in place largely obviate their need.

If you don't have SAV, you could even have:

    smtpd_data_restrictions =
        check_helo_access pcre:${config_directory}/helo.pcre

and defer the HELO checks to DATA, by which point all the RBL checks
have run, invalid recipients have been turned away, ...

Anyway, perhaps more than you wanted to know, or this list wants to
read.  But if you do want to explore your options further, you could ask
on the postfix-users list.

-- 
    Viktor.

---

From: "Andrew G. Malis" <[email-masked]>
Date: Mon, 16 Dec 2019 12:13:30 -0500

Back in his original email on the topic, John quoted RFC 5321. However,
5321 also says, in section 7.9, that "... an SMTP server may refuse to
accept mail for any operational or technical reason that makes sense to the
site providing the server.", and then "When mail is rejected for these or
other policy reasons, a 550 code SHOULD be used in response to EHLO (or
HELO), MAIL, or RCPT as appropriate."

So I'm also +1 on keeping the policy intact for the reasons cited by Glen,
and updating the 550 message sent by Postfix. This is conformant to the RFC.

Cheers,
Andy

On Mon, Dec 16, 2019 at 11:32 AM Phillip Hallam-Baker <[email-masked]>
wrote:

> What concerns me is not that the old protocols are failing but that the
> new protocols that are being deployed are closed. Sure, the Signal spec is

---

From: Glen <[email-masked]>
Date: Mon, 16 Dec 2019 09:31:43 -0800

I sense another summoning... truncating for clarity...

On Mon, Dec 16, 2019 at 9:05 AM Viktor Dukhovni <[email-masked]> wrote:
> So at least the second message should probably be different,
> if the rule is retained.

Easily able to be done.

> subject to further scrutiny.

+10000000000000  :-)

> the message.  One should indeed strive to use that sparingly.

Postconfirm is a really nice tool written by the Tools Team that uses
an email challenge-response system to confirm unknown people.  It has
grown to take on a number of other functions for the IETF, including
supporting DMARC-rewriting and so forth.  It is rock solid, and a
pleasure to use.

https://ietf.org/about/groups/tools/
https://tools.ietf.org/tools/postconfirmd/

> Do you know which of the two accounts for the bulk of the rejected traffic?

A quick check this morning suggests that it is the second rule
accounting for the bulk of traffic.

> Anyway, perhaps more than you wanted to know,

I always want to know and understand more.  (Alas, there is always
more to know and understand!  :-) _

> But if you do want to explore your options further, you could ask on the postfix-users list.

I am, and always have been, grateful for the presence and support -
and knowledge! - of the community.  Very seriously!  As a vendor I
cannot be a participant, but I am always appreciative of this and the
many messages containing support and guidance I've received over the
years.

But John's original email on this thread explains that - even though I
am interested, and even though I want things to "work right" - the
decisions about what changes to make, and how to make them, do not
rest with me (as the vendor.)  I know that I*** leadership is here,
and considering everything, and I will be happy to make whatever
changes - if any - they direct me to make.

Glen

---

From: John C Klensin <[email-masked]>
Date: Mon, 16 Dec 2019 12:50:21 -0500

--On Monday, December 16, 2019 16:18 +0000 Nick Hilliard
<[email-masked]> wrote:

>> In just seconds, I can easily change the messages, or remove
>> the rules, either with complete ease.
> 

Nick,

(1) Again, this is more of an issue for the ietf-smtp list than
this one, but, because RFC 5321 says, early in Section 4.1.4,
"If the EHLO command is not acceptable to the SMTP server, 501,
500, 502, or 550 failure replies MUST be returned as
appropriate" and not, e.g., "whenever the server gets around to
it" or "it is ok to wait until after the first RCPT command",
there is still probably a violation of the standard.  Since you
and others are encouraging just changing the error report and
moving on, there is also the mostly-technical question of
whether saying "policy violation" and leaving legitimate senders
to guess what undocumented rule they might have tripped over or
explaining that the problem is with the argument to EHLO (even
though it might encourage the bad guys (including spammers) to
get smarter) is better.

(2) One thing Glen did not mention is that, noting that it is
hard to imagine that fixing something that has been in place and
unchanged for a decade is a screaming emergency, I encouraged
him to not rush to make changes until some consensus emerged,
particularly on the ietf-smtp list, as to whether the right
solution was to tune the error message, to remove the check, or
do something else _and_ until whatever mechanism is used to tell
him, as a vendor, what "we" want him to do makes a decision and
revises the instructions (or not).

(3) Once again, it is the apparent violation of the standard by
an IETF server that caused me to raise the issue on this list.
If we don't care about non-conformance to our own specs, then,
although I'd be interested in knowing what percentage of the
total number of messages the IETF servers reject a day these IP
literals in EHLO represent, I'm likely persuaded from Glen's
figures that accepting them is not worth the added cycles.  If
we do care, then either we should be accepting the mail session
opening attempt and dealing with the problems later in the
process (and accepting the marginal overhead) or we should be
seeing a proposal to change  the standard.  And, again if we
care, we should be looking at, or encouraging someone to look
at, whether we should make some adjustments to lower the odds of
similar non-conformance with IETF standards occurring in the
future.

Finally, to summarize part of an off-list discussion that may be
relevant, I am not accusing anyone (especially not Glen) of
having done something wrong here.  I simply believe that the
question of whether we feel an obligation to conform to our own
standards is important.  If the answer is "yes", then some
adjustments are probably in order going forward, adjustments
that might include a little more transparency about how
instructions to Glen and his colleagues that might interact with
IETF standards are generated and by whom and/or a requirement
that a decision to deviate from relevant standards requires
either an I-D proposing a change to those standards or one
explaining why the deviation is appropriate.  And, for the
record, I have no reason to believe that those who generated the
instructions to Glen were aware that they were directing
non-conformance with the relevant standard.  I assume it just
slipped by (5321 is really hard to read on these issues) and
that the question is only what, if anything, we should do
differently in the future to lower the odds of a recurrence
(again, if we care).

best,
   john

---

From: "Andrew G. Malis" <[email-masked]>
Date: Mon, 16 Dec 2019 13:54:15 -0500

John,

As I said on another email earlier today, I disagree with you that we're
non-conformant to RFC 5321.

Cheers,
Andy

---

From: Randy Bush <[email-masked]>
Date: Mon, 16 Dec 2019 13:10:53 -0800

hi exceedingly esteemed friend glen :)

thanks for overly lucid clues.

> emails to the (potentially forged) senders of all of those emails.

/me does not like.  path to pain.

randy

---

From: Jay Daley <[email-masked]>
Date: Tue, 17 Dec 2019 10:46:45 +1300

Hi

While there is not unanimous consensus, I think the mood is clearly to leave this as an operational decision.  In which case, taking into account the following recommendation ...

> On 17/12/2019, at 5:18 AM, Nick Hilliard <[email-masked]> wrote:
> 

… and the following technical comment … 

> On 17/12/2019, at 6:04 AM, Viktor Dukhovni <[email-masked]> wrote:
> 
>> There is a configuration file, with two lines in it:
>> 

… the following has now changed from

to

	/^[0-9.]+$/             550 RFC2821 violation
	/^\[[0-9.]+\]$/         550 Policy violation

As to the question of data, we cannot say for certain that the rejected messages were all spam, but we have only received one complaint in 10 years and so we can reasonably assume this rule has not caused problems that need to be addressed.

Please let me know if you have any questions, comments or recommendations.

kind regards
Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]
+64 21 678840

---

From: "Andrew G. Malis" <[email-masked]>
Date: Mon, 16 Dec 2019 16:51:03 -0500

Jay,

One comment - RFC 2821 has been obsoleted by RFC 5321, so on the first
pattern, replace 2821 with 5321.

Thanks,
Andy

On Mon, Dec 16, 2019 at 4:47 PM Jay Daley <[email-masked]> wrote:

> Hi
>
> While there is not unanimous consensus, I think the mood is clearly to
> leave this as an operational decision.  In which case, taking into account
> the following recommendation ...
>
> As to the question of data, we cannot say for certain that the rejected
> messages were all spam, but we have only received one complaint in 10 years
> and so we can reasonably assume this rule has not caused problems that need
> to be addressed.
>

---

From: Jay Daley <[email-masked]>
Date: Tue, 17 Dec 2019 12:02:15 +1300

Andrew

> On 17/12/2019, at 10:51 AM, Andrew G. Malis <[email-masked]> wrote:
> 
> Jay,
> 
> One comment - RFC 2821 has been obsoleted by RFC 5321, so on the first pattern, replace 2821 with 5321.

Will do.  

thanks
Jay

> 
> Thanks,
> Andy
> 
> 
> On Mon, Dec 16, 2019 at 4:47 PM Jay Daley <[email-masked] <mailto:[email-masked]>> wrote:
> Hi
> 
>> On 17/12/2019, at 5:18 AM, Nick Hilliard <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> On 17/12/2019, at 6:04 AM, Viktor Dukhovni <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> if the rule is retained..
> 
> 
> 
> [email-masked] <mailto:[email-masked]>
> +64 21 678840
> 

-- 
Jay Daley
IETF Executive Director
[email-masked]
+64 21 678840

---

From: Rob Sayre <[email-masked]>
Date: Mon, 16 Dec 2019 16:43:38 -0800

On Mon, Dec 16, 2019 at 1:47 PM Jay Daley <[email-masked]> wrote:

>
>  we have only received one complaint in 10 years
>

...by that measure, it performs better than most IETF functions.

This seems like a non-issue to me.

thanks,
Rob

---

From: John C Klensin <[email-masked]>
Date: Mon, 16 Dec 2019 21:59:37 -0500

Jay,

Three questions:

* Did you review and understand the extensive discussion that
has occurred on the [email-masked] since the beginning of
the month or just the (relatively few) comments on this IETF
list?

* Did you consider the issues associated with the possibility of
IETF servers being out of conformance, whether clearly or
marginally, with IETF standards and the implications if that
were the case?

* Is the answer to the question of where the Secretariat will
get guidance and instructions on technical issues going forward
that it will come from you and your assessments of  those issues
and community consensus about them?

thanks,
   john

--On Tuesday, December 17, 2019 10:46 +1300 Jay Daley
<[email-masked]> wrote:

> Hi
> 
> While there is not unanimous consensus, I think the mood is
> clearly to leave this as an operational decision.  In which
> case, taking into account the following recommendation ...
> 
>> While the patterns look similar, the first one rejects
>> non-compliant "EHLO 192.0.2.1" and similar dotted quads (or
>> more generally some mixture of digits and dots), the second
>> rejects RFC-compliant address literals.  So at least the
>> second message should probably be different, if the rule is
>> retained.
> 
> 
> 
> As to the question of data, we cannot say for certain that the
> rejected messages were all spam, but we have only received one
> complaint in 10 years and so we can reasonably assume this
> rule has not caused problems that need to be addressed.
>

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 17 Dec 2019 14:15:36 +0000

John I strongly believe you are in the rough here.

Among other things, "we" (for some definition of we) hire the secretariat and exec dir to do this day-to-day crap.

---

From: Alissa Cooper <[email-masked]>
Date: Tue, 17 Dec 2019 12:15:04 -0500

Hi John, all,

The principles that the IESG has laid out for spam control on IETF lists are available here: <https://www.ietf.org/about/groups/iesg/statements/spam-control-2008-04-14/>. These principles continue to guide spam control as implemented by the AMS IT team. The IESG is not involved in day-to-day decisions about how the principles are operationalized and was not involved at all in the handling of the ticket you cite below.

With Glen’s help this week I’ve come to understand the history here, which was unknown to me before. It seems there was some sort of committee or group that existed a decade ago. Once when they met, one of their discussion topics was spam. Many measures including the one discussed in this thread were proposed, considered, and implemented. I don’t know much else about the group but it does not exist now.

As you’ve seen from Jay’s email, he has taken the lead on operationalizing a response. Based on discussions with him and Glen I think they both know they can reach out to the IESG at any time if they have questions in interpreting the IESG statement above or any other IESG statements.

Best,
Alissa on behalf of the IESG

> On Dec 15, 2019, at 4:15 PM, John C Klensin <[email-masked]> wrote:
> 
> Hi.
>

---

From: Jay Daley <[email-masked]>
Date: Wed, 18 Dec 2019 07:55:26 +1300

Hi John

Thanks for taking the time to ask these questions and I hope my answers do not discourage you from continuing to ask such questions as it is important that my decisions, even operational ones. are not considered sacrosanct.  Alissa has answered the more general question of authority and background here, but to answer your specific questions:

> On 17/12/2019, at 3:59 PM, John C Klensin <[email-masked]> wrote:
> 
> Jay,
> 
> Three questions:
> 

No.  First, I don’t believe it is possible for me to monitor every IETF list that may impact operational management of our services.  Second, having been pointed to an extensive technical discussion I don’t believe it is my place to attempt to understand it and draw my own technical conclusions.  If there were a specific list or group the exists for discussion on matters of operational guidance then I would of course monitor that, but there isn’t and so I think it is entirely reasonable for me to base my call on the messages to this IETF list.  

> marginally, with IETF standards and the implications if that

That is not my job to do, that is for the community to make a call on, both in terms of what is conformance and what is not conformance, and what is the implication of that.

> and community consensus about them?

As I hope is clear from above I am not assessing the issues.  Alissa has explained the history of community guidance being provided, which we have followed for 10+ years and the IESG spam control principles, which we are compliant with.  The decision on consensus is therefore "is there consensus to overturn this long-standing practice?", which clearly there is not.  In other circumstances, such as if the call were the other way around (i.e. is there consensus to do X) then a different mechanism or approach could well be required, such as the lengthy consultation on the revised privacy statement.

kind regards
Jay

-- 
Jay Daley
IETF Executive Director
[email-masked]
+64 21 678840

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 Dec 2019 14:02:29 -0500

On 12/16/2019 4:46 PM, Jay Daley wrote:
> leave this as an operational decision.

Hi,

The operation decision appears to be inconsistent in its erroneous and 
RFC2821 violating application of an aggressive mail filter.

Let me state, that am implementer and commercial mail package, 
operators always have the last say and I am always for local policy 
considerations.

But here is I see it:

1) Yes, everyone agree the response text needs to be fixed up, but

2) It is in fact a violation of RFC2821/5321 specification when a 
rejection is applied by a server to a perfectly valid ip-literal per 
specification, and

3) It lacks consistency in its operational decision on what Client 
Mail/Host Names are rejected or accepted.

If a rejection is going to apply to ip-literals, hence enforcing a 
FQDN, then at the very least, it should validate the FQDN.

The mail.ietg.org servers appears to accept any FQDN including a 
existing FQDN which does not match the connecting IP address and a 
non-existing FQDNs:

The mail.ietf.org returns

EHLO [ip-literal]        <- not acceptable
EHLO santronics.com      <- acceptable, but non-matching IP::DOMAIN
EHLO foo.santronics.com  <- acceptable, but not existing domain

Yet, it does not validate the FQDN. Why?

Because we have a history of this FQDN check to be unreliable, hence 
why a MUST NOT reject on the basis of a HELO/EHLO check.

Finally, imeo, to prudent implementators who follow the IETF cogs to 
engineer consistent and logical protocols, it is expected the IETF 
mail servers themselves to also follow the same rules asked of them. 
If the operational decision remain to reject ip-literals but accept 
invalid FQDNs, then the specifications will need to change.  If the 
FQDN are now validity and rejection occurs for mismatches, then the 
specification needs to change.  So to me, its beyond being an 
operational decision.

The next implementer will read what is written it says MUST NOT, not 
SHOULD NOT, or MAY NOT.  We pay close attention to the protocol design 
elements.  The IETF should also.

Thanks

-- 
HLS

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 17 Dec 2019 14:24:43 -0500

> On Dec 17, 2019, at 2:02 PM, Hector Santos <hsantos=[email-masked]> wrote:
> 
> But here is I see it:
> 
> 2) It is in fact a violation of RFC2821/5321 specification when a rejection is applied by a server to a perfectly valid ip-literal per specification, and

It is not in fact.  A receiving MTA can refuse your email for any reason.
As a matter of RFC-compliance it MUST recognize address literals as
valid syntax (which it did by returning a 550 rather than 500 or 501),
but is then free to reject them on policy grounds.

> 3) It lacks consistency in its operational decision on what Client Mail/Host Names are rejected or accepted.

This is also not true.  It consistently rejects address literals because
doing so carries little risk of false positives (as already explained on
the ietf-smtp list, where the more technical discussion belongs).  "Real"
MTAs use domain names.  It is as simple as that.

> If a rejection is going to apply to ip-literals, hence enforcing a FQDN, then at the very least, it should validate the FQDN.

No, because enough "Real" MTAs use HELO domain names that don't map to
their own IP address, or any address at all.  So the risk of FPs is
too high.

There is no a priori discrimination here, it is all just based on what
one can get away with to reduce spam without blocking a non-trivial
volume of legitimate email.

> The mail.ietg.org servers appears to accept any FQDN including a existing
> FQDN which does not match the connecting IP address and a non-existing FQDNs:

As they must for operational reasons.

> Yet, it does not validate the FQDN. Why?

Because, much as one might want to, too many "Real" MTAs (sending
legitimate traffic) have FQDNs that would fail verification.

-- 
	Viktor.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 18 Dec 2019 09:40:35 +1300

On 18-Dec-19 06:15, Alissa Cooper wrote:
> Hi John, all,
> 
> The principles that the IESG has laid out for spam control on IETF lists are available here: <https://www.ietf.org/about/groups/iesg/statements/spam-control-2008-04-14/>. 
An interesting feature of that statement is that it contains this: "This supercedes [sic] a previous IESG statement on this topic, dated 9 Jan 2006." However, the hyperlink to "previous IESG statement" is incorrect as it is actually a link back to the identical 2008 statement. Applying my archaeological skills, the original statement is at:
https://www6.ietf.org/iesg/statement/spam-control-2006-01-09.html

It did mention "c. Past behavior by that particular sending IP address;" as a criterion. So this is not a recent addition by any means. I agree of course that it's a policy choice, not a standards issue.

As far as I can tell there was no specific committee; the 2006 statement was drafted by Bill Fenner who wrote to the IESG that "It's been discussed on the wgchairs list a few times, and this document is the result of the feedback."

(There's an even older version at
https://www6.ietf.org/IESG/STATEMENTS/mail-submit-policy-20020313.txt)

   Brian

> These principles continue to guide spam control as implemented by the AMS IT team. The IESG is not involved in day-to-day decisions about how the principles are operationalized and was not involved at all in the handling of the ticket you cite below.
>

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 Dec 2019 18:04:23 -0500

Viktor,

I believe we are in agreement with the same principles. But maybe not. 
Not sure. I already implied local policy always prevails.

That said, I believe the IETF "operational decision" was incorrect and 
inconsistent. To honor it, it may set a negative precedent for future 
implementer trust and confidence in protocol design.  Is it advocating 
others to follow? After all, an SDO did it, so why not across all 
systems?

We may have to explicitly embed this clause in the protocol:

      A "MUST NOT" protocol element MAY be overridden with "550" Policies.

That is the implication we have here.  Keeping in mind that these new 
550 policies are generally based on some "Bad Guy" presumptions. It 
did not explicitly exist before RFC5321. Previously, 550 was like a 
"Catch all" reply code.   Since 2003, with GreyListing, SPF, DKIM/SSP, 
DKIM/ADSP, DKIM/DMARC, etc, the beauty of these new protocols to help 
deal with "bad guys," they were ALL consistent with the written SMTP 
specifications.  They did not change the SMTP protocol except for one 
thing.  I was instrumental during RFC2821BIS in proposing and Klensin 
adding section 7.9. "Scope of Operation of SMTP Servers."   With the 
advent of the new DNS TXT-based applications, we needed new SMTP 
justifications for rejections.  The bar was raised for SMTP compliancy 
and correction.

    7.9.  Scope of Operation of SMTP Servers

    It is a well-established principle that an SMTP server may refuse to
    accept mail for any operational or technical reason that makes sense
    to the site providing the server.  However, cooperation among sites
    and installations makes the Internet possible.  If sites take
    excessive advantage of the right to reject traffic, the ubiquity of
    email availability (one of the strengths of the Internet) will be
    threatened; considerable care should be taken and balance maintained
    if a site decides to be selective about the traffic it will accept
    and process.

    In recent years, use of the relay function through arbitrary sites
    has been used as part of hostile efforts to hide the actual origins
    of mail.  Some sites have decided to limit the use of the relay
    function to known or identifiable sources, and implementations SHOULD
    provide the capability to perform this type of filtering.  When mail
    is rejected for these or other policy reasons, a 550 code SHOULD be
    used in response to EHLO (or HELO), MAIL, or RCPT as appropriate.

What it needs to update here is to include DATA because of ADSP and 
DMARC.

Again, now of the new augmented protocols did not change SMTP.

For me, being consistent allows for raising the bar.  In regards to 
the new ietf mail policy, if it was consistent with this new 550 
policy to reject legitimate ip literals for a yet to be explained 
non-reason, it would be more consistent and acceptable,  if it also 
consistently enforced a correct FQDN using a new 550 policy 
justification to reject IP::DOMAIN mismatches.  After all a "real" MTA 
is expected to use rDNS to obtain the correct FQDN for the sender 
machine.  No?  That is not always optimal. Small lite weight SMTP 
clients exist.  PTR slows them down.  What if there is no PTR record? 
   After all, again, more inconsistency, are we promoting PTR records 
now? For a certain period there, we were discouraging them, in fact, I 
think SPF tried to deprecate it. I am not going to bother to confirm 
but I recall the debates.

Consistency with everyone "expected" to play by the same rules is 
paramount for maximum interoperability and with minimum support. IMO, 
this mail.ietf.org odd "operational decision" could drastically change 
things because now others will follow.

I am still trying to understand why the legit IP-literal is being 
rejected in the first place.  I don't see any legit reason for it.  Do 
you?

Thanks

-- 
HLS

-- 
HLS

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 18 Dec 2019 00:40:44 +0000

>          A "MUST NOT" protocol element MAY be overridden with "550" Policies.
    
Bleh. We often say "we're not the protocol police."

I think it's always clear that local policy can override.

Is the IETF going to come and arrest it's own website?

---

From: Keith Moore <[email-masked]>
Date: Tue, 17 Dec 2019 20:50:43 -0500

> I think it's always clear that local policy can override.

Emphatically disagree.   We aren't the protocol police, but the idea 
that local policy can degrade interoperability at will and still comply 
with the standards makes no sense.

Spam filters are a good example.   Poorly chosen spam filters are the 
primary reason that email has become so unreliable.

I believe our job as protocol engineers is to define protocols that will 
work reliably without operators needing to second-guess them, not to 
support cargo cult practices that degrade interoperability.   If 
operators think they need to second-guess the standards, that's 
something we should take notice of - maybe we as engineers haven't done 
a good enough job at designing and maintaining our protocols.

Keith

---

From: "John Levine" <[email-masked]>
Date: 17 Dec 2019 21:07:25 -0500

>primary reason that email has become so unreliable.

Actually, spam is the primary reason that email has become so
unreliable.  If you did't do spam filtering, however imperfectly, your
mailbox would be so awash in junk you couldn't find the trickle of
real mail.

On my tiny system that handles a dozen users I deal with upwards of
50,000 spam messages a day.  I gather that for large commercial
systems it's in the billions.

R's,
John

---

From: Keith Moore <[email-masked]>
Date: Tue, 17 Dec 2019 21:37:20 -0500

> mailbox would be so awash in junk you couldn't find the trickle of

I did say "poorly chosen" spam filters.   I don't claim that all spam 
filters are poorly chosen, but I've seen many that are.

As it turns out, I have several mailboxes with no spam filtering, and 
they're not awash in junk.   But mileage does vary - I have other 
mailboxes that do have spam filtering and are still awash in junk.

Keith

---

From: George Michaelson <[email-masked]>
Date: Wed, 18 Dec 2019 13:13:08 +1000

I know this is on the 'your idea will not work because' [x] list. But,
I am going to put it out there, because it came up in X400
discussions. X400 people
discussed if the security tokens across the MTA-to-MTA path were
sufficient. And, they discussed cost recovery models because it was
consciously designing for utility scale deployment in nation-states,
in a time where sendmail was the norm, and mail was paid for by the
state, for education and research.

IF we had implemented sender-pays, the SPAM problem would be radically
different. It would still be there, and the existence proof is SMS
spam.

But, I believe the intensity would be different, and the chain back to
the cause would be different. Money talks. Money leaves trails. Money
would have exposed promiscuous senders to exposure of cost. We didn't
like to talk about money and so we painted money evil, but money was
always there: we just invented a virtual world, where other people did
money and we did things, as if money had no role to play.

SPAM filter on receive, is about the worst possible way to limit the
impact of spam. In other domains of problem in the internet we discuss
things like BCP38 which are also doomed, and also stand as existence
proof that it won't work, but the idea has significant merit in this
regard: you are asked to look at your customers, and try to limit
their bad behaviour facing outward. SPAM on ingress is dropping bad
things, after the gun has fired. BCP38 and like, is trying to stop the
gun being Fired.

there. I did it. I crossed the beams into Second amendment rights. Let
mayhem ensue...

[x] your idea will not work because we are scared to revisit the past.

---

From: Randy Bush <[email-masked]>
Date: Tue, 17 Dec 2019 19:59:20 -0800

> I think it's always clear that local policy can override.

that's a SHOULD NOT

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 Dec 2019 23:23:02 -0500

> Is the IETF going to come and arrest it's own website?

No. It should keep it simple - follow the standard protocol.

The bottom line is this breaks code, 37+ years of consistent protocol 
SMTP state machine logic, its not a bad vs good thing, regardless of 
the frequency of usage, regardless of scale, even the hobbyist, the 
entire community, in my long pre-internet protocol design and 
engineering world, a MUST concept is burned in, SHOULDs and MAYs are 
optional features provided to sysops/customers, your switches.

Do it this way, and the network/system homogeneous or heterogeneous, 
it all works.   Due to inconsistent application of some unknown rule, 
it broke a SMTP "standard" protocol.   In lieu of a specific black 
listing/block "trained thing," my concern this gem will most 
definitely begin to be copied and soon enough it will trap more 
clients using ip literals.

Its wrong. its that simple.  I am the OP. I filed the action item.  I 
only discovered it because the long time whitelisted mail.ietf.org 
server changed it's whitelisted ip address.  So a unsolicited mail 
sender CBV (Call Back Verification) was triggered. It failed due to 
the mail.ietf.org server violated the RFC2821 protocol. It had no 
legit reason for the rejection.

For the record, a CBV is the most effective spoof/fake reverse path 
filter method that exist.  I have 16 uses of daily statistics to prove 
it, yield 50-80% rejection rates near near-zero (hate to say none) 
false/true positive/negatives.

http://www.winserver.com/public/spamstats.wct

I will follow the cogs with this. Just do the right thing. I am doing 
whats prudent and practice for my compliant SMTP package ALWAYS. In 
fact, we filter on violations of the ip-literal mismatches. I am not 
going to reject legitimate SMTP compliant machine HELO/EHLO 
ip-literals. If my sysops and admins want to do that, thats up to them 
but its not coming from me, I am not going to encourage it because 
there is absolute no legitimate reason to do so and I think the 
mail.ietf.org admins, whoever, are wrong about this. It is unfair to 
Klensin and the mail community who work hard to get these things right.

I'm done.

--
HLS

---

From: Hector Santos <[email-masked]>
Date: Tue, 17 Dec 2019 23:23:45 -0500

>> I think it's always clear that local policy can override.

+1, exactly.

-- 
HLS

---

From: John C Klensin <[email-masked]>
Date: Tue, 17 Dec 2019 23:28:55 -0500

Brian and Alissa,

Glen pointed me/us to that spam policy statement and I read it
some weeks ago (as well as when it was posted).  What I found
notable about it in the context of the present discussion is
something Brian calls out: "c. Past behavior by that particular
sending IP address;".  Our not knowing whether the reason for
the rejection was specific to a particular IP address (whether a
sending one or one announced in EHLO) was exactly the reason I
thought that submitting the trouble ticket was in order.  If
Glen had responded "we didn't accept your mail transaction
because the relevant address had behaved badly in the past",
that would have been very clear, clearly within policy, and we
would not be having this discussion.   Instead, he told us that
the IETF was blocking _all_ mail sessions that contained an IP
address literal, a piece of syntax that is authorized, and (at
least normally) required to be accepted by an IETF Standards
Track specification, syntax with which we have had more than 37
years of wide deployment and experience.

I can find nothing in the anti-spam policy that says "if some
spammers are using a particular SMTP feature, we will ban the
feature" or even "if this SMTP feature appears to be used more
often by spammers, than by legitimate message senders, we will
ban the feature".  As long as no wording to that effect is
present, there may be good reasons for rejecting mail session
setup commands that contain IP address literals, but the
decision to reject them is not mandated by or even supported by
the policy statement.   

Somewhat cynically, I also note that, if the generally-held
belief that there are more spam messages on the Internet these
days than legitimate messages, an argument that something should
be banned because more spammers than legitimate senders use it
would be an argument for shutting down SMTP entirely.

Because there is nothing in the spam control principles
statement that authorized blocking all message transactions that
use IP address literals (or any other feature specifically
authorized by SMTP, especially ones associated with "MUST NOT
reject" language, this actually is a standards matter, not just
an operational policy matter.  It may be the latter too, but my
note to the IETF list was prompted by the former and the
question of whether, like Julius Caesar's supposed comment about
his wife, the IETF should be extremely observant and above
suspicion wrt the applicability of its own standards.

The questions of whether, if RFC 5321 said nothing on the
subject, IP address literals should be blocked, what the
anti-spam policy says and whether it should be updated to cover
this case are really interesting.  The technical aspects of them
have been discussed at considerable length (and, IMO, technical
depth) on the ietf-smtp list.   I tried to keep those
discussions from being repeated on the IETF list, in part
because of assorted policies about what belongs on this list
when  there are other lists dedicated to the particular
discussion area.  I don't believe that anyone who participated
actively in the discussion of the ietf-smtp list would claim
that the discussion that did occur on the IETF list represented
a comprehensive summary of that earlier (and continuing)
discussion among those with enough involvement in SMTP to be
following that list.

However, when I introduced a piece of the subject on the IETF
list, I was interested in only one question here.  That question
was whether the IETF was now ok with operational decisions that
put the IETF's practices out of conformance with IETF standards.
Whether one thinks that the 5321 requirement for acceptance of
IP address literal overrides or becomes subsidiary to its very
general statements about exceptions for operational purposes
affects whether this IP address literal issue is a sufficient
approximation to a smoking gun to make a good example or not,
but not the question.

The discussion led to an additional question and hence my "three
questions" note to Jay.  It is moot if we don't care whether
"we" conform to our standards or not (and, again, separate from
whether this is a good example of that or not).  Given his
response and Alissa's note, an updated version of that question
would be whether, if it appears to be operationally appropriate
for our practices to be non-conformant to our standards, how
does that decision get made and what procedures, if any, are
needed to be sure the IETF LLC and its contractors are aware of
possible standards non-conformance.  I note in that regard that
reasonable people might construe an IETF LLC or contractor (the
Secretariat included) decision to not conform to an IETF
standard as the LLC getting involved in the standards process,
something the community (or at least the IASA2 WG) was assured
would not happen.

best,
   john

p.s. Jay's response is much appreciated even though I see subtle
issues there that I infer from the response that he may not see.
And I still owe Rich a response to his comment claiming I'm in
the rough (and implying, I think, that I should just shut up).

--On Wednesday, December 18, 2019 09:40 +1300 Brian E Carpenter
<[email-masked]> wrote:

>> The principles that the IESG has laid out for spam control on
>> IETF lists are available here:
>> <https://www.ietf.org/about/groups/iesg/statements/spam-contr
>> ol-2008-04-14/>. 
> An interesting feature of that statement is that it contains
> this: "This supercedes [sic] a previous IESG statement on this
> topic, dated 9 Jan 2006." However, the hyperlink to "previous
> IESG statement" is incorrect as it is actually a link back to
> the identical 2008 statement. Applying my archaeological
> skills, the original statement is at:
> It did mention "c. Past behavior by that particular sending IP
> address;" as a criterion. So this is not a recent addition by
> any means. I agree of course that it's a policy choice, not a
> standards issue.
> 
> As far as I can tell there was no specific committee; the 2006
> statement was drafted by Bill Fenner who wrote to the IESG
> that "It's been discussed on the wgchairs list a few times,
> and this document is the result of the feedback."
> 
>> These principles continue to guide spam control as
>> implemented by the AMS IT team. The IESG is not involved in
>> day-to-day decisions about how the principles are
>> operationalized and was not involved at all in the handling
>> of the ticket you cite below.
>> 
>> With Glen's help this week I've come to understand the
>> history here, which was unknown to me before. It seems there
>> was some sort of committee or group that existed a decade
>> ago. Once when they met, one of their discussion topics was
>> spam. Many measures including the one discussed in this
>> thread were proposed, considered, and implemented. I don't
>> know much else about the group but it does not exist now.
>> 
>> As you've seen from Jay's email, he has taken the lead on
>> operationalizing a response. Based on discussions with him
>> and Glen I think they both know they can reach out to the
>> IESG at any time if they have questions in interpreting the
>> IESG statement above or any other IESG statements.
>> 
>> Best,
>>> lose all credibility in recommending to others that they
>>> follow our standards.  This practice has been referred to in
>>> many discussion threads over the years as "eating our own
>>> dog food".  
>>> 
>>> It has recently come to the attention of several of us, via
>>> an extended discussion on the SMTP list, that the IETF email
>>> servers are rejecting all SMTP connections whose EHLO
>>> commands contain IP address literals.   While the text
>>> describing the appropriateness of use of IP literal is RFC
>>> 5321 is more complicated than it probably ought to be, the
>>> discussion in Section 4.1.4 of that document seems quite
>>> clear that an SMTP server MUST NOT reject a message simply
>>> because an IP address literal (or a domain name that does
>>> not point to a host) is used. Those interested in the
>>> niceties of that issue should review the correspondence on
>>> the [email-masked] list and comment there if appropriate.
>>> 
>>> the month about the ietf.org mail servers apparently
>>> rejecting messages with IP address literals in the EHLO
>>> field.  The rejection is accompanied by a reply message that
>>> appears to be inappropriate in multiple ways; again, those
>>> interested should see the ietf-smtp list for an
>>> already-extensive discussion.  The Secretariat responded by
>>> indicating that all such addresses were being rejected and
>>> that the rejection was occurring under instructions from
>>> IETF leadership, instructions that were reaffirmed after the
>>> ticket was filed.  Whatever the problem is, and indeed,
>>> whether there is a problem, the Secretariat is therefore
>>> blameless.  I suggest that the IETF has a problem.
>>> 
>>> technical issues, what should be done about them, or whether
>>> the text in RFC 5321 should be improved.  Those, it seems to
>>> me, are topics for the ietf-smtp list.   They have been
>>> discussed there at length and presumably will continue to be
>>> discussed there. It is whether there is consensus among IETF
>>> participants that "the leadership" (I presume whatever
>>> bodies, individuals, or their designees are relevant) should
>>> have the authority to instruct the Secretariat to violate an
>>> IETF standard without consultation of appropriate experts
>>> within the community (presumably on relevant mailing lists),
>>> evidence of IETF rough consensus, and/or Internet Drafts
>>> that specify alterations to the relevant standard(s).  I
>>> also don't want to cast blame about decisions of the past,
>>> only to understand what the process is for giving
>>> instructions to the Secretariat (or approving their

---

From: Eliot Lear <[email-masked]>
Date: Wed, 18 Dec 2019 09:12:23 +0100

Hi John,

On the general issue, as the late Brian Kantor said, RFCs serve us best when they document existing practice.  This is entirely in accord with the “running code” part of our mantra: if network needs dictate that the RFC not be followed to the letter, then so be it.  I would actually like that we are demonstrating this point and reenforcing that our standards are voluntary, but for the fact that I don’t think the standard was substantially violated.  That’s my second point.

If one looks at RFC 5321, there are three reasons to think that really the standard does not prohibit the behavior being described.  First, the text in 4.1.4 doesn’t actually prohibit the rejecting literals.  Here is what is said:

   If the EHLO command is not acceptable to the SMTP server, 501, 500,
   502, or 550 failure replies MUST be returned as appropriate.

At this point in the transaction, it may not be readily apparent that the EHLO indeed is unacceptable.  That’s because the server may need to gather more information, such as the recipient, in order to check against SPF records or other inputs delivered later to make a decision.

   An SMTP server MAY verify that the domain name argument in the EHLO
   command actually corresponds to the IP address of the client.
   However, if the verification fails, the server MUST NOT refuse to
   accept a message on that basis.

That “MUST NOT” conflicts with the most important principle we have to go with in order to defend against spam and malware, as clearly stated in Section 7.9:

    It is a well-established principle that an SMTP server may refuse to
   accept mail for any operational or technical reason that makes sense
   to the site providing the server.

And so the server chose to reject a message.  I would add that an erratum should be filed to resolve this conflict.

Finally, we cannot CANNOT CANNOT micromanage secretariat and mailing list operations on the IETF list. This does NOT impact open participation, when anyone can get free email service on any number of platforms that work just fine with the IETF.  That this message didn’t meet the extremely low barrier of setting up a PTR record when > 99% of SMTP client connections are best classed as attacks.  That THAT isn’t recognized as THE problem the IETF should work on with regard to email is disturbing.

Eliot

> On 16 Dec 2019, at 18:50, John C Klensin <[email-masked]> wrote:
> 
> 
>

---

From: Alissa Cooper <[email-masked]>
Date: Wed, 18 Dec 2019 05:56:27 -0500

Hi John,

I have some personal thoughts below, speaking as an individual participant.

> On Dec 17, 2019, at 11:28 PM, John C Klensin <[email-masked]> wrote:
> 
> Brian and Alissa,
> 

No one has claimed that enforcement of these anti-spam rules is mandated by the IESG statement. But the rules are consistent with the statement. The whole point of a high-level statement like this is to provide guardrails for the support staff to operate within, not to dictate in detail what they can or cannot do.

> possible standards non-conformance.  

The two tools the community has for articulating policy of this sort are consensus RFCs and IESG statements. Personally I think a policy that would require every aspect of our IT operation to be fully standards-compliant would put us out of touch with reality and make it even more difficult to run the operation and serve the needs of our community than at present. In the real world operators find themselves in situations where the overall benefits to their own users of deviation or non-implementation of a standard outweigh the drawbacks, and we too live in that world. I will note that even the IETF tools team that largely develops its tools from scratch expresses a preference but not a requirement for its tools to be standards-based: https://ietf.org/about/groups/tools/.

I also believe that our IT staff know more about IETF standards conformance and non-conformance than any IT staff in the world. The suggestion to consider additional procedures to educate them belies all they do already to understand how to operationalize the technologies that we standardize.

> I note in that regard that

I disagree that this would be a reasonable conclusion. The standards process is specified in BCP 9 and operational decisions about how the IETF’s IT systems work are not within the scope of that BCP.

Best,
Alissa

---

From: "John R Levine" <[email-masked]>
Date: 18 Dec 2019 09:21:12 -0500

> different. It would still be there, and the existence proof is SMS

I wrote a white paper on e-postage in 2004.  Nothing has changed since 
then other than that some of the numbers would have a few more zeros.

https://www.taugh.com/epostage.pdf

tl;dr:

Nobody knows how to run a payment system for billions of messages a day

Nobody knows how to pay for a payment system for billions of messages a day

Nobody knows how to manage a payment system for billions of messages a day

Typical problem: botnet takes over Grandma's computer and sends a spam 
blast.  Does she pay the postage?  If not, who does?  If the answer is 
nobody does, how is that an e-postage system?  Keep in mind that every 
spammer in the world will claim to be a botted grandmother (many already do.)

R's,
John

PS: The current STIR/SHAKEN mess reminds us that settlements in the phone 
system no longer work either.

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 18 Dec 2019 15:46:53 +0000

I overstated, sorry.

On 12/17/19, 11:23 PM, "Hector Santos" <[email-masked]> wrote:

    +1, exactly.

    -- 
    HLS

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 18 Dec 2019 14:12:05 -0500

If we knew how to deploy such a radical change as sender pays, we would
surely also know how to do the much easier task of replacing SMTP with a
protocol that is secure by default.

* Every message is signed by the sending client.
* Every message is signed by the originating mail service.
* Every mail receiving service performs access control on inbound messages
* Every mail client performs access control on messages

Messaging abuse isn't entirely absent on Facebook, Skype, Signal, etc. but
it is virtually non-existent compared to telephone and email. We are
rapidly reaching the point where a large number of customers are going to
start unplugging from POTS and only use it for voice mail because the level
of abuse is utterly insane.

Subjecting every message to access control is pretty straightforward when
you can start with the principle that every message is authenticated by the
sender. Defining a set of access control rules that work for me is pretty
straightforward.

1) I will accept a contact request of 200 characters or less from anyone

2) I will accept requests from anyone in my contact book that are
compatible with the authorizations specified there (e.g. Alice can send me
mail or voice, Bob only voice, Carol can send me code, etc).

3) I will accept messages from anyone who has attended an IETF meeting or
is a member of an an affinity group I am a member of (school, university,
etc. etc.)

4) Reject everything else

Trying to shoehorn this into the legacy SMTP environment is tough because
the default is insecure. But there are plenty of closed environments that
don't use SMTP which could switch to another messaging protocol.

While I was writing this, I was interrupted by the Nest telling me
something I didn't need to know. There is also a 'secure' messaging center
on the Nest site and I have the same for each of my banks, brokers, etc.
Wouldn't it be nice if all of those could send me messages using a protocol
they know is secure and meets their HIPPA, GDPR, MMQF, etc. requirements?

As John points out the telephone system is collapsing. Most of the
complexity goes into collecting payments at the per call level that are
irrelevant in a world that is moving to flat rate.

---

From: Keith Moore <[email-masked]>
Date: Wed, 18 Dec 2019 18:03:17 -0500

On 12/18/19 3:12 AM, Eliot Lear wrote:

> On the general issue, as the late Brian Kantor said, RFCs serve us 
> best when they document existing practice.

I do not think this is defensible as a general statement, at least not 
when stated in that way.   There are cases when this is true, such as 
when existing practice has evolved to figure out what actually works 
well, and RFCs have evolved to document it. But that neglects the very 
purpose of engineering of our protocols, which is to determine what will 
work well before it becomes existing practice.   Engineering is 
imperfect, of course, which is why we revise those RFCs in light of 
experience, but good engineering produces good practice much more often 
than not.

Sometimes our first RFC documents a protocol that is already in use, in 
which case we have a dilemma - should the RFC document existing practice 
or should it document what is believed to be good practice?   I have 
seen many instances in which this conflict produces poor specifications, 
in which either documenting existing practice, or specifying what is 
believed to be good practice, would produce a more useful result than 
trying to do both.   It is imperative that the WG in such a case have 
clear instructions as to which it should do.   But it's not always the 
case that the WG should favor existing practice.

Another problem with that statement is that, if taken as true on its 
face, _any_ existing practice, no matter how poorly chosen or rarely 
employed, can be used to argue for ignoring an RFC or changing text in 
the RFC when it is revised.

In the case of SMTP, of course, many changes made between RFC 821 and 
RFC 5321 reflect lessons learned in the ~26 years of experience with 
SMTP during that interval, just as RFC 821 reflected several years of 
experience with email-over-FTP prior to the invention of SMTP.    The 
rule in RFC 5321 that permits an IP address literal in HELO/EHLO is one 
such example.   Servers that rejected messages based on HELO/EHLO were 
found to frequently reject valid mail.   Perhaps unfortunately, RFC 5321 
only reflects the change learned from that experience and does not 
record the experience that led to that decision - so people today may 
naively believe that the language in 5321 is a mistake or accident.   It 
was neither of those.

Email usage does continue to change, so a decision should not be taken 
as valid forever just because it was once sound.   This is equally true, 
BTW, for the language in RFC 5321, as it is for an operational change to 
reject EHLO with IP address literals.

I do support empowering operators, in exigent circumstances, to do 
whatever appears to be necessary to allow successful operation to 
continue.  But the process should not stop there.   When the effect of 
such decisions accumulates over time, and the corrective measures 
continue to be justified because "they've been that way forever", the 
result is generally to degrade the ability of the network to support 
applications and changing usage patterns.   So IMO there's a need to 
document the reasons for such measures, and to measure and track their 
effectiveness over time.   And because this check happens before the 
message content is actually transmitted, the only way to measure 
effectiveness of a rejection on EHLO arguments is to let some 
statistically representative sample of such messages through from time 
to time and actually measure how valid a check it turns out to be.

> This is entirely in accord with the “running code” part of our mantra: 
> if network needs dictate that the RFC not be followed to the letter, 
> then so be it.  I would actually *like* that we are demonstrating this 
> point and reenforcing that our standards are voluntary, but for the 
> fact that I don’t think the standard was substantially violated. 
>  That’s my second point.
>
This is not "running code" in the sense that we usually use the term.   
It's not a change made by a well-supported SMTP server. It's a 
configuration change made by one site for which very little evidence to 
support it has been cited.   Even then, the assertion was that this 
configuration was more efficient, not that it resulted in a better 
overall ham-to-spam ratio.   And of course IETF is not just an ordinary 
site, it sets (or should set) an example for others to follow.

>
> If one looks at RFC 5321, there are three reasons to think that really 
> the standard does not prohibit the behavior being described.  First, 
> the text in 4.1.4 doesn’t actually prohibit the rejecting literals. 
>  Here is what is said:
>
> At this point in the transaction, it may not be readily apparent that 
> the EHLO indeed is unacceptable.  That’s because the server may need 
> to gather more information, such as the recipient, in order to check 
> against SPF records or other inputs delivered later to make a decision.
I disagree.   The EHLO argument is immaterial to any check on validity 
of a recipient address.   And one subtlety of SMTP is that a response to 
a RCPT TO command is generally taken as a response for that specific 
recipient, so it's generally not appropriate to complain about EHLO in 
response to RCPT TO.
> That “MUST NOT” conflicts with the most important principle we have to 
> go with in order to defend against spam and malware, as clearly stated 
> in Section 7.9:
>

The MUST NOT exists for the very reason that rejection of messages 
because of EHLO arguments was found to be detrimental to 
interoperability, because too many SMTP servers were conducting 
inappropriate checks.

>
> And so the server chose to reject a message.  I would add that an 
> erratum should be filed to resolve this conflict.

I disagree.   Or at least, I don't think IETF should dismiss the many 
person-decades of experience that went into RFC 5321, based on a single 
assertion by one operator that they found it beneficial to do so.   I do 
think IETF should probably research the matter, but I'm not sure that an 
erratum is appropriate.

> Finally, we cannot CANNOT CANNOT micromanage secretariat and mailing 
> list operations on the IETF list.

I mostly agree, but this is still a red flag.   When the secretariat or 
operators believe they see a need to violate IETF consensus, they should 
at least explain to IESG why they are doing so, and IESG should refer 
the matter to the appropriate area or working group for 
investigation.    Partially because this is potentially valuable 
feedback for IETF standards, but also because it reflects poorly on our 
organization and impairs the perceived value of our standards when we 
refuse to eat our own dog food.

And it should be fine to discuss these things on the IETF list or on an 
appropriate topical IETF mailing list.   But neither the IETF list nor 
an IETF WG should not be trying to issue any kind of immediate 
instructions to IETF operations personnel, unless asked to do so by the 
IESG.

> This does NOT impact open participation, when anyone can get free 
> email service on any number of platforms that work just fine with the 
> IETF.  That this message didn’t meet the extremely low barrier of 
> setting up a PTR record when > 99% of SMTP client connections are best 
> classed as attacks.

As I understand the situation, that wasn't the problem. Messages 
were/are being rejected merely because they used IP address literals in 
EHLO, whether or not there was a PTR record.

> That THAT isn’t recognized as THE problem the IETF should work on with 
> regard to email is disturbing.

To me the disturbing thing is that so many operators feel empowered to 
impose arbitrary and often meaningless checks on email in the name of 
filtering supposed spam, checks that are quite often not backed up by 
careful and continued measurement and appropriate use of statistics.    
But again, this may be due to a failure of IETF to make appropriate 
protocol changes and/or operational recommendations to facilitate more 
accurate spam filtering.    IMO the "operators can do whatever they 
want" idea has degraded interoperability for far too long, but we can't 
really expect operators to do better unless we give them better tools.

Keith

---

From: Hector Santos <[email-masked]>
Date: Wed, 18 Dec 2019 20:42:01 -0500

+1 Well said Keith. It represents my view as well. Thank you.

On 12/18/2019 6:03 PM, Keith Moore wrote:
> purpose of engineering of our protocols, which is to determine what
> will work well before it becomes existing practice.   Engineering is
> experience, but good engineering produces good practice much more
> often than not.
>
> Sometimes our first RFC documents a protocol that is already in use,
> in which case we have a dilemma - should the RFC document existing
> practice or should it document what is believed to be good practice?
> I have seen many instances in which this conflict produces poor
> specifications, in which either documenting existing practice, or
> specifying what is believed to be good practice, would produce a more
> useful result than trying to do both.   It is imperative that the WG
> in such a case have clear instructions as to which it should do.   But
> it's not always the case that the WG should favor existing practice.
>
> rule in RFC 5321 that permits an IP address literal in HELO/EHLO is
> one such example.   Servers that rejected messages based on HELO/EHLO
> were found to frequently reject valid mail.   Perhaps unfortunately,
> RFC 5321 only reflects the change learned from that experience and
> does not record the experience that led to that decision - so people
> today may naively believe that the language in 5321 is a mistake or
> accident.   It was neither of those.
>
> as valid forever just because it was once sound.   This is equally
> true, BTW, for the language in RFC 5321, as it is for an operational
> change to reject EHLO with IP address literals.
>
>> This is entirely in accord with the “running code” part of our
>> mantra: if network needs dictate that the RFC not be followed to the
>> letter, then so be it.  I would actually *like* that we are
>> demonstrating this point and reenforcing that our standards are
>> voluntary, but for the fact that I don’t think the standard was
>> substantially violated.  That’s my second point.
>>
> This is not "running code" in the sense that we usually use the
> term.   It's not a change made by a well-supported SMTP server. It's a
> configuration change made by one site for which very little evidence
> to support it has been cited.   Even then, the assertion was that this
> overall ham-to-spam ratio.   And of course IETF is not just an
> ordinary site, it sets (or should set) an example for others to follow.
>
>>
>> If one looks at RFC 5321, there are three reasons to think that
>> really the standard does not prohibit the behavior being described.
>>  First, the text in 4.1.4 doesn’t actually prohibit the rejecting
>> literals.  Here is what is said:
>>
>> At this point in the transaction, it may not be readily apparent
>> that the EHLO indeed is unacceptable.  That’s because the server may
>> need to gather more information, such as the recipient, in order to
>> check against SPF records or other inputs delivered later to make a
>> decision.
> of a recipient address.   And one subtlety of SMTP is that a response
> to a RCPT TO command is generally taken as a response for that
> specific recipient, so it's generally not appropriate to complain
> about EHLO in response to RCPT TO.
>>
>> That “MUST NOT” conflicts with the most important principle we have
>> to go with in order to defend against spam and malware, as clearly
>> stated in Section 7.9:
>>
> person-decades of experience that went into RFC 5321, based on a
> single assertion by one operator that they found it beneficial to do
> so.   I do think IETF should probably research the matter, but I'm not
> sure that an erratum is appropriate.
>
> I mostly agree, but this is still a red flag.   When the secretariat
> or operators believe they see a need to violate IETF consensus, they
> should at least explain to IESG why they are doing so, and IESG should
> refer the matter to the appropriate area or working group for
> feedback for IETF standards, but also because it reflects poorly on
> our organization and impairs the perceived value of our standards when
> we refuse to eat our own dog food.
>
> And it should be fine to discuss these things on the IETF list or on
> an appropriate topical IETF mailing list.   But neither the IETF list
> nor an IETF WG should not be trying to issue any kind of immediate
> instructions to IETF operations personnel, unless asked to do so by
> the IESG.
>
>> email service on any number of platforms that work just fine with
>> the IETF.  That this message didn’t meet the extremely low barrier
>> of setting up a PTR record when > 99% of SMTP client connections are
>> best classed as attacks.
>
> were/are being rejected merely because they used IP address literals
> in EHLO, whether or not there was a PTR record.
>
>> That THAT isn’t recognized as THE problem the IETF should work on
>> with regard to email is disturbing.
>
> careful and continued measurement and appropriate use of
> statistics.    But again, this may be due to a failure of IETF to make
> appropriate protocol changes and/or operational recommendations to
> facilitate more accurate spam filtering.    IMO the "operators can do
> whatever they want" idea has degraded interoperability for far too
> long, but we can't really expect operators to do better unless we give
> them better tools.
>
> Keith
>
>

-- 
HLS

---

From: John C Klensin <[email-masked]>
Date: Wed, 18 Dec 2019 23:42:11 -0500

Eliot,

I'm obviously not being clear enough about what I'm concerned
about and looking for, so I should stop commenting for a while.
Before I do and in line with your note...

(1) If an issue has already been discussed, at length and by
participants who are well-informed on the issues, on a
topic-specific IETF mailing list, making decisions based only on
a much more abbreviated discussion on the IETF list is probably
not a good idea.  While we have historically not made a
distinction between non-WG lists that are created for
discussions by a person or two and non-WG lists that are the
result of WG efforts that resulted in standards and that were
explicitly left open for discussions as issues with those
standards arise, perhaps we should do so.   In that regard, I
applaud Warren's and the IESG's decision to distinguish, by
name, between affinity group lists and lists on which it is
sensible to talk about IETF consensus.

(2) I believe that my continued pushing back on making technical
decisions on the IETF list (or the Last Call list), other than
in response to well-formulated Last Calls when there has already
been extensive discussion on a specialized ex-WG list,
especially if there has been no effort at a careful and balanced
summary of that prior discussion, is a bad idea from a technical
standpoint.  I believe that is true whether there is an issue in
which the Secretariat is involved or not.  I think that position
is entirely consistent with the policy statements about the IETF
list, SAA efforts to push technical discussions onto more
specialized lists, etc. (including IAB decisions to push
strategic discussions of RFC Editor Function futures onto
rfc-interest as reflected in Ted's recent notes).  Personally, I
don't like some of the implications of those policies and have
said so, but I'm trying to conform and think others should too.

(3) I don't believe anyone is proposing micromanagement in this
area, just somewhat more awareness of what we are doing.  Let me
borrow an idea from your note as a possible suggestion in that
area.  Suppose we decide that operational decisions that diverge
from IETF standards should be associated with errata (or
announcements to ex-WG lists, or both) indicating that the
standard has gotten out of touch with reality and practice.
That would completely meet my needs, at least if discussion of
those errata on relevant mailing lists was not prohibited.
However, it does imply two other issues about which we should be
aware: (i) Most such errata are (IMO completely appropriately)
going to get marked "hold for document update" and then ignored
until a decision is made to revise the document for other
reasons.  However, an AD, a document author, or an ordinary IETF
participant who happens to notice the erratum may bring the
erratum up as an issue on the relevant ex-WG mailing list,
perhaps even to stimulate a discussion of whether it is time to
revise the document or produce a new document updating it.
Should a discussion occur as a result and relatively clear
consensus emerge that the erratum is incorrect and the implied
requested change inappropriate, I would hope that there would be
a reasonable way to communicate that result to the Secretariat
and/or the IETF LLC ExecDir and would hope that the earlier
decision would be reviewed and revised as appropriate.   (ii) I
do not believe that the job descriptions of either the
Secretariat or the IETF LLC ExecDir includes evaluating
conformance with IETF Standards or the skills and background
needed to do so.  If we want these kinds of decisions, including
any requirement to generate errata, to be delegated to them,
then it is probably appropriate to review and update the
relevant contracts and job descriptions and make resource
adjustments as appropriate.  Again, this has nothing to do with
micromanaging anything or anyone.  It has to do with being aware
of what we are doing.

(4) If we are going to honor the principle that IETF standards
(and/or RFCs generally) should be consistent with existing (and
changing) practices in the field (whether in Brian's formulation
or otherwise), we need either a better and more systematic way
to notify the community (or the subset of the community
concerned about a particular standard) than waiting for someone
to complain or generate trouble tickets.  Having the Secretariat
and/or IETF LLC ExecDir be sensitive to when they are departing
from the standard and notify someone would be one way.  Your
idea about errata might be another.  Another would be to shift
to a model similar to the one ISO and many of its National
Member Bodies adopted some years ago in response to very similar
problems.   In their case, there is a required review of every
standard on a three-year cycle after which it must be either
revised, explicitly reaffirmed, or withdrawn.  

In any event, at the time RFC 5321 was published, it represented
the rough consensus among those involved, affirmed in an IETF
Last Call that was somewhat more active and technically-specific
than I think has been the norm lately, that it was closely
aligned with existing practice.   So no problem with Brian's
observation or anything else at that time.   Prior to a
discussion that started only a month or two ago, there has been
no serious consideration of revising it in the subsequent eleven
plus years.  So, again, if you don't believe the way decisions
are made about what the IETF servers will allow needs tuning,
and even if you are not bothered by those servers departing from
IETF Standards, it seems to me that we have an issue with the
absence of a systematic process for being sure that our
standards remain current with changing practices.

(5) Your analysis of the reasons that a response to an IP
address literal in an EHLO command is interesting.  It is a bit
inconsistent with the command-response model of RFC5321, but,
perhaps, if there were evidence that what is being done
represented very common practice, that would be irrelevant.
However, there is no such evidence and, perhaps more to the
point, it is not what Glen says the IETF servers are doing --
they are rejecting _all_ EHLO commands with IP address literals
as the argument but, for reasons I haven't asked him about,
deferring the responses until after a RCPT command is received.
Such deferred responses may be consistent with pipelining, but
pipelining was not requested in this case.  Again, it seems to
me that this is a topic for the ietf-smtp list, not the main
IETF one for the reasons given above.

(6) Finally, I think we need to be cautious about saying that it
is ok if conforming SMTP clients don't work with the IETF
servers because people can always find free email services that
work with those servers.  In general, we know who the most
obvious of those free email services are.  Several of them are
not strictly standards-conforming (although they may involved in
a sufficient fraction of Internet mail traffic that an extreme
interpretation of Brian's statement, one that I think Brian
would not have agreed to, is that _they_ are the standard, not
whatever the IETF specs have to say), they raise some of the
issues about concentration that the IAB has been trying to
address, and they raise some privacy concerns as well (even
though I personally believe that nothing said to the IETF that
might influence a standard should be treated as private or
allowed to be anonymous).  More generally, I don't find "a
workaround exists" to be a satisfactory answer if we are going
to claim to be in the standards business.

best regards,
   john

--On Wednesday, December 18, 2019 09:12 +0100 Eliot Lear
<[email-masked]> wrote:

> Hi John,
> 
> On the general issue, as the late Brian Kantor said, RFCs
> serve us best when they document existing practice.  This is
> entirely in accord with the "running code" part of our
> mantra: if network needs dictate that the RFC not be followed
> to the letter, then so be it.  I would actually like that we
> are demonstrating this point and reenforcing that our
> standards are voluntary, but for the fact that I don't think
> the standard was substantially violated.  That's my second
> point.
> 
> If one looks at RFC 5321, there are three reasons to think
> that really the standard does not prohibit the behavior being
> described.  First, the text in 4.1.4 doesn't actually
> prohibit the rejecting literals.  Here is what is said:
> 
>    If the EHLO command is not acceptable to the SMTP server,
> 501, 500,    502, or 550 failure replies MUST be returned as
> appropriate.
> 
> At this point in the transaction, it may not be readily
> apparent that the EHLO indeed is unacceptable.  That's
> because the server may need to gather more information, such
> as the recipient, in order to check against SPF records or
> other inputs delivered later to make a decision.
> 
>    An SMTP server MAY verify that the domain name argument in
> the EHLO    command actually corresponds to the IP address of
> the client.    However, if the verification fails, the server
> MUST NOT refuse to    accept a message on that basis.
> 
> That "MUST NOT" conflicts with the most important
> principle we have to go with in order to defend against spam
> and malware, as clearly stated in Section 7.9:
> 
>     It is a well-established principle that an SMTP server may
> refuse to    accept mail for any operational or technical
> reason that makes sense    to the site providing the server.
> 
> And so the server chose to reject a message.  I would add that
> an erratum should be filed to resolve this conflict.
> 
> Finally, we cannot CANNOT CANNOT micromanage secretariat and
> mailing list operations on the IETF list. This does NOT impact
> open participation, when anyone can get free email service on
> any number of platforms that work just fine with the IETF.
> That this message didn't meet the extremely low barrier of
> setting up a PTR record when > 99% of SMTP client connections
> are best classed as attacks.  That THAT isn't recognized as
> THE problem the IETF should work on with regard to email is
> disturbing.

---

From: Eliot Lear <[email-masked]>
Date: Thu, 19 Dec 2019 09:16:23 +0100

> On 19 Dec 2019, at 05:42, John C Klensin <[email-masked]> wrote:
> 
> Eliot,
> 

Agree, although I think Warren’s efforts are a bit separate.

> said so, but I'm trying to conform and think others should too.

I understand.  However, sometimes the devil is in the detail.  For the record, I disagree with the view that the LCs should take place on a separate list, but that is a separate discussion.

> needed to do so.

If someone is running a mail service, the responsibility falls on them to manage it.  How they manage it on aggregate is something that should occasionally be evaluated by the ExecDir.  Whether IETF standards are followed is entirely secondary to whether the service is properly managed and providing appropriate utility to the community.  Indeed, if the community wants to worry about whether these standards are being properly followed, it falls on us and not the secretariat to note when they are and when they are not, and take what standards actions we deem appropriate as a result (this also addresses your point 4).

As a customer of the service, you get to complain if there is a problem.  You did so.  Good.  That leads to an escalation path when things go wrong.  But let’s please separate the service not functioning as you think it should from whether the standards are being applied as you envisioned.  That is the main thrust I am trying to convey.

> plus years.

Fair enough.  That is a standards decision for the community.  It takes time and effort (I’ll note mostly your generous time and effort) to update these documents, and not all errata either individually or in aggregate rise to the need to do a document update.  But that doesn’t mean the erratum isn’t something to correct.

> IETF one for the reasons given above.

Yeah, we can take the specifics off the list.  I gave a rational reason as to why someone might choose to reject later, not that this was the logic being used in this specific case.

> allowed to be anonymous).

I would agree with you– and I share some of the IAB concerns on this point.  However, my point was really this, and I’d like it not swept aside:

It is not a simple task to run an MTA these days for all the reasons I previously mentioned. The low bar here for communicating with this particular server was having a PTR record, and the standards community does not get to second guess the person who set that low bar.  We only get to note that they did so and to decide whether it is a counter-example to what our voluntary standards specify.

If you can’t manage to get a PTR into the DNS, there are a great many options that include, but are not limited to using a free or pay-for mail service.  You can still build your own, but you may have to use a data center that agrees to update the PTR record for you (this is what I have done).  A mechanism even exists to accomplish this with EC2 at very nominal prices (~USD $10/month)

Eliot

---

From: Alessandro Vesely <[email-masked]>
Date: Thu, 19 Dec 2019 10:00:21 +0100

On Thu 19/Dec/2019 09:16:23 +0100 Eliot Lear wrote:
> If you can’t manage to get a PTR into the DNS, there are a great many options
> that include, but are not limited to using a free or pay-for mail service.  You
> can still build your own, but you may have to use a data center that agrees to
> update the PTR record for you (this is what I have done).  A mechanism even
> exists to accomplish this with EC2 at very nominal prices (~USD $10/month)

That's hardly acceptable.  For one point, being forced to virtualize the mail
server, to locate it in a remote data center, or to outsource its service
tout-court is not the kind of solution to be encouraged.  It exacerbates the
trend toward few giant email operators.

I respect IETF's decision to outsource the management of its mailing lists.  On
the other hand, the standard should respect the will to keep mail servers in
house, and try and ease doing so at current market conditions.

Best
Ale
--

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 19 Dec 2019 04:09:21 -0500

On Thu, Dec 19, 2019 at 10:00:21AM +0100, Alessandro Vesely wrote:

> > If you can’t manage to get a PTR into the DNS, there are a great
> > many options that include, but are not limited to using a free or
> > pay-for mail service.  You can still build your own, but you may
> > have to use a data center that agrees to update the PTR record for
> > you (this is what I have done).  A mechanism even exists to
> > accomplish this with EC2 at very nominal prices (~USD $10/month)
> 
> That's hardly acceptable.  For one point, being forced to virtualize
> the mail server, to locate it in a remote data center, or to outsource
> its service tout-court is not the kind of solution to be encouraged.
> It exacerbates the trend toward few giant email operators.

The actual reported problem does not in fact require a client MTA to
have a PTR record for its IP address, all that the client needs to do is
send a HELO name that resembles an FQDN.

Mind you an MTA that lacks a PTR for its IP address will be reject by
pretty much every peer MTA on the Internet.  The ietf.org lists would be
the least of its problems.  

Given that said PTR is published, its target may as well be the HELO
name, there's no reason to take a principled stand on one's moral right
to use an address literal, we have more important issues to consider.
Over and out.

-- 
    Viktor.

---

From: Eliot Lear <[email-masked]>
Date: Thu, 19 Dec 2019 11:07:10 +0100

> On 19 Dec 2019, at 10:00, Alessandro Vesely <[email-masked]> wrote:
> 
> That's hardly acceptable.

Alessandro, I respect your opinion,  but we don’t get to decide that for others. The person who gets to decide whether a PTR is required is the MTA administrator and nobody else.  I take the point about market concentration, but that has to be balanced against other operational considerations, and it will likely lose every time.

Eliot

---

From: Hector Santos <[email-masked]>
Date: Thu, 19 Dec 2019 12:18:53 -0500

On 12/19/2019 5:07 AM, Eliot Lear wrote:
> Alessandro, I respect your opinion,  but *we* don’t get to decide that
> for others. The person who gets to decide whether a PTR is required is
> the MTA administrator and nobody else.  I take the point about market
> concentration, but that has to be balanced against other operational
> considerations, and it will likely lose every time.
>
> Eliot

Hi Eliot,

I think we need to take a step back and recognize the developer vs 
administrative conflicts and the contention this particular ip-literal 
issue is highlighting.

For nearly four decades, we asked implementators to follow IETF 
guidelines and we did. Code is burned in and for SMTP, the ip literals 
is an acceptable base component of the protocol.  It is part of the 
base SMTP engine and state machines, so even if all the guidelines are 
followed 100%, what has occurred here, breaks the protocol, change 
code.

The impact may be small, but nonetheless, there are violations of the 
specs on both side.

So who takes responsibility, what are we asking of implementators?

Technically, the mail.ietf.org is RFC2821 correct in one respect.  If 
the Client has any public FQDN for the machine, then per spec it MUST 
use one. Per spec, IFF there are no public FQDN, then it MUST use a 
squared bracket IP-literal.

Thats that rule and I will 100% accept it because I can't argue it if 
it was the reason for the mail.ietf.org rejection.  My MTA did not use 
an FQDN when in fact there is one.

But the SMTP specs also say, it MUST NOT reject on this basis and I 
was part of the WG encouraging text that we now find in section 7.9 
because we had new situations where a "MUST NOT reject" rule was not 
appropriate for 100% detectable MTA problems using deterministic 
protocols.  We had greylisting, SPF, CBV and DKIM/ADSP coming and we 
needed to arm administrators with some new negative reply codes to 
explain and justify rejections.  Please note that none of the new SMTP 
add-on technology broke SMTP or changed it. It enhanced it. It was 
backward compatible.

However, mail.ietf.org rejection on a perfectly defined per spec item 
was not done because a valid FQDN was found to be available, hence 
reject on that basis. No, it simply flat out rejected ip literal usage 
all together despite correction.  The odd thing of forcing FQDN is not 
validating it.

In other word,  the mail.ietf.org implementation is promoting 
obsolescence and implementers who may use this server as a test site 
and example of the "standards" to resolve ambiguous design issues, it 
could begin to change things and I am not sure this is a desirable 
situation. Or is it?

I don't think we are at evolutionary SMTP stage yet were eliminating 
the IP-literal would be desirable.

 From a DNS standpoint, we advocate no PTR records, but from a Mail 
Admin AVS standpoint we do.  But even SPF as deprecated the use of PTR 
directives.

I ask for sound engineering consistency.  That is all I even ask in 
all my time participating the WG as an implementator. I don't say much 
until I have too because now it steps on my toes.

Do I have to change my SMTP code to avoid using ip literals? Do I even 
give operators the opportunity to set it up this way?

Thanks

-- 
HLS

---

From: Brandon Long <[email-masked]>
Date: Thu, 19 Dec 2019 12:52:02 -0800

> they know is secure and meets their HIPPA, GDPR, MMQF, etc. requirements?

The challenge with the secure messaging systems is only partially due to
transit concerns, they also want to be in the loop for the ACL to view the
message and control the lifecycle of the message as well.

Brandon

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 19 Dec 2019 17:55:06 -0500

On Thu, Dec 19, 2019 at 3:52 PM Brandon Long <[email-masked]> wrote:

>
>
> On Wed, Dec 18, 2019 at 11:12 AM Phillip Hallam-Baker <
> [email-masked]> wrote:
>
>> Messaging abuse isn't entirely absent on Facebook, Skype, Signal, etc.
>> but it is virtually non-existent compared to telephone and email. We are

Oh don't tempt me, I would love to be able to add Micali Fair Contracts to
my code...

---

From: "Valdis Klētnieks" <[email-masked]>
Date: Sun, 22 Dec 2019 01:45:56 -0500

> because we had new situations where a "MUST NOT reject" rule was not

As it turns out, it wasn't rejecting on that basis, it was rejecting on a "not
having a FQDN is a very strong spam signal" policy basis and erroneously
issuing the wrong message.

Given that there were *two* rules involved, one of which *did* flag an actual
protocol violation, I'm willing to accept it's just a cut-n-paste error that
nobody actually noticed for a decade or so...

And maybe it's time to let that "address literal" sail into the sunset. If
you're an MUA on a DHCP'ed device, there's a good chance that you don't know
your FQDN.  But at that point, the reality is that you're probably talking to
an MSA on port 587 rather than a full MTA on port 25, and the MSA is (or should
be) going to be authenticating you far more strictly than checking your EHLO.
And if you're a full MTA on the public internet today, and *don't* have a FQDN,
you're going to have a really bad day in 2019, *totally notwithstanding what
the RFCs say about how it's totally legal*.

What consenting MTA's do inside a walled garden is, of course, totally
up to the administrators involved.  They can use X.400 for all the rest
of the net cares. We passed the point of a significant divergence between
"what works in a walled garden" and "what works in the wild" at least
a decade ago. And at *some* point, that's a big pile of technical debt
that's going to need to be paid back. At this point, the least bad option
is probably producing an updated rfc5321/5322 that documents the
protocol, and a BCP/guidance document of "but here's what you need
to be doing to ensure you can survive in the wild".

---

From: Hector Santos <[email-masked]>
Date: Sun, 22 Dec 2019 17:22:21 -0500

On 12/22/2019 1:45 AM, Valdis Klētnieks wrote:>
> issuing the wrong message.

If the SDO receiver did a PTR lookup for the existence of a FQDN, I 
can see the beginnings of a justification related to the SMTP spec and 
a MTA issuing an ip-literal when a FQDN exist.

But the reality is the SDO receiver does not do a PTR check to see if 
a FQDN even exist.  I tested this case by using a machine without a 
PTR record.  It is doing a flat out rejection of "EHLO [ip-literal]" 
usage.

> nobody actually noticed for a decade or so...

And what would be the message for the human?

550 Please use a FQDN since one exist.
550 IP literals are no longer acceptable at the site.

or since its not doing a PTR check:

500 Please use a non-ip-literal. Junk host names are valid. We won't 
check.

> And maybe it's time to let that "address literal" sail into the sunset.

In my view, one server behavior, SDO or not, should not represent the 
entire world-wide community of valid mail bots that exist.  We really 
don't have any clue or stats how big this problem really is because so 
far, its only one server, afaik.

Are you ready to blindly accepts SDO reasons, follow their lead and 
reject all EHLO [ip-literal] clients?  Why not try it and give feedback?

The onus should be on SDO site to prove to the community why the 
protocol should be changed and why MTA code should be changed.

> the RFCs say about how it's totally legal*.

In the 16 yrs of using CBV, I haven't had seen a problem nor received 
a report from any customer except for two cases I found; this recent 
flat out rejection of ip-literals by the SDO site and another broken 
receiver who didn't like the HELO command and was enforcing EHLO.  It 
had no problem with using IP Literals as long as it with the EHLO command.

> to be doing to ensure you can survive in the wild".

I have a difference of a technical opinion for this particular item 
related to the adherence to a protocol feature of nearly 40 years. I 
don't see a justification for what is a uncommon behavior with 
standard SMTP servers.

If the SDO feels the SMTP community should eliminate it for the sake 
of the public SMTP service good, then propose it, show evidence, show 
legitimate justifications in the IETF-SMTP to convince implementators 
to change their code ($$$) and also update the SMTP specs.  Again, its 
not even doing an PTR check to see if an FQDN exist to use this as a 
reason.  So whatever proposal is offered has to make common IETF 
engineering sense.  This one does not.

If you believe we should tighten SMTP, enforce no ip-literals, always 
a FQDN, then why just raise the bar and go for the maximum payoff 
possible - the big bang, and officially begin to validate FQDN for 
100% correctness, no exception. After all, with the way it is is now 
done by the SDO, if others begin to follow their lead, disallowing 
ip-literal but allow invalid FQDNs, this would be encouraging more bad 
behavior and fact less spam detection.

--
HLS

---

From: John C Klensin <[email-masked]>
Date: Sun, 22 Dec 2019 17:30:17 -0500

Hi.

First, my apologies for taking so long getting this note out.
I've got a large number of end-of-year commitments and deadlines
(unrelated to the IETF and mostly unrelated  to the Internet)
that have to take priority over volunteer activities like this
one.  And each time I've come back to this since Tuesday morning
I've wanted to review all subsequent postings on the IETF list
and the parallel discussions on the ietf-smtp one to see if they
would change the response (some have).   For those who don't
like my long notes, please average this one out over nearly a
week and the multiple shorter notes I could, in principle, have
written during that tine. In any event and rather than trying to
respond to postings and points one at a time...

I think several recent postings are missing the point I (and a
few others) have been trying to make even if they are
interesting and/or useful, so let's back up a little bit.  I've
changed the subject line a bit in the hope that we can get back
to the original discussion and let the other subthreads go where
they will.

If some random, or not-to-random, party decides it doesn't want
mail traffic from anyone who uses an IP address literal in the
EHLO and/or MAIL commands, or doesn't want to accept mail that
uses, e.g., an RU or рф TLD or that has a PTR record or
originating connection address that appears to be from that
country [1], they have the perfect right to reject such traffic.
It makes little difference whether they think the spam:wanted
mail ratio is too high for mail that appears to be originating
there or because of some unrelated (and either rational or
irrational) dislike of the country, etc.   One can substitute
any domain, address type, address location, etc., one wants in
that example and the principle remains the same.  As an even
more extreme example, if someone is stuck in the 1990s or
suffering from an extreme case of ICANN aversion decides to
reject any mail transaction with a TLD name more than three
characters long or any DNS label that is an IDN, that is not an
IETF problem.   If we were writing a best practices document we
might recommend against several of those things, but that is a
different matter entirely -- the receiving MTA operator still
has the perfect right to decide what they will accept.

Russ Allbery's comments about different priorities among
different mail servers operators are, it seems to me, exactly on
point in this regard.  If we were writing a best practices
document and wanted to be honest about it and reflect
present-day realities, we'd probably need to say "if you are
trying to optimize for this, then you should do... and if you
are instead trying to optimize for that, then ...".  I'm not
sure "reliable delivery of anything that might be legitimate"
and "an absolute minimum of spam gets through" are the only two
categories, but they are  a start.  Unlike Keith (if I make the
right inference from his comments), I think we have some
responsibility toward people who are making both sets of
choices, including discussing tradeoffs, not just taking
responsibility for the "maximize reliable delivery" folks and
letting the others dig their own versions of whatever holes they
want to get into.

However, the thing that convinced me there was an issue worth
raising on this list is that these are IETF mail servers.  I
think that factors into the choices of how to optimize and what
to optimize for.  Either they should adhere to the standards
(which, in the email case, are definitely optimized for reliable
delivery because that seemed to be the only reasonable practice
when they were written) or we --the IETF community or at least
the subset who are concerned with mail standards-- should know
about it and consider whether the standards need upgrading or
supplementation. 

Deviations on "our" servers from what the standards require (or
even recommend) should be taken seriously for several reasons:

(1) We claim to be -- and need and want to be -- open to anyone
who is interested in participating.  Making would-be
participants jump through hoops, switch mail providers, shut
down mail systems that are personally and locally operated in
favor of virtual arrangements in easier-to-work-with
environments, or forcing switching of mail or Internet service
providers, is, at least IMO, inconsistent with that goal.
Noting the protections built into or procedures for taking
posting rights away from individuals, it seems unreasonable to
deny such rights and protections to those who systems conform to
our standards but are somehow a bit out of some definition of
the mainstream.  It also suggests that we should go to extra
efforts to be sure that any actions that are taken in the IETF's
name that could reject legitimate mail should be examined to be
sure there are not more finely-focused alternatives that would
have a lower false rejection rate.  For the particular case of
email spam (and has others have pointed out in comments that
also seem to have gotten lost), that argues for concentrating
more on message-specific properties such as known-problematic
origin addresses, the arguments to the MAIL command, and/or even
specific subject lines or content or pattern indications
(although others have pointed out the disadvantages of doing
in-depth message body inspection and I agree) than on blunt
instruments like rejecting all IP address literals (or
everything from a given domain).  If we are going to create
barriers to participation in the IETF for classes of people
based on their choices of mail clients, submission servers, or
mail providers --people who, as far as we know, would be
constructive and helpful participants who would bring us more
diversity and broader perspectives--  I think that deserves at
least as much consideration by the IETF (or at least the IESG)
as a decision to exclude a specific individual for a pattern of
bad behavior.    

None of that is micromanaging the Secretariat or anyone else:
they are principles and guidelines about who gets to participate
in the IETF, who we are willing to exclude, and why.   

(2)  There are many reasons why what the IETF mail servers do
should align with what the IETF is telling others to do.
Getting into "do what I say, not what I do" situations is almost
never productive.  Standards bodies who don't follow their own
standards develop sometimes-serious credibility problems and may
call the relevance of their standards. even in completely
different areas, into doubt. As things evolve, that may mean
changing the standards to align with the practices, it may mean
aligning  the practices more closely with the standards, or it
may mean just documenting the differences and the reasons for
them.  It does mean that either the Secretariat or IETF LLC need
to be thoroughly familiar with the relevant standards or that
they need to have an advisory process that can provide that
calibration because if we care about encouraging the use of our
standards, at least IMO, the one thing we need to avoid is
making uninformed or accidental decisions.  "Ask the IESG if you
have questions" is perfectly reasonable as long as they have the
expertise to know when to ask and, unless  the answers are
really obvious, the IESG consults community experts in an open
and transparent way.   Again, that isn't micromanaging the
Secretariat -- it is making general expectations clear and being
sure the Secretariat has the resources to meet them.

In that context, it is likely that the "IESG Statement on Spam
Control on IETF Mailing Lists" policies to which Alissa pointed
[2] is actually in need of review for at least two reasons:

(2.i)  The first is a nit (or set of them) but may point to what
got us into what I see as a problematic situation.  While it
could probably be corrected by tuning the error message, bullet
4, "IETF mailing lists MUST provide a mechanism for legitimate
technical participants to determine if an attempt to post was
dropped as apparent spam." is not satisfied by an error message
that claims a RFC 2821 (or 5321 if it were updated) restriction
since there are no restrictions in either document about spam
and they allow (and at least strongly encourage) allowing IP
literals.  If the message said "your attempt to post was
rejected because you used an IP address literal in EHLO and we
therefore think you are spammer", that would comply with this
MUST requirement.   It would be more in the spirit of the sixth
(last) bullet if the message went on to say "if you think this
rejection is in error, contact..." with appropriate information
there. 

(2.ii) As discussed in (1) above and in the light of many IETF
discussions and statements made in the last decade it seems to
me that an IESG anti-spam policy statement should be clear that
particular decisions to avoid spam should carefully avoid
discriminating against current or potential IETF participants
based on variations in connectivity, locations, and the like
unless there are no plausible alternatives.

(2.iii) In addition, as I think others have pointed out, the
statement points to a previous one for justification and
implementation advice, but the link given just points back to
the 14 April 2008 document.  And, in light of administrative and
other structural changes made since 2008, the Managing Director
of the IETF Secretariat, the IETF LLC Executive Director, and
probably the IAB and IRTF Chairs and the Independent Submissions
Editor, should probably be added to the "MUST be able to post"
list and the term "The Internet draft editor" clarified.  Under
normal circumstances, I would be among the first to argue that
the IESG has better things to do than to track this type of
statement regularly and tune it, but, since the statement has
come up in this discussion and been cited as the current policy,
those fixes should probably have some priority. 

(3) Several people have suggested that, if only one problem has
been reported in a decade, there is really no problem worth
worrying about.  This calls for a short statistical lecture
about sampling from censored data and/or a population different
from the one that inferences are being made about that I've sent
to a few people offlist, but I'll skip it here.  The short,
practical, version is that, if someone wants to participate in
the IETF but their attempts to post to our mailing lists are
frustrated by error conditions that are not properly reported,
with error messages that they may not even see, and with no
obvious (to them) way to get help with the problem, most or all
such people just go away.  They do not become part of our record
about who had problems and we have no plausible way to determine
whether there are no such people or hundreds.  So "there have
been few reported issues and we shouldn't spend time on this"
just does not hold water.  If we could know how many legitimate
messages or participants had been rejected by filtering
connection attempts with IP address literals in the EHLO field,
we could have a useful discussion about whether the number (and
types) of people who are being inconvenienced or deterred from
participation are ok to lose.  But we don't know... and, if
Hector had not been a very long-term participant who knew which
list to ask on and had alternate posting mechanisms readily
available, it is reasonable to assume we would not have heard
about his case either.

best regards and best wishes for whatever people are celebrating
and for a 2020 with fewer issues like this one.

    john

  ----------

[1] Examples chosen because I see a lot of spam from that
country, but other examples would work equally well.

[2]
https://www.ietf.org/about/groups/iesg/statements/spam-control-2008-04-14/

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 22 Dec 2019 19:33:27 -0500

> On Dec 22, 2019, at 5:30 PM, John C Klensin <[email-masked]> wrote:
> 
> But we don't know... and, if

One relevant point here is perhaps that (based on my understanding
of Hector's posts) even his MTA did not actually use address literals
when posting messages *to* IETF lists, and his messages to IETF lists
were getting through just fine.

The problem was that his callback verification system (variously called
CBV, sender address verifcation, ...) used address literals, and so
messages coming *from* IETF lists to him were rejected by his server
when callback probes were rejected by mail.ietf.org.

In my view, this makes the problem report more marginal.

I also posit that the IETF mail servers are not running some bespoke
version of SMTP that is inconsistent with IETF standards.  The
"550 5.7.1" reject is a clear policy reject, and the MTA adds further
context that makes it clear that the policy rejects the EHLO command
on the basis of the EHLO name.

The poorly chosen additional text added by the administrator has
since been amended, and the rejects now take the form:

  550 5.7.1 <[192.0.2.1]>: Helo command rejected: Policy violation

The policy in question is presumably effective, and can make it
possible to avoid more fragile and less transparent content-based
policies downstream.

Address literals remain a valid part of the SMTP protocol, useful
between minimal SMTP clients (special-purpose appliances of various
sorts, and some MUAs) and their configured SMTP relays (smarthosts).

They are much less useful between the SMTP relays and SMTP servers
operated by others on the public Internet.  The same applies to
sending from machines with no PTR records, from sender addresses in
non-existent domains, or non-FQDN HELO names that are not address
literals...

While Hector feels passionately[1] that he should not have to change
the CBV code he's been using for over a decade, others may well view
that code as a potential abuse vector in its own right, that masks
the origin of directory harvesting attacks.

I, for one, do not share his passion for keeping address literals
a viable HELO name form for MTA-to-MTA traffic across the public
Internet, nor do I feel that a policy that rejects these, even at
mail.ietf.org is a failure of the IETF to adhere to its own
standards.

On the contrary, I prefer filters that use static criteria over
fragile content filters, unavoidably error-prone (but sadly needed
RBLs), etc.  If the HELO filters work well enough (block
non-negligible quantities of junk, don't block non-negligible
quantities of legitimate email, allow the sending system to
generate a local bounce back to the message sender) they are
precisely the sort of filters that should be used when available,
and while still effective.

-- 
	Viktor.

[1] Along with a passion about the purported "inconsistency" of
    blocking IP literals, while allowing unvalidated FQDNs.  This
    would be a valid critique if the aim were to police protocol
    correctness, but since the aim is merely to cheaply block
    traffic from clients effectively don't resemble an MTA, there
    is no inconsistency, policing FQDN "validity" is not effective.

    Of course MTAs should have valid HELO FQDNs, these could even be
    useful some day in support of client-side DANE TLSA records,
    hardening the message relay path with hop-by-hop client
    authentication, when possible, but that's falls squarely outside
    the issue at hand.

---

From: "Eliot Lear (elear)" <[email-masked]>
Date: Mon, 23 Dec 2019 09:44:35 +0000

This is really only practical for MSAs. Special purpose clients should not be talking directly to other people’s MTAs in this day and age. 

Eliot

> On Dec 23, 2019, at 00:34, Viktor Dukhovni <[email-masked]> wrote:
>