Subject: filtering of mailing lists and NATs

From: Keith Moore <[email-masked]>
Date: Mon, 21 May 2001 18:00:02 -0400

it occurs to me that most of the methods that have been proposed
for filtering spam from mailing lists have a lot in common with NATs.

in both cases, the proponents say (in effect) "if it works for me and 
for my small set of test cases, it must be okay to impose this on 
everyone.  if some legitimate traffic is excluded by my filters, they is 
of no consequence - they should be willing to jump through whatever hoops
that I believe are appropriate.  and if people have to abandon practices 
that they find useful in order to to get around my filters, that is of 
no consequence either, because they do not need to be doing those things 
anyway"

Keith

---

From: grenville armitage <[email-masked]>
Date: Mon, 21 May 2001 16:37:25 -0700

> for filtering spam from mailing lists have a lot in common with NATs.

actually, have more in common with firewalls.

firewalls serve a filtering purpose, and (gasp!) people have
learned to configure proxies into their www, etc, clients.
we've gotten over it. and so can it be with mailing lists.

cheers,
gja

---

From: Keith Moore <[email-masked]>
Date: Mon, 21 May 2001 22:53:40 -0400

> actually, have more in common with firewalls.

I beg to differ.  People install firewalls to filter their own incoming
and/or outgoing traffic.  Personally I think firewalls are overrated,
but people can filter traffic on their own networks if they want to,
using whatever criteria they think best. 
 
Those who are complaining about spam on this list have the ability to 
filter their own incoming traffic.  What they're wanting is for someone 
else to filter their traffic for them, and for everyone else on the list 
also, using poorly-chosen filtering criteria on which there's no consensus. 

Keith

---

From: grenville armitage <[email-masked]>
Date: Mon, 21 May 2001 20:21:10 -0700

Aidan Williams wrote:
	[..]
> To extend the analogy again in the opposite direction: now that
> software is available to tunnel random traffic over HTTP, we can
> expect firewall filtering to get harder, and become less effective.
> Why would this not happen for email lists too?

Most spammers strike me as opportunistic and not overly interested
in special-case-handling a couple of subscribe-to-send lists,
given the hundreds and thousands of target addresses they
purchased on a CDROM. Yes, they could get around pre-subscribe
schemes. Yet it seems likely most wouldn't bother, and would
instead just end up ignoring us. Which would be nice.

cheers,
gja

---

From: grenville armitage <[email-masked]>
Date: Mon, 21 May 2001 20:33:39 -0700

Keith Moore wrote:

> and/or outgoing traffic.

D'oh. I thought firewalls where also used to filter traffic
one did *not* ask for. Stuff that wasn't apriori declared part of
one's community of interest. Seemed a reasonable analogy.

	[..]
> filter their own incoming traffic.

Kinda, sorta. No reason not to propose schemes that operate closer
to the source, even by one hop.

>  What they're wanting is for someone
> else to filter their traffic for them,

Yup.

> and for everyone else on the list
> also,

Right again.

> using poorly-chosen filtering criteria on which there's no consensus.

It is a fragile universe one inhabits where asking people to subscribe
to the community of interest before posting is equated to censorship.
If that's what you mean by "poorly-chosen" then... oh well. Thread
fizzles to an end.

cheers,
gja

---

From: Eric Rosen <[email-masked]>
Date: Tue, 22 May 2001 08:43:01 -0400

So, here are the choices:

1. Save thousands of people from having to deal with multiple spams per day,
   at the cost of presenting a minor inconvenience to a few, or

2. Require thousands  of people to receive  and deal with spam  (or to learn
   all about mail filtering), in order to avoid inconveniencing a few.

Easy decision  to make.   For every  bit of whining  by the  usual suspects,
there are thousands of  folks that are very happy to have  the spam kept out
of their mailbox automatically.  (Every  mailing list manager knows that the
whining by  Keith and Lloyd is nothing  compared to the whining  by the list
members as they get spammed multiple times per day.)

Indeed, this is a lot like the arguments re NAT.  There are the thousands of
people it helps, vs. the few who are yelling that the sky will fall if it is
not stamped out.

---

From: Robert Elz <[email-masked]>
Date: Tue, 22 May 2001 19:52:02 +0700

Date:        Mon, 21 May 2001 20:21:10 -0700
    From:        grenville armitage <[email-masked]>
    Message-ID:  <[email-masked]>

  | Most spammers strike me as opportunistic and not overly interested
  | in special-case-handling a couple of subscribe-to-send lists,

Of course, and as long as they can get to the vast majority of their
target, it will probably remain that way.

But as soon as the spammers need to go to some extra effort to reach
their audience, you can be sure they will.  Remember, once, they sent
from any random invented host name - then everyone started having their
mailers reject mail from unknown domains - now all the spam comes from
perfectly valid domains, which not only makes the checks for invalid
domains a waste of time (the check spends time achieving nothing at all)
but also results in all the failed spam (the bounces - and the abuse from
people who received it) being dumped on whichever unfortunate site they
picked to use as the domain name.

A supposed technological fix to a non-technological problem that just
made things worse, not better.

Now we're having suggested that only "known" e-mail addresses be allowed
to send to certain destinations.   Assuming that becomes really popular
(rather than just used on a small set of irrelevant lists) how long do you
think it will be before the spammer's lists of names contain not only the
destination address, but the From: address they should use to send to that
address?

I mean, how hard do you think it is to stick From: [email-masked]
in the heading of the mail?

One more technological fix that won't work.   And again, it will make
things worse, since then we won't be able to tell easily what is
traffic from people we expect to send to the lists, and what is not.

This is not a technological problem - it is a social problem.  We cannot
fix spam by technological means - it has to be fixed by social means.

And not only are technological "fixes" making things actually worse as
illustrated above, they're also suggesting to people that perhaps there
may be a technological fix that will actually finally solve the problem,
which lessens the demand for real social fixes instead.

Give up, let the spam through, deluge everyone with it - then the
opposition to it will rise quickly enough, and become urgent enough,
that the correct kind of remedies can be put in place.

kre

---

From: Francis Dupont <[email-masked]>
Date: Tue, 22 May 2001 15:51:36 +0200

In your previous mail you wrote:

=> thanks for this nice summary about the spam problem!

[email-masked]

---

From: Leo Vegoda <[email-masked]>
Date: Tue, 22 May 2001 15:59:52 +0200

>    all about mail filtering), in order to avoid inconveniencing a few.

Another similarity to NATs is that you don't know how many people are
behind a single (subscribed) address. For instance, I read your message
via a local news server. Of course, this means that any attempt to work
out the utility value of a filtering system must fail.

I'm perfectly happy to filter messages to this list locally. To be
frank, it takes a very small amount of my time. Surely people who want
to subscribe to this list are capable of setting up local filters?

Regards,

-leo

---

From: [email-masked]
Date: Tue, 22 May 2001 06:52:25 -0700 (PDT)

> their audience, you can be sure they will.

> ...

> destination address, but the From: address they should use to send to that

A long time, actually. While it is true that spammers will work around anything
that seriously impedes the flow of spam, you have not shown that spam sent to
lists is at all important to spammers. Every indication I see is that lists are
primarily useful to spammers as a source of addresses to send spam to directly,
and less as a target for spamming lots of people indirectly. Indeed, most
spammers that send to lists seem totally uninterested in the fact that they are
sending to a list; it is simply another address they have culled from some sort
of scan.

And while there have been some isolated reports of subscribe-then-send and
send-using-a-subscriber strategies used by spammers, the frequency of their use
appears to be way out of porportion to the number of lists that have
successfully fended off spam by using various subscriber-only techniques.

> in the heading of the mail?

Actually, maintaining an additional per-list address and keeping that address
up to date is pretty difficult. It is much easier -- and quite effective -- to
simply prowl for addresses that reach users directly.

> fix spam by technological means - it has to be fixed by social means.

In general, I agree with this assessment. But that doesn't mean that some point
fixes don't help in some cases.

				Ned

---

From: Keith Moore <[email-masked]>
Date: Tue, 22 May 2001 11:30:42 -0400

>    all about mail filtering), in order to avoid inconveniencing a few.

you have it backwards.  all subscribers of the list are 'inconvenienced'
if we discourage legitimate contributions from folks who are not willing
to jump through arbitrary and time-consuming hoops that we impose on them
just because a few people insisted (even in the face of evidence to the
contrary) that they knew what was best for everyone else.

calling those hoops a 'minor inconvenience' is also misleading.

> people it helps, vs. the few who are yelling that the sky will fall if it is

the people who are helped by NAT are also hurt by NAT.  but they might not realize
that NAT is the reason that they cannot deploy IP telephony.  they'll blame the
new application rather than the NAT because they've been brainwashed into thinking
that NAT is the right thing to do, and also because the guy who bought the NATs
in the first place is not going to admit that he was wrong.

Keith

---

From: Lloyd Wood <[email-masked]>
Date: Tue, 22 May 2001 12:28:21 +0100 (BST)

> to the community of interest before posting is equated to censorship.

Be liberal in what you accept.

Frankly, I don't think the community of interest happens to directly
correspond to the subscription list of a relevant mailing
list. Otherwise we wouldn't need web or ftp archives.

L.

<[email-masked]>PGP<http://www.ee.surrey.ac.uk/Personal/L.Wood/>

---

From: [email-masked]
Date: Tue, 22 May 2001 09:34:08 -0700 (PDT)

> contrary) that they knew what was best for everyone else.

This assumes that list filtering cannot be done sensibly. This assumption is
false; it can be done sensibly and is done sensibly all the time. And when it
is done sensibly the amount of inconvenience is unnoticeable. Sure, there are
plenty of lists that don't do filtering sensibly (including, alas, some IETF WG
lists), but there are many others that do.

Whether or not list filtering can be sensibly applied to a list with the
characteristics of the main IETF list is just a matter of resources. The
necessary technologies exist to cope with all the trickiness the IETF list
presents and more. All we have to do is agree to apply them and find the
resources to make it happen.

> calling those hoops a 'minor inconvenience' is also misleading.

Only if the lists aren't managed correctly.

Keith, I have to say that you are becoming your own worst enemy in this
discussion. By insisting on an absolute policy of no filtering at all your
ability to influence the policy that eventually is adopted is being
compromised. As a result we are increasingly likely to end up with a list
policy imposed that doesn't accomodate some aspect of real world behavior that
could have been dealt with.

I also find the comparisons with NAT to be strained at best.

				Ned

---

From: grenville armitage <[email-masked]>
Date: Tue, 22 May 2001 10:16:06 -0700

> Be liberal in what you accept.

I am not a protocol. I am a human being.

cheers,
gja
____________________________________________________________________
Grenville Armitage                http://members.home.net/garmitage/

---

From: Keith Moore <[email-masked]>
Date: Tue, 22 May 2001 13:21:19 -0400

> lists), but there are many others that do.

I also think that list filtering can be done sensibly, and I agree that this
is mostly (though not entirely) a matter of resources.  what I am objecting to
is the notion that 'sensible filtering' (particularly on the IETF list)
equates to 'filtering postings from non-subscribers'.

> Only if the lists aren't managed correctly.

which is, in my experience, all too often the case.   and the knowledge 
required to 'correctly' manage a list seems to be in short supply.
it would be useful to collect such knowledge into an RFC.

> compromised. 

but I have never insisted on such a policy.  I have only insisted that 
it's not appropriate to expect people to subscribe to the list in order
to contribute to the discussion.  in fact I use various kinds of filtering
on the lists that I maintain (different degrees depending on the nature
of the list), so I agree that filtering can be useful and appropriate.

> As a result we are increasingly likely to end up with a list

as you might imagine I am also frustrated by the tendency of this kind of
debate to polarize people around extreme positions, rather than to encourage
brainstorming about solutions that would address the entire spectrum of
interests and concerns that are expressed.

at the same time, I feel that it's important to argue against proposals
for quick fixes that seem shortsighted.  we have too many of those already.
we need to understand the problem from a variety of perspectives before 
insisting that our proposed solutions are appropriate to impose on everybody.

> I also find the comparisons with NAT to be strained at best.

I'm sure we can all come up with  examples of 'solutions' that served one
interest while harming others, or that served short term goals while doing 
harm in the long term.  NATs aren't an especially unusual example of this,
they're just an example that can be understood by most of the list.

Keith

---

From: grenville armitage <[email-masked]>
Date: Tue, 22 May 2001 11:16:58 -0700

> target, it will probably remain that way.

Good enough for me. It is said that to avoid a bear, you don't
need to be faster than the bear, just faster than someone else
in your group ;)

	[..]
> how long do you

Who knows. I suspect it would be a *vastly* long time before the
ratio of 'blocked mailing list' to 'personal email addresses' becomes
so high that spammers will special-case their code just to target
mailing lists. Today mailing lists are accidental inclusions on spammer
master target lists. They already deal with email addresses that get
stale and bounce, the trick is to convince them our mailing list address
is similarly 'stale'. This *is* social engineering, by us, of them,
using technology.

cheers,
gja

---

From: grenville armitage <[email-masked]>
Date: Tue, 22 May 2001 11:25:06 -0700

Robert Elz wrote:
	[..]
> fix spam by technological means - it has to be fixed by social means.

If you remove technological means you're left with "Bad spammer,
please don't send email to our list". The non-technical escalation
path (aka 'social means') then leads to lawyers and politicians
thinking they know what's best. Then you'd really see what it feels
like to have the wrong weapon brought to bear on a problem.

I shudder at the thought.

cheers,
gja

---

From: James M Galvin <[email-masked]>
Date: Wed, 23 May 2001 10:49:06 -0400 (EDT)

Keith,

Your NAT analogy is weak, very weak, at best.  It's opening premise is
flawed, as is this entire discussion of mail list filtering, because it
confuses policy with implementation.

The IETF has a policy of "openness" for all its mailing lists.  The
problem is most of the argument against filtering defines openness as
"all messages shall be distributed."  This is false.

Every IETF mailing list has a charter, a known purpose for its use.  It
is entirely reasonable and legitimate to reject all submissions that are
outside the scope of the charter.  If we can not agree on that point
this whole discussion is pointless.

Implementation is wholly separate from policy, and a primary concern for
the list maintainer.  A list maintainer needs to figure out how to
identify messages that are within scope and ideally would like to
automate that process.  I would assert they can do this without anyone's
approval or guidance.  The only issue anyone in the IETF can have with
that is if the list maintainer has a skewed sense of "within scope" or
if whatever process they use generates false positives.  But you can not
know this until after the fact.  We do so many things in this
"organization" on the basis of "subjective judgement with after decision
peer review," (less so now than even just 5 years ago but still) why
should this be any different?

Mail filtering is not in and of itself a bad thing.  It is a tool, a
legitimate tool, that when used as part of a larger solution to the
problem of maintaining the integrity of a mailing list is extremely
valuable.

Restricting the posting of messages to subscribers is not bad, it is an
excellent choice for the first line defense against off-topic messages.
The issue is whether it is the only solution employed.

Messages from non-subscribers need to be reviewed to determine if they
are within scope.  In a worst case this review is done manually but it
doesn't need to be.  There are a few (I mean less than 5) additional
technological criteria that can be applied that will correctly review
95% or more of the non-subscriber messages.  This minimizes the manual
work.

I know this because I do this and have been doing it for years.  I have
a 100% success rate at keeping spam off mailing lists and no complaints.
The total volume of email I deal with far exceeds the needs of all the
IETF lists combined.  This is not rocket science.

Furthermore, I don't see how the occasional 24-48 hour delay in getting
an occasional message distributed is bad.  So many people have this
idealistic view of email immediacy.  Have you ever really looked at the
Received: lines for messages distributed to the main IETF list?
Messages to me typically take about 6 hours to get delivered but I've
seen delays as long as 18 hours.  And the delay is *not* at my end.

Jim

On Mon, 21 May 2001, Keith Moore wrote:

    Date: Mon, 21 May 2001 18:00:02 -0400
    From: Keith Moore <[email-masked]>
    To: [email-masked]
    Cc: [email-masked]
    Subject: filtering of mailing lists and NATs

    
    Keith

---

From: [email-masked]
Date: Mon, 28 May 2001 09:50:38 +0530

I am sorry, if I sound harsh, but I think this chain of mails is becoming more
concerning than the amount of spam one receives. Could we put an end to it?

--
************************************************
The only way to solve a problem is to look at it in the face.
************************************************