Subject: Re: duplicate messages on IETF mailing lists

From: Vernon Schryver <[email-masked]>
Date: Thu, 26 Jul 2001 07:54:51 -0600 (MDT)

> From: stanislav shalunov <[email-masked]>

> ...
> It appears that astro.cs.utk.edu has transmitted the message at least
> five times, with probably sendmail running with `-q30m' option, the
> message was transmitted successfully, but astro.cs.utk.edu didn't
> learn about the success at least the first four times.
>
> The only reasonable explanation for this behaviour would be that
> odin.ietf.org is in violation of RFC1047 (or that astro is broken,
> which is ruled out by the fact that there are numerous messages from
> various places sent to various ietf.org lists that arrived in
> duplicates and triplicates). ...

Please consider the rest of RFC 1047, and note that while other messages
have been duplicated, only that one was multipled a dozen times.  That
suggests that the problem with that message is more on astro.cs.utk.edu,
perhaps because astro.cs.utk.edu is running a short timeout and ietf.org
was so busy pumping viruses and warnings that ietf.org was repeatedly
exceding that short timeout.  Of course it could have been some other
problem at astro.cs.utk.edu, or perhaps something in the message
triggered a long delay at ietf.org.

Since at least about 8.11.1, sendmail has had a default hour or two
timeout for the DATA command.  The fact that the messages from
astro.cs.utk.edu were coming more frequently than once an hour is evidence
that it has shorter than default sendmail timeout, and might be related
those duplicates.  As I recall, sendmail 5 in the mid-1980's had what
comments in sendmail.cf called a non-standard long timeout.  I don't have
convenient access to a 8.9.3 sendmail such as appears to be running on
astro.cs.utk.edu, but I bet the default timeout for 8.9.3 was longer than
the delays  between some of its duplicate messages.

A secondary use of a bulk mail detecting checksumming system like that
described at http://www.rhyolite.com/dcc/ is watching such problems. 
This is the DCC header from what is so far the 11th and last duplicate
copy of the message (not counting my directly addressed copy and one
of the copies via ietf.org):

} X-DCC-RHYOLITE-Metrics: calcite.rhyolite.com 101; IP=249 env_From=ok From=23
}         Subject=many Message-ID=13 Received=13 Body=13 Fuz1=13

The count on the Received: header is for the textually last, usually
temporally first one.

This morning (7/27) ietf.org or optimus.ietf.org claims via the HELP
command to be running Sendmail 8.9.1a.  There are reasons that many
people consider sufficent to switch from ancient 8.9 to current 8.11.4
or even 8.12.0.

Vernon Schryver    [email-masked]

---

From: [email-masked]
Date: Thu, 26 Jul 2001 14:39:02 -0400

> people consider sufficent to switch from ancient 8.9 to current 8.11.4

Hey wait a minute - I'm not done finding bugs in the 8.12.0 betas (just caught
one yesterday, in fact ;)

Actually, it's fairly stable, and in public beta.  But remember guys,
8.12.0 *IS* still beta.

On the other hand, the general consensus at a recent get-together of
Sendmail people was that running anything pre 8.9.3 isn't a good
idea at all in today's Internet.  Upgrade to 8.11.4, or 8.12.0 will
be out very shortly.

-- 
				Valdis Kletnieks
				Operating Systems Analyst
				Virginia Tech

---

From: stanislav shalunov <[email-masked]>
Date: 26 Jul 2001 15:43:27 -0400

Vernon Schryver <[email-masked]> writes:

> have been duplicated, only that one was multipled a dozen times.

There were other messages that were delivered multiple times,
including those large virus files that mostly came two or three times.
Here are just a few:

   4 <[email-masked]>
   3 <[email-masked]>
   3 <[email-masked]>
   3 <[email-masked]>
   3 <[email-masked]>
   3 <[email-masked]>

> astro.cs.utk.edu were coming more frequently than once an hour is
> evidence that it has shorter than default sendmail timeout, and
> might be related those duplicates.

Timeout.datafinal is, indeed, usually 1 hour.  The astro machine must
be using a shorter timeout, but other machines have delivered messages
multiple times, too.  Incidentally, this happens to include your
machine, which delivered the message
<[email-masked]> (your internal id
f6JF51CK005137) to odin.ietf.org on Thu, 19 Jul 2001 11:04:07 -0400
(EDT) and then again, less than three minutes later, on Thu, 19 Jul
2001 11:06:43 -0400 (EDT) [timestamps generated by odin].

> people consider sufficent to switch from ancient 8.9 to current
> 8.11.4 or even 8.12.0.

Why would you look at optimus.ietf.org when ietf.org has a single MX
record: odin.ietf.org?

ietf.org        43200 IN        MX      10 odin.ietf.org

$ telnet odin.ietf.org 25
Trying 132.151.1.176...
Connected to odin.ietf.org.
Escape character is '^]'.
220 *********************************2******200************0*00 *****
HELP
500 Command unrecognized: "XXXX"
QUIT
221 ietf.org closing connection

This clearly is no sendmail.  And whatever it is, it clearly is
causing trouble with duplicates.

-- 
Stanislav Shalunov		http://www.internet2.edu/~shalunov/

"According to my math:   2**32 = 4,294,967,295  [...]"
	-- John S. Giltner, Jr. in comp.protocols.tcp-ip, 20010305

---

From: Keith Moore <[email-masked]>
Date: Thu, 26 Jul 2001 16:50:37 -0400

> exceding that short timeout. 

astro is using the default timeouts.  if those are the wrong timeouts, 
blame sendmail. (or NetBSD, if they changed them)

the problem could also be in the network - if connectivity between here
and there were sufficiently lousy.

it should be obvious that there are several potential sources of error,
that the condition was probably caused by a combination of factors,
and that we can't pinpoint the true nature of the problem unless
it recurs often enough that we can diagnose it.  (I could have left
the message in the queue for such purposes, but it didn't seem worth
the trouble).  Fortunately, unless the problem does recur (which given
history doesn't seem likely), it's already solved.  

isn't your finger tired of pointing by now?

Keith

---

From: Gordon Cook <[email-masked]>
Date: Thu, 26 Jul 2001 19:08:33 -0400

I guess with refusal to set their own sircam proc mail filter the 
owners of this list are tying to separate the men from the boys so to 
speak

In nine hours off line last night I received 6 new copies of 
sircam....five from this list.

Now with some help from a couple of friends I have for the first time 
my own procmailrc running at the unix level on netaxs

Its been there about 7 hours and so far NO new  sircams

this is my new .procmailrc just in case any of you find it useful

# sircam
      :0HB:
      * ^I send you this file in order to
      /dev/null

      :0HB:
      * ^Te mando este archivo para que me
      /dev/null

      /^.*\.lnk.*$/ reject

      /^.*\.bat.*$/ reject

      /^.*\.pif.*$/ reject

at least the virus warning messages from this list can be deleted 
with a keystroke and take little bandwidth

-- 
****************************************************************
The COOK Report on Internet, 431 Greenway Ave, Ewing, NJ 08618 USA
(609) 882-2572 (phone & fax) [email-masked]   http://cookreport.com

---

From: Robert Elz <[email-masked]>
Date: Fri, 27 Jul 2001 14:53:52 +0700

Date:        26 Jul 2001 15:43:27 -0400
    From:        stanislav shalunov <[email-masked]>
    Message-ID:  <[email-masked]>

  |    3 <[email-masked]>
  |    3 <[email-masked]>

That's me.    That is, that's my laptop.   It runs sendmail 8.11.0
with default timeouts - but being a laptop, there's no guarantee
at all that it will be hanging around for an hour for a message to
get delivered.   (And yes, it delivers directly to the MX target,
I don't go through local relays whenever I can avoid it).

But ...

Those messages went nowhere near ietf.org they were sent to the ipngwg
list ([email-masked]).

What's more the messages left brandenburg only once.  I have the logs..

Those duplicates were caused by a system receiving mail from the ipng list
and returning it to the list again.

If you looked at them, that was obvious from the headers.   Even more than
that, the 2nd of those actually contained this same answer to this same
question, part of it was a reply to someone who asked why the duplicates
were happening.

I suspect that all the messages you saw with 3 copies were from the same
problem (I have no idea if it is still happening, I filter duplicates,
and don't bother to count them) - the "3" is because that's how long it
took for the Received header loop protection to eventually drop the message.

Could we please stop bickering about mail agent behaviour, and let us all
go back to collecting all the stuff that this virus has been kind enough to
send our way (so far I have DES keys, project proposals, and quite a bit
of noise - wedding pictures, prayers, ...).   It is amazing what having
a few very well publicised email addresses will attract from those poor
unfortunates running trashware from microtrash.

kre