Subject: Re: Last Call: draft-ietf-imapext-sort (INTERNET MESSAGE ACCESS PROTOCOL - SORT AND THREAD EXTENSIONS) to Proposed Standard

From: Dan Karp <[email-masked]>
Date: Sun, 2 Mar 2008 23:07:50 -0800 (PST)

Let's step back for a moment.

The purpose of sorting in mail clients is so that users can find
messages they're looking for.

After sorting on INTERNALDATE, the user skips down the "Received" column
to the range that their target messages fall in.  And, voila!  The
messages are there.

After sorting on SUBJECT, the user skips through the results to the set
of messages with the correct normalized subject.  This works because the
user mentally normalizes the Subject header in the same way that the
SORT extension does.  (The transform's easy to do, and years of mail
client use have trained them to do this; they're not looking under 'R'
for "re: Your Brains".)  And, voila!  They find the messages.

But sorting on FROM?  That's where things break down for the draft.

I took a look at 7 different mail clients: Gmail, mutt, Outlook Express,
PC-Pine, Thunderbird, Yahoo! Mail, and Zimbra.  In every single one of
these clients, there is a list of mail messages.  And in that list,
every client has a "From" column.  And every one of those "From" columns
contains the RFC2822 display-name of the From header of the relevant
message, or the RFC2822 addr-spec if there's no display-name.  Every
single one.

Gmail ("search, not sort") doesn't allow you to sort on "From".  But
every other client offers a "From" sort.  And every client *but* PC-Pine
does it exactly the same way: A straight text sort on the data displayed
in the "From" column.  And this makes sense, because a user knows the
name of the author of the message they're trying to find, and they can
skip to the right part of the list.  Things work because (a) the user
knows the sort key of the message, (b) the sort key is displayed in the
user interface, and (c) the list is ordered on that displayed sort key.
It's just as intuitive as the received-date search.

PC-Pine is different.  Looking at the first page of results when I point
it at the INBOX of my test account, I see the following "FROM sorted"
list (names but not sort order have been changed to protect the
innocent):

  N  21 Jun 21 Betsy Avertain                  (787) ...
  N  22 Sep 24 Hurricane Electric, Inc.       (3465) ...
  N  23 Sep 19 Waverly Bindle                  (11K) ...
  N  24 Dec  7 [email-masked]         (12K) ...
  N  25 Feb 23 Brian Pearson                  (5878) ...
  N  26 Jan 24 patrick walton                  (66K) ...
  N  27 Nov  9 build                          (2022) ...
  N  28 Nov 18 Calvin Poutine                 (6275) ...
     29 Feb  8 Leonardo Merman                (2535) ...

Those are the messages that PC-Pine, using the FROM sort rules as
defined in the draft, has decided to sort in the "B-C" range.  Note that
the sort key itself (the local-part of the RFC2822 addr-spec) isn't
displayed at all.  And it's not just PC-Pine: NO CLIENT that I know of
displays that information by default in the message listing; most if not
all can't even be configured to show it.

So if I'm looking for messages from Pat Walton, I'm going to first look
under "P" and not find anything, then wonder if I haven't actually filed
his messages into a different mailbox, then rack my brain until if I'm
lucky I remember that his username is actually "bpw", then skip to
somewhere in the "B"s and start trying to remember if Brian Pearson's
username was "brianp" (in which case I need to scroll up to find the
block of messages from Pat) or "bpearson" (in which case I need to
scroll down).

In order to find a message in this "sorted" list, you have to first
remember a piece of information that the client will not display.  If
you can't remember this piece of information -- and, in the world of
display-name-based autocomplete and correspondents you rarely send
mail to, that's a large and growing probability -- you're completely
hosed.  You then have to guess roughly the right position in the list to
jump to, and then in order to scroll/page you have to again remember a
non-displayed piece of information for each sender in the page you're
looking at.

This is not a joke case or an outlier.  This is reality.  I've
personally had usernames that started with "y.", "li", "da", "dk", "t-",
and "ka", most of them not by choice.

There is a reason that your cell phone's address book sorts by name,
not by phone number.

> I see no reason to remove/deprecate the FROM, TO, and CC sorts.
> 
> You have presented no reason other than your dislike of their
> definition.  The provision of FROM, TO, and CC sorts in no way blocks
> future extensions that offer other forms of sorting these (and other
> fields).

Once the draft makes it to Proposed Standard, getting a second draft out
with a variant version of these sorts becomes exponentially harder.
Getting server and client uptake for that revision is even less likely.
It is better for "the ecosystem" (I cannot believe I just typed that) to
solve the problem just once and to do it as correctly as possible.

There is a clear consensus in the mail user agent marketplace for how
address sorts should behave.  Intentionally specifying a different
algorithm, especially one whose results can be as counterintuitive as
the one in the draft, seems unwise.

> SORT is implemented by at least two clients and at least four servers.
> 
> All of these would have to change to support the proposed crippled
> SORT.  Worse, my client would require substantial new, special case
> code to restore the functionality lost due to crippled SORT; and after
> this work will have worse performance.

Legacy servers such as yours would advertise "SORT".  Legacy clients
looking for "SORT" would find it and use it.  Nothing would break.

New revisions of the legacy servers would advertise both the legacy
"SORT" and the standard "SORT=BASE".  Legacy clients would see "SORT"
and behave exactly as before.  New clients would see both "SORT" and
"SORT=BASE" and act accordingly.

I believe that your client already does client-side FROM sorting for
servers that don't advertise SORT.  Determining whether to do client- or
server-side FROM sorting based on whether the server advertised "SORT"
or just "SORT=BASE" is probably on the order of 10 lines of code.
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Mark Crispin <[email-masked]>
Date: Sun, 2 Mar 2008 23:53:04 -0800 (PST)

> with a variant version of these sorts becomes exponentially harder.

When there is both a de facto standard that has existed for over 10 years, 
and a completely incompatible de jure standard created because some guy 
didn't like the de facto standard, then getting any standard becomes 
impossible.

The SORT specification is infinitely extensible.  The original keys are 
there, and remain there forever.  However, nothing prevents the creation 
of FROM2, FROM.PLUS, ZIMBRAFROM, etc. as extensions to the keys.  That is, 
all new forms of SORT would be a superset.

The only thing that a client need concern itself about with a server is if 
the server implements (at least) the SORT level it desires.  If the client 
just cares about the original, 10+ year old, keys, then any server SORT 
will work, because all SORT is a superset of the original.

Should you get your way, there would be now two incompatible sets of keys. 
There would be the set that has been around for more than a decade, and 
the new incompatible set that *subsets* the original set instead of 
*supersets* it.

Clients would have to know which set is implemented by the server.  If the 
server implements the wrong set, then the client will have to do all the 
work in the client.  The end result is sufficient fear, uncertainty, and 
doubt that nobody will use server-based SORT at all.

> address sorts should behave.

Empirical testing of a few clients does not constitute a "concensus".

> "SORT=BASE" and act accordingly.

I already explained why this can not work.

SORT=BASE, by definition in the current SORT specification, is a 
*superset* of the 10+ year SORT.

    A server which supports the base-level SORT extension indicates this
    with a capability name which starts with "SORT".  Future,
    upwards-compatible extensions to the SORT extension will all start
    with "SORT", indicating support for this base level.

Consequently, current software does
    if (strcasencmp (cap,"SORT",4)) hasSort = 1;

Your demand has the effect of breaking this software.  The new extension 
would have to be called something like ZIMBRASORT, since the 10+ year old 
SORT specification reserves the entire SORT* extension namespace for 
upwards-compatible extensions.  ZIMBRASORT, by being a subset (and hence 
incompatible) must therefore have a different name.

The result is that, with IMAP sorting forked into two incompatible forms 
of sort, nobody will implement either.

> servers that don't advertise SORT.

Yes, and it is as slow as cold molasses in January when such a server is 
encountered.  Users generally don't even try to sort or thread on such 
deficient servers since it is entirely too painful.

I do not understand why you are so insistant upon demanding that 
something that has been in use for 10+ years be removed.  Forks in 
specifications are bad things.

Let me take that back.  Forks in specifications are not bad things; forks 
in specifications are VERY bad things.  The Achilles' heel of UNIX has 
been the multiplicity of forks; and it's not just the forks of SysV vs. 
BSD vs. Linux but incompatible forks within SysV, BSD, and Linux.

When IMAP forked in the past the result was incredible chaos and confusion 
that should never be repeated.  Ever since IMAP has gone to considerable 
effort to maintain upwards compatibility and not fork.  That is why, among 
other things, we have BODY and BODYSTRUCTURE.  That is why IMAP has at 
times chosen ugly syntax and mechanisms, because doing so provided the 
functionality without creating a fork.

If it was difficult to implement the keys in question, that would be one 
thing; but it is utterly trivial to do so.  The necessary framework is 
provided by virtue of the ENVELOPE (a base specification requirement since 
the very beginnings of IMAP) and SUBJECT sorting.  A high school kid can 
implement these keys with that framework in place.

Nothing prevents the addition of the "better" keys that you advocate, 
other than your vaguely expressed fear that the "better" keys won't happen 
if the "not better" keys are allowed to survive.  If the "better" keys 
can't survive that modest hurdle then maybe they really aren't "better" 
after all -- or maybe the entire argument is pointless and you should just 
let sleeping dogs lie.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 3 Mar 2008 11:40:45 +0100

Dan:

The draft can't be changed any more.

But there's good news. Like you, I have a database, and I've implemented 
SORT. It took less than a day. A part of those hours was wasted time, 
because as you point out, only Mark's client wants the kind of address 
sorting which the draft offers. But it was less than a day. Wasting 
part of a day isn't really something to get excited about. Worse things 
happen every month.

Arnt

---

From: Dave Cridland <[email-masked]>
Date: Mon, 03 Mar 2008 16:14:57 +0000

On Mon Mar  3 15:03:20 2008, Cyrus Daboo wrote:
> --On March 2, 2008 11:07:50 PM -0800 Dan Karp <[email-masked]>  
> wrote:
> 
> Actually you need to look at your use cases in more detail because  
> a lot of times searching is a much better solution than sorting.  
> e.g. the case of trying to find email from a particular person - a  
> search is much better. Yes you do have to do a little more work to  
> setup the search (type something in) rather than the single click  
> sorting requires, but on large mailboxes you will usually see what  
> you want immediately without having to scroll through the sorted  
> list looking for what you want.
> 
> 
I don't think Dan's saying that using SORT for searching is a good  
idea, but it's certainly what people do based on my observations,  
too. (In some clients, the sorted list is scrolled to whichever  
message was previously selected, so it's a fast way of finding other  
messages by the same person).

> For the most part I question why anyone would really want to sort  
> on any of the text fields. The only sort I have found useful is  
> sort by size - and then only to help in culling large messages when  
> my quota-limited mailboxes get full.

My client does no sorting at all, so I'm hard pressed to give any  
alternative use-cases. Generally, people seem to like sorting by date  
fields, though again not a text field. I've also seen people who like  
to sort by subject, essentially as a grouping thing - better done  
with threading, but still.

> View filtering can often be done with a single click (e.g.  
> option-click on the from address of a message currently in view  
> causes the view to show only those messages with that address).  
> This is far superior to sorting.

My client does the same (albeit with different clicks involved), and  
I too find it very useful. I'm not sure I'd say "far superior to  
sorting", since I don't see them as related, but I get your drift.

Still, though, the presence of useless SORT keys would only be a  
significant problem if they were especially difficult to implement,  
and the large deployments of SORT indicate that they're not. Even if  
it's only PINE, that's still a pretty vast number of users who may be  
relying on this behaviour.

When I last brought this up, I got the impression that it was  
considered by the vast majority that it was simply too late to change  
FROM to be useful.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade
_______________________________________________
IETF mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ietf

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 03 Mar 2008 16:23:01 +0000

Speaking as email client user:

>messages by the same person).
Yea, I do this frequently in Thunderbird.

>with threading, but still.
I frequently sort by subject, because many clients still don't support 
proper threading (threads can become broken due to lack of the 
References header).

I also sort by IMAP flags (i.e. I want to see all important messages), 
size and rarely by date.

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 03 Mar 2008 10:03:20 -0500

Hi Dan,

> messages they're looking for.

Actually you need to look at your use cases in more detail because a lot of 
times searching is a much better solution than sorting. e.g. the case of 
trying to find email from a particular person - a search is much better. 
Yes you do have to do a little more work to setup the search (type 
something in) rather than the single click sorting requires, but on large 
mailboxes you will usually see what you want immediately without having to 
scroll through the sorted list looking for what you want.

For the most part I question why anyone would really want to sort on any of 
the text fields. The only sort I have found useful is sort by size - and 
then only to help in culling large messages when my quota-limited mailboxes 
get full. The rest of the time I either use no sort, or threading, or view 
filtering (view only the results of a search). View filtering can often be 
done with a single click (e.g. option-click on the from address of a 
message currently in view causes the view to show only those messages with 
that address). This is far superior to sorting. At least that works well 
for me. Others may have different ways of processes their email where 
sorting may make more sense.

-- 
Cyrus Daboo

---

From: Mark Crispin <[email-masked]>
Date: Mon, 3 Mar 2008 11:06:29 -0800 (PST)

On Mon, 3 Mar 2008, Dave Cridland wrote:
> Still, though, the presence of useless SORT keys would only be a significant 
> problem if they were especially difficult to implement, and the large 
> deployments of SORT indicate that they're not. Even if it's only PINE, that's 
> still a pretty vast number of users who may be relying on this behaviour.

+1.

It amazes me that this is even questioned, much less that the entire 
[email-masked] list has been dragged into the fray.  This discussion 
belongs either in ietf-imapext, or if an appeal is to be lodged it should 
be made with the IESG.

-- Mark --

http://panda.com/mrc

---

From: Mark Crispin <[email-masked]>
Date: Mon, 3 Mar 2008 10:44:35 -0800 (PST)

On Mon, 3 Mar 2008, Alexey Melnikov wrote:
>> (In some 
>> clients, the sorted list is scrolled to whichever  message was previously 
>> selected, so it's a fast way of finding other  messages by the same 
>> person).

As do I.

Note that an address sort that collates all "mrc" messages together is 
more likely to do the right thing that one that collates by name and has 
to figure out that "Mark Crispin", "Mark R. Crispin", "Crispin, Mark", "M 
Crispin", "Mr. Mark Crispin", "IMAP Support", etc. are all the same.

> I frequently sort by subject, because many clients still don't support proper 
> threading (threads can become broken due to lack of the References header).

I also sort by Subject (although more commonly ORDERSUBJECT threading than 
direct SUBJECT sorting) in cases where the thread tree has become so 
complex that it has lost value.  A classic example is when a new thread is 
started by replying to a message in a thread that has little or no 
relationship to the thread.  Think USENET.

> I also sort by IMAP flags (i.e. I want to see all important messages), size 
> and rarely by date.

Usually for flags I would do a filter rather than a SORT, but it seems 
reasonable to me to add such a thing in the first round of extensions.

Date sorting seems to be pretty common with our users.  Some people who 
prefer their mail "in order" use it instead of sequence order in the 
mailbox (no sort) which tends to be an INTERNALDATE sort.

-- Mark --

http://panda.com/mrc