Discussion:
[Dovecot] Future enhancement of imap: sorting?
Andy Dills
2008-03-07 01:59:00 UTC
Permalink
After solving a problem with a sweet little application I discovered (more
on that below), I wanted to get some feedback about a potential
enhancement for IMAP: standardized, MUA independant, server-side sorting.

Currently, there are two basic solutions for sorting: handle it during
delivery or handle it in the MUA. However, if you handle it in the MUA,
you lose some of the client independancy and portability that imap was
designed to address, as there is no mechanism to sync sorting rules
between clients, and many lightweight clients don't support sorting at
all. On the flip side, in distributed mail envrionments sorting at
delivery time isn't feasible, as you typically have the entire mailspool
owned by a single UID and mounted via NFS. My basic point is that the two
paradigms for sorting have fundemental flaws that severaly limit them.

It occured to me that it would be ideal to enable sorting in the imap
daemon, enabling users to implement sorting rules which are stored in
their directory (similar to the other imap metadata), and which get
invoked whenever mail is checked. Perhaps that is beyond the reasonable
scope of the imap protocol, but in my mind sorting is a critical task that
is poorly addressed and needs to be standardized. I'd love to hear
opinions on this.


See, in our mail cluster environment, it's basically impossible to use
procmail without forwarding your mail to our shell server (nor are we able
to use the dovecot LDA). So, the question presented to me by one of my
business partners was, since Outlook was shortsighted and didn't include
the ability to sync sorting rules between clients, nor did they implement
sorting functionality whatsoever on the windows mobile version, what
options does he have to ensure proper sorting of his mail without having
to leave outlook running 24x7? He doesn't want to have to wade through
mailing list email on his mobile outlook, nor does he want to have to
ensure his filter rules are consistent in both his outlook and our
webmail.

So, I poked around in the ports tree, and found a gem: imapfilter.

http://imapfilter.hellug.gr/

It's configured via lua, supports a very rich array of actions and
queries, and has proven to be very robust and lightweight. I helped him
get started with some sample rules, added it to run every minute as a cron
job, and boom: robust, fully featured, MUA independant filtering in an
environment that can't support procmail or the dovecot LDA.

As nice as it is, it's still a personal solution. In my mind we need to
have standardized, server-side, MUA-independant sorting rules that get
invoked when a mailbox is checked. Thoughts?

Andy

---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
Timo Sirainen
2008-03-07 02:07:16 UTC
Permalink
Post by Andy Dills
As nice as it is, it's still a personal solution. In my mind we need to
have standardized, server-side, MUA-independant sorting rules that get
invoked when a mailbox is checked. Thoughts?
Filtering is a better name for this, took me a few paragraphs before I
understood what you meant by sorting. :)

You could already create a plugin that runs a command when it sees new
messages in a mailbox. I'm not really interested in writing this
feature..

A somewhat related feature that I'm going to implement is virtual
mailboxes where messages are shown according to specified search rules.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20080307/cde68329/attachment-0001.bin
Andy Dills
2008-03-07 02:23:11 UTC
Permalink
Post by Timo Sirainen
Post by Andy Dills
As nice as it is, it's still a personal solution. In my mind we need to
have standardized, server-side, MUA-independant sorting rules that get
invoked when a mailbox is checked. Thoughts?
Filtering is a better name for this, took me a few paragraphs before I
understood what you meant by sorting. :)
Heh, sorry about that. In my mind, sorting implies directing an object to
a location based on some arbitrary criteria, whereas filtering (in my
mind) implies rejecting or accepting based on some arbitrary criteria.
Post by Timo Sirainen
You could already create a plugin that runs a command when it sees new
messages in a mailbox. I'm not really interested in writing this
feature..
I didn't mean to imply this as a request for functionality...it would be
worthless as a feature in dovecot without support in the IMAP protocol and
thus the MUAs. I was thinking more along the lines of a future enchanement
for the actual IMAP protocol, as this seems like something that should
have been addressed but wasn't, leading to an array of flawed solutions. I
assume you feel that this is beyond the scope of what IMAP should handle
natively? On one hand it feels like it is, but on the other hand it seems
like something in desperate need of standardization.

The idea for virtual folders based on search criteria is quite
interesting, by the way. How would you enable users to create them from
their client?

Thanks,
Andy

---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
Timo Sirainen
2008-03-07 02:29:31 UTC
Permalink
Post by Andy Dills
Post by Timo Sirainen
You could already create a plugin that runs a command when it sees new
messages in a mailbox. I'm not really interested in writing this
feature..
I didn't mean to imply this as a request for functionality...it would be
worthless as a feature in dovecot without support in the IMAP protocol and
thus the MUAs. I was thinking more along the lines of a future enchanement
for the actual IMAP protocol, as this seems like something that should
have been addressed but wasn't, leading to an array of flawed solutions. I
assume you feel that this is beyond the scope of what IMAP should handle
natively? On one hand it feels like it is, but on the other hand it seems
like something in desperate need of standardization.
I think Sieve (+ ManageSieve) could already do all of this. It's just
normally meant to be used with MDA, but nothing would prevent from
running the script when IMAP server sees it for the first time.
Post by Andy Dills
The idea for virtual folders based on search criteria is quite
interesting, by the way. How would you enable users to create them from
their client?
I'm just going to make it read the configuration from a file. How it
gets there is not my problem, at least not yet. :) So it'll probably get
some kind of a web configuration tool.

There do exist a couple of drafts related to this, but I think they're
all abandoned now.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20080307/32c25921/attachment.bin
Andy Dills
2008-03-07 03:17:16 UTC
Permalink
Post by Timo Sirainen
I think Sieve (+ ManageSieve) could already do all of this. It's just
normally meant to be used with MDA, but nothing would prevent from
running the script when IMAP server sees it for the first time.
I hadn't considered invoking Sieve from imap, I'll look into that as a
more scalable solution.
Post by Timo Sirainen
Post by Andy Dills
The idea for virtual folders based on search criteria is quite
interesting, by the way. How would you enable users to create them from
their client?
I'm just going to make it read the configuration from a file. How it
gets there is not my problem, at least not yet. :) So it'll probably get
some kind of a web configuration tool.
Makes sense. This feature has a large amount of potential and I would love
to see it implemented. The possibilities are exciting, such as a adding a
button to atmail's addressbook to create a virtual folder for that
contact, which would contain mail sent from them, with a sub folder
containing mail sent to them. Even something as simple as being able to
associate a message with multiple folders without requiring multiple
copies on disk is a nice improvement.

Do you have any timelines in mind for this feature? I'd be happy to help
work out the bugs. No pressure, I'm just excited to offer new and useful
functionality to our users.

Andy

---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
Timo Sirainen
2008-03-07 09:13:38 UTC
Permalink
Post by Andy Dills
Post by Timo Sirainen
Post by Andy Dills
The idea for virtual folders based on search criteria is quite
interesting, by the way. How would you enable users to create them from
their client?
I'm just going to make it read the configuration from a file. How it
gets there is not my problem, at least not yet. :) So it'll
probably get
some kind of a web configuration tool.
Makes sense. This feature has a large amount of potential and I would love
to see it implemented. The possibilities are exciting, such as a adding a
button to atmail's addressbook to create a virtual folder for that
contact, which would contain mail sent from them, with a sub folder
containing mail sent to them. Even something as simple as being able to
associate a message with multiple folders without requiring multiple
copies on disk is a nice improvement.
Do you have any timelines in mind for this feature? I'd be happy to help
work out the bugs. No pressure, I'm just excited to offer new and useful
functionality to our users.
I'm not sure. It's the next thing on my TODO list, so maybe in a
couple of months. I'm soon going to be quite busy for 5 weeks,
probably won't get much Dovecot work done.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20080307/dd8ad6de/attachment.bin
Loading...