[Dovecot] dovecot as gmail imap proxy

[Farkas Levente]

hi,
may be this is a bit of topic, but i hope not. it's always a big
question which is the better to choose a hosting provider to keep and
manage you mail or setup your own mail server, hosting, virus and spam
filter etc.. now as more and more people and company move to gmail or
google apps mail service it seems google can't handle the load and it's
getting more and more slower. what's more not just the imap interface
but nowadays the web interface used to be hang or even stop working.
it's getting more and more annoying that the speed is worst than in a 10
years ago. i thing about how can this be solved without totally give up
gmail as a mail service provider (and may be they can solve it in a few
months/years:-).
would it be possible to create a dovecot server as an imap proxy for
gmail and google apps?
imho many people/company would be love to use such a setup where he can
use gmail in vacation or any other time, but when the email is real
critical apps (not acceptable to wait 1 minutes for an email to load or
to open a folder/label) than he can use a proxy server eg. dovecot which
act as a imap proxy to gmail and can be hosted on closer, faster server
and this speed problem never happend. what's more it can be used as a
mail backup server too.
anybody has any such setup, idea? would it be possible with dovecot.
any other suggestion?
yours.

[Alexander Prinsier]

To have a speed increase, dovecot needs direct access to the emails. So
you'll need them stored locally. You can't use it as a proxy, just to
speed up things, because dovecot would just need to wait for gmail too,
to load an email (except if dovecot would cache the emails, but then
again, it's just a cache, and you're talking about 'critical applications').

I'm not sure, but I don't think there's even a way to run dovecot as a
proxy.

Why don't you forward mail from your gmail account to your own server?
That's so much easier and more common. You get the benefit of good
filtering, and you have your own server to access your mail. If your own
server goes down, you still have gmail as backup.

Alexander

Post by Farkas Levente
hi,
may be this is a bit of topic, but i hope not. it's always a big
question which is the better to choose a hosting provider to keep and
manage you mail or setup your own mail server, hosting, virus and spam
filter etc.. now as more and more people and company move to gmail or
google apps mail service it seems google can't handle the load and it's
getting more and more slower. what's more not just the imap interface
but nowadays the web interface used to be hang or even stop working.
it's getting more and more annoying that the speed is worst than in a 10
years ago. i thing about how can this be solved without totally give up
gmail as a mail service provider (and may be they can solve it in a few
months/years:-).
would it be possible to create a dovecot server as an imap proxy for
gmail and google apps?
imho many people/company would be love to use such a setup where he can
use gmail in vacation or any other time, but when the email is real
critical apps (not acceptable to wait 1 minutes for an email to load or
to open a folder/label) than he can use a proxy server eg. dovecot which
act as a imap proxy to gmail and can be hosted on closer, faster server
and this speed problem never happend. what's more it can be used as a
mail backup server too.
anybody has any such setup, idea? would it be possible with dovecot.
any other suggestion?
yours.

[Peter Hessler]

I do this exact thing, and it is faster than gmail itself. With IDLE, I
get the msg alert on my dovecot server before the web interface shows a
new message.

Also, far more reliable, with a superior UI.


On 2008 Jun 17 (Tue) at 17:13:32 +0200 (+0200), Alexander Prinsier wrote:
:Why don't you forward mail from your gmail account to your own server?
:That's so much easier and more common. You get the benefit of good
:filtering, and you have your own server to access your mail. If your own
:server goes down, you still have gmail as backup.

[Farkas Levente]

Post by Peter Hessler
I do this exact thing, and it is faster than gmail itself. With IDLE, I
get the msg alert on my dovecot server before the web interface shows a
new message.

which exact thing?
and what does the "with idle" means?

Post by Peter Hessler
Also, far more reliable, with a superior UI.
:Why don't you forward mail from your gmail account to your own server?
:That's so much easier and more common. You get the benefit of good
:filtering, and you have your own server to access your mail. If your own
:server goes down, you still have gmail as backup.

[Asheesh Laroia]

Post by Peter Hessler
I do this exact thing, and it is faster than gmail itself. With IDLE, I
get the msg alert on my dovecot server before the web interface shows a
new message.
Also, far more reliable, with a superior UI.

Which is the superior UI? Honest question - I'm shopping around for nice
IMAP interfaces, if that's what you mean. I'd love to know what you're
using.

-- Asheesh.

[Peter Hessler]

Post by Asheesh Laroia

Post by Peter Hessler
I do this exact thing, and it is faster than gmail itself. With IDLE,
I get the msg alert on my dovecot server before the web interface shows
a new message.
Also, far more reliable, with a superior UI.

Which is the superior UI? Honest question - I'm shopping around for nice
IMAP interfaces, if that's what you mean. I'd love to know what you're
using.
-- Asheesh.
--
Marriage is the triumph of imagination over intelligence. Second marriage is
the triumph of hope over experience.

I <3 mutt. Easy enough, but lots of power is available.

I haven't found any webmail interfaces that I like, but then again, I have yet
to find *any* web interface that I like.

[Geert Hendrickx]

Post by Peter Hessler
I <3 mutt. Easy enough, but lots of power is available.

Yes, mutt is great! :-)

Btw, you can use mutt with dovecot without running dovecot as a daemon/server,
by putting this in your .muttrc:

set tunnel="MAIL=/home/geert/mbox /usr/pkg/libexec/dovecot/imap"

(adapt paths for your own system)

This way mutt will talk to the dovecot imap binary via stdin/stdout. This
will also work over ssh. This way you don't need to run dovecot as a server
and you still benefit from eg. the cache etc.

Geert

[Timo Sirainen]

Post by Geert Hendrickx

Post by Peter Hessler
I <3 mutt. Easy enough, but lots of power is available.

Yes, mutt is great! :-)
Btw, you can use mutt with dovecot without running dovecot as a daemon/server,
set tunnel="MAIL=/home/geert/mbox /usr/pkg/libexec/dovecot/imap"

"dovecot --exec-mail imap" is better. Then it uses the settings from
dovecot.conf as well as the proper default settings. With the above
you're for example using mbox_lazy_writes=no

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20080618/562e5b1b/attachment.bin

[Timo Sirainen]

Post by Farkas Levente
would it be possible to create a dovecot server as an imap proxy for
gmail and google apps?
imho many people/company would be love to use such a setup where he
can use gmail in vacation or any other time, but when the email is
real critical apps (not acceptable to wait 1 minutes for an email to
load or to open a folder/label) than he can use a proxy server eg.
dovecot which act as a imap proxy to gmail and can be hosted on
closer, faster server and this speed problem never happend. what's
more it can be used as a mail backup server too.

I've thought about a caching IMAP mail storage backend for Dovecot and
even started implementing it once, but it didn't get very far. If
someone wants to try to continue it, I can send what I've written so
far. Although it's mostly a copy of cydir backend with a tiny bit of
it replaced with IMAP code.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20080617/e4a61174/attachment-0001.bin

[Timo Sirainen]

Post by Timo Sirainen

Post by Farkas Levente
would it be possible to create a dovecot server as an imap proxy
for gmail and google apps?
imho many people/company would be love to use such a setup where he
can use gmail in vacation or any other time, but when the email is
real critical apps (not acceptable to wait 1 minutes for an email
to load or to open a folder/label) than he can use a proxy server
eg. dovecot which act as a imap proxy to gmail and can be hosted on
closer, faster server and this speed problem never happend. what's
more it can be used as a mail backup server too.

I've thought about a caching IMAP mail storage backend for Dovecot
and even started implementing it once, but it didn't get very far.
If someone wants to try to continue it, I can send what I've written
so far. Although it's mostly a copy of cydir backend with a tiny bit
of it replaced with IMAP code.

Now that I think of it, the IMAP backend should do only direct IMAP
backend accessing and if caching is wanted it could be a generic
middle layer. Might be also useful for other things like caching mails
from NFS server. Although Linux already has a cachefs for that.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20080617/636c582d/attachment.bin

[Farkas Levente]

Post by Timo Sirainen

Post by Timo Sirainen

Post by Farkas Levente
would it be possible to create a dovecot server as an imap proxy for
gmail and google apps?
imho many people/company would be love to use such a setup where he
can use gmail in vacation or any other time, but when the email is
real critical apps (not acceptable to wait 1 minutes for an email to
load or to open a folder/label) than he can use a proxy server eg.
dovecot which act as a imap proxy to gmail and can be hosted on
closer, faster server and this speed problem never happend. what's
more it can be used as a mail backup server too.

I've thought about a caching IMAP mail storage backend for Dovecot and
even started implementing it once, but it didn't get very far. If
someone wants to try to continue it, I can send what I've written so
far. Although it's mostly a copy of cydir backend with a tiny bit of
it replaced with IMAP code.

Now that I think of it, the IMAP backend should do only direct IMAP
backend accessing and if caching is wanted it could be a generic middle
layer. Might be also useful for other things like caching mails from NFS
server. Although Linux already has a cachefs for that.

yes, that's exactly what i'm thinking about. all imap command which
issued by the client to the middle layer should have to propagate back
to the backed (in this case gmail). but imho the hard part that we've to
assume that the backend (in this case gmail) is slow, so the middle
layer have to cache the imap commands to be able to propagate all
command to the backend (and command queue for all mailbox). so if the
connection between dovecot and gmail are fast then both has the same
messages and flags, etc. if the connection is slow than gmail may have a
small delay, but this delay probably won't be visible to the user (if he
not use the web interface at the same time too).

[Farkas Levente]

Post by Timo Sirainen

Post by Timo Sirainen

Post by Farkas Levente
would it be possible to create a dovecot server as an imap proxy for
gmail and google apps?
imho many people/company would be love to use such a setup where he
can use gmail in vacation or any other time, but when the email is
real critical apps (not acceptable to wait 1 minutes for an email to
load or to open a folder/label) than he can use a proxy server eg.
dovecot which act as a imap proxy to gmail and can be hosted on
closer, faster server and this speed problem never happend. what's
more it can be used as a mail backup server too.

I've thought about a caching IMAP mail storage backend for Dovecot and
even started implementing it once, but it didn't get very far. If
someone wants to try to continue it, I can send what I've written so
far. Although it's mostly a copy of cydir backend with a tiny bit of
it replaced with IMAP code.

Now that I think of it, the IMAP backend should do only direct IMAP
backend accessing and if caching is wanted it could be a generic middle
layer. Might be also useful for other things like caching mails from NFS
server. Although Linux already has a cachefs for that.

it seems other people also thing about this:
http://community.livejournal.com/linux/1748995.html
so it'd be useful thing for many of us:-)

[Asheesh Laroia]

hi,
may be this is a bit of topic, but i hope not. it's always a big question
which is the better to choose a hosting provider to keep and manage you mail
or setup your own mail server, hosting, virus and spam filter etc.. now as
more and more people and company move to gmail or google apps mail service it
seems google can't handle the load and it's getting more and more slower.
what's more not just the imap interface but nowadays the web interface used
to be hang or even stop working. it's getting more and more annoying that the
speed is worst than in a 10 years ago. i thing about how can this be solved
without totally give up gmail as a mail service provider (and may be they can
solve it in a few months/years:-).
would it be possible to create a dovecot server as an imap proxy for gmail
and google apps?

What I have is:

* rose, my primary server (but far away)
* supercore, another server at my parents' house

rose is the primary MX.

I run offlineimap on supercore to sync a local Maildir on supercore with
rose's IMAP - that way, my parents can use a server with <1ms ping, and as
they use Dovecot to modify the local Maildir on supercore, a minute
later offlineimap will wake up and create IMAP operations to synchronize
rose with the state on supercore.

FWIW.

-- Asheesh.

[Farkas Levente]

Post by Asheesh Laroia

Post by Farkas Levente
hi,
may be this is a bit of topic, but i hope not. it's always a big
question which is the better to choose a hosting provider to keep and
manage you mail or setup your own mail server, hosting, virus and spam
filter etc.. now as more and more people and company move to gmail or
google apps mail service it seems google can't handle the load and
it's getting more and more slower. what's more not just the imap
interface but nowadays the web interface used to be hang or even stop
working. it's getting more and more annoying that the speed is worst
than in a 10 years ago. i thing about how can this be solved without
totally give up gmail as a mail service provider (and may be they can
solve it in a few months/years:-).
would it be possible to create a dovecot server as an imap proxy for
gmail and google apps?

* rose, my primary server (but far away)
* supercore, another server at my parents' house
rose is the primary MX.
I run offlineimap on supercore to sync a local Maildir on supercore with
rose's IMAP - that way, my parents can use a server with <1ms ping, and
as they use Dovecot to modify the local Maildir on supercore, a minute
later offlineimap will wake up and create IMAP operations to synchronize
rose with the state on supercore.

i'm just look into offlineimap. half of the code for ui, another part to
implement maildir and imap. so it seems for me there is only a little
code which do the synchronization.
Timo what do you think how much work would be add such a middleware imap
server capabilities (ie. imap backend) to dovecot?

[Timo Sirainen]

Post by Farkas Levente
Timo what do you think how much work would be add such a middleware
imap server capabilities (ie. imap backend) to dovecot?

I think it would be quite a lot of work. Definitely a lot more than I
have time for currently. The most difficult thing is probably writing
IMAP parsing and event handling for it. I've written such code for my
imaptest tool already though, maybe at some point they could be more
or less merged to a common lib-imap-client.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20080618/d47717d5/attachment.bin