Announce: Plugin for iOS Push Email support

Discussion:

Stefan Arentz

2014-09-05 01:35:22 UTC

I?ve been hacking on a personal side project to support native iOS Push Email in Dovecot. This is specifically for people who are migrating their mail away from OS X Server while keeping their existing Push Email functionality.

Native Push Email has some great advantages: it speeds up email notifications (usually within seconds of being handled by dovecot-lda) and it improves battery life since the native notifications flow over a single highly optimized connection to Apple?s infrastructure.

Although this is at version 0.1, it is working pretty well for me and I am looking for some additional testers that are interested.

Please note that it is not possible to use this project without legally running a copy
of OS X Server. You can purchase OS X Server on the Mac App Store or download
it for free if you are a registered Mac or iOS developer.

This feature is enabled by two projects:

https://github.com/st3fan/dovecot-xaps-plugin <https://github.com/st3fan/dovecot-xaps-plugin>
https://github.com/st3fan/dovecot-xaps-daemon <https://github.com/st3fan/dovecot-xaps-daemon>

Both projects contain a README that describes how to get things going. I have only developed and tested on Ubuntu 12.04.5 so ideally you run the same version if you are interested in playing with this project.

You will need to compile some code and not be afraid of a little admin work.

I am also interested in a code review. The Dovecot plugin API is barely documented and I had to guess a lot of things by looking at other plugins. An extra pair of eyes specifically on that code would be awesome. Also from a security perspective.

Please file bugs!

S.

admin

2014-09-05 07:55:35 UTC

Permalink

This is interesting; I'm not sure if I got that right from scanning
the READMEs. Do I really need to have a working OSX-server setup
which I then do migrate away from? What makes this certificate that
special such that I only can export it from the existing server
setup?

-M

Post by Stefan Arentz
I?ve been hacking on a personal side project to support native iOS Push Email in Dovecot. This is specifically for people who are migrating their mail away from OS X Server while keeping their existing Push Email functionality.
Native Push Email has some great advantages: it speeds up email notifications (usually within seconds of being handled by dovecot-lda) and it improves battery life since the native notifications flow over a single highly optimized connection to Apple?s infrastructure.
Although this is at version 0.1, it is working pretty well for me and I am looking for some additional testers that are interested.
Please note that it is not possible to use this project without legally running a copy
of OS X Server. You can purchase OS X Server on the Mac App Store or download
it for free if you are a registered Mac or iOS developer.
https://github.com/st3fan/dovecot-xaps-plugin <https://github.com/st3fan/dovecot-xaps-plugin>
https://github.com/st3fan/dovecot-xaps-daemon <https://github.com/st3fan/dovecot-xaps-daemon>
Both projects contain a README that describes how to get things going. I have only developed and tested on Ubuntu 12.04.5 so ideally you run the same version if you are interested in playing with this project.
You will need to compile some code and not be afraid of a little admin work.
I am also interested in a code review. The Dovecot plugin API is barely documented and I had to guess a lot of things by looking at other plugins. An extra pair of eyes specifically on that code would be awesome. Also from a security perspective.
Please file bugs!
S.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5820 bytes
Desc: not available
URL: <http://dovecot.org/pipermail/dovecot/attachments/20140905/d2d615d4/attachment.p7s>

Charles Marcus

2014-09-05 09:53:51 UTC

Permalink

Post by Stefan Arentz
Although this is at version 0.1, it is working pretty well for me and I am looking for some additional testers that are interested.
Please note that it is not possible to use this project without legally running a copy
of OS X Server. You can purchase OS X Server on the Mac App Store or download
it for free if you are a registered Mac or iOS developer.

Well, that is pretty much a deal breaker for just about everyone (that
isn't already running an OSX server)...

Would you mind explaining *why* the above is required? Is this just a
temporary limitation of some kind?

Stefan Arentz

2014-09-05 13:22:05 UTC

Permalink

Well, that is pretty much a deal breaker for just about everyone (that isn't already running an OSX server)...
Would you mind explaining *why* the above is required? Is this just a temporary limitation of some kind?

Sorry I should have been more clear.

Let me explain.

To send native push email notifications, you need a certificate that is generated by Apple.

When you enable Push Email on OS X Server, that certificate will be generated for you and stored in your Keychain.

There is no other way to generate this certificate. This is why you need a legal copy of OS X Server.

You don?t actually have to run OS X Server. I migrated away from OS X Server to regular Dovecot on Ubuntu. So I simply took the generated certificate with me. How to export and convert it is described in the README of the daemon project.

This is possible because the certificate is not bound to specific hardware or a specific email domain. It is however connected to your Apple ID.

I understand that this is not ideal but unfortunately there is no way around this. OS X Server however is just $19.99.

When my code is a bit more final, I will try to contact Apple and find out if they are willing to open up the certificate generation. That would sure benefit a larger audience.

S.

Charles Marcus

2014-09-05 13:23:54 UTC

Permalink

Post by Stefan Arentz
When you enable Push Email on OS X Server, that certificate will be generated for you and stored in your Keychain.
There is no other way to generate this certificate. This is why you need a legal copy of OS X Server.
You don?t actually have to run OS X Server. I migrated away from OS X Server to regular Dovecot on Ubuntu. So I simply took the generated certificate with me. How to export and convert it is described in the README of the daemon project.
This is possible because the certificate is not bound to specific hardware or a specific email domain. It is however connected to your Apple ID.
I understand that this is not ideal but unfortunately there is no way around this. OS X Server however is just $19.99.
When my code is a bit more final, I will try to contact Apple and find out if they are willing to open up the certificate generation. That would sure benefit a larger audience.

Ok, thanks for the explanation...

Gotta love (hate) closed source software...

Daniel Reynolds

2014-09-05 13:28:46 UTC

Permalink

"... I will try to contact Apple and find out if they are willing to open
up the certificate generation. "

I'm willing to bet the answer will be no.

Apple != Open

Post by Stefan Arentz

On Sep 5, 2014, at 5:53 AM, Charles Marcus <CMarcus at Media-Brokers.com>

Post by Stefan Arentz
Although this is at version 0.1, it is working pretty well for me and I

am looking for some additional testers that are interested.

Post by Stefan Arentz
Please note that it is not possible to use this project without

legally running a copy

Post by Stefan Arentz
of OS X Server. You can purchase OS X Server on the Mac App Store

or download

Post by Stefan Arentz
it for free if you are a registered Mac or iOS developer.

Well, that is pretty much a deal breaker for just about everyone (that

isn't already running an OSX server)...

Would you mind explaining *why* the above is required? Is this just a

temporary limitation of some kind?
Sorry I should have been more clear.
Let me explain.
To send native push email notifications, you need a certificate that is generated by Apple.
When you enable Push Email on OS X Server, that certificate will be
generated for you and stored in your Keychain.
There is no other way to generate this certificate. This is why you need a
legal copy of OS X Server.
You don?t actually have to run OS X Server. I migrated away from OS X
Server to regular Dovecot on Ubuntu. So I simply took the generated
certificate with me. How to export and convert it is described in the
README of the daemon project.
This is possible because the certificate is not bound to specific hardware
or a specific email domain. It is however connected to your Apple ID.
I understand that this is not ideal but unfortunately there is no way
around this. OS X Server however is just $19.99.
When my code is a bit more final, I will try to contact Apple and find out
if they are willing to open up the certificate generation. That would sure
benefit a larger audience.
S.

admin

2014-09-05 13:40:56 UTC

Permalink

No matter what the answer's gonna be the "solution" till then still might be
the z-push+exchange-approach? Or did I miss anything more sophisticated out
there?

Post by Stefan Arentz

Well, that is pretty much a deal breaker for just about everyone (that isn't already running an OSX server)...
Would you mind explaining *why* the above is required? Is this just a temporary limitation of some kind?

Sorry I should have been more clear.
Let me explain.
To send native push email notifications, you need a certificate that is generated by Apple.
When you enable Push Email on OS X Server, that certificate will be generated for you and stored in your Keychain.
There is no other way to generate this certificate. This is why you need a legal copy of OS X Server.
You don?t actually have to run OS X Server. I migrated away from OS X Server to regular Dovecot on Ubuntu. So I simply took the generated certificate with me. How to export and convert it is described in the README of the daemon project.
This is possible because the certificate is not bound to specific hardware or a specific email domain. It is however connected to your Apple ID.
I understand that this is not ideal but unfortunately there is no way around this. OS X Server however is just $19.99.
When my code is a bit more final, I will try to contact Apple and find out if they are willing to open up the certificate generation. That would sure benefit a larger audience.
S.

Alessio Cecchi

2014-09-12 10:14:20 UTC

Permalink

Hi Stefan,

I have installed your code on my personal email server (Dovecot 2.2.13
and Debian 7.6) and works fine! The only note is to copy the plugins
file (imap_xaps_plugin.so and xaps_plugin.so) manually in /usr/lib/dovecot/.

I hope you will release a version with the ability to run in background
the daemon.

Thanks

Timo Sirainen

2014-09-15 12:58:44 UTC

Permalink

Nice. I had been planning to look into this as well, but never had much time. Some comments from a quick look:

- xaps_str_append_quoted() doesn't quote CR/LF/TABs. Of course those shouldn't normally exist in any of the fields, but older Dovecot versions would have allowed creating folder names with them. Using the str_append_tabescaped() would escape them as well, although then you'd have to implement the unescaping also with Python.

- ideally the communication between Dovecot and xaps-daemon would be done asynchronously, but as a quick change you could also use i_stream_create_fd() + i_stream_read_next_line() instead of the strtok_r call. Oh and in internal Dovecot protocols it has always used LF, not CRLF, which would also make it a bit easier to handle.

- in xaps_notify() especially async IO would be much better to avoid latency. It's also easier to do than in init, since it wouldn't really even need to wait for the response. Even without async IO does it really even need to read the answer from the server?

- also to avoid bursts of notifications in some situations, you could use timeout_add(NOTIFY_DELAY_MSECS, ...) and have the timeout function send the notification. The delay being for example 0,5 seconds or maybe configurable.

BTW. We could have some other Dovecot plugins that need to be developed if you want more (paid) work. :) We need to hire more developers to reduce my work load..

Stefan Arentz

2014-09-23 13:48:57 UTC

Permalink

Post by Timo Sirainen

- xaps_str_append_quoted() doesn't quote CR/LF/TABs. Of course those shouldn't normally exist in any of the fields, but older Dovecot versions would have allowed creating folder names with them. Using the str_append_tabescaped() would escape them as well, although then you'd have to implement the unescaping also with Python.
- ideally the communication between Dovecot and xaps-daemon would be done asynchronously, but as a quick change you could also use i_stream_create_fd() + i_stream_read_next_line() instead of the strtok_r call. Oh and in internal Dovecot protocols it has always used LF, not CRLF, which would also make it a bit easier to handle.
- in xaps_notify() especially async IO would be much better to avoid latency. It's also easier to do than in init, since it wouldn't really even need to wait for the response. Even without async IO does it really even need to read the answer from the server?
- also to avoid bursts of notifications in some situations, you could use timeout_add(NOTIFY_DELAY_MSECS, ...) and have the timeout function send the notification. The delay being for example 0,5 seconds or maybe configurable.

Hi Timo,

Thank you for these hints. I appreciate your input. This all sounds pretty good and I will try to make these changes soon and push out a new release.

For people who are interested in trying our this project, I have made considerable improvements to the daemon side of things. It is now a proper Twisted application that you can start using twistd, so that it goes in the background, writes a pid file, etc.

See https://github.com/st3fan/dovecot-xaps-daemon/blob/master/README.md#installing-and-running-the-daemon

S.

Patrick Domack

2014-09-23 14:47:34 UTC

Permalink

What would happen if say, the dovegot xaps plugin is installed and the
daemon crashed?

Would peoples iphones just wait for the push notification? get email
every like 30/60min? or would it just use imap idle instead?

Just wondering about sideeffects :)

Post by Stefan Arentz

Post by Timo Sirainen

Nice. I had been planning to look into this as well, but never had
- xaps_str_append_quoted() doesn't quote CR/LF/TABs. Of course
those shouldn't normally exist in any of the fields, but older
Dovecot versions would have allowed creating folder names with
them. Using the str_append_tabescaped() would escape them as well,
although then you'd have to implement the unescaping also with
Python.
- ideally the communication between Dovecot and xaps-daemon would
be done asynchronously, but as a quick change you could also use
i_stream_create_fd() + i_stream_read_next_line() instead of the
strtok_r call. Oh and in internal Dovecot protocols it has always
used LF, not CRLF, which would also make it a bit easier to handle.
- in xaps_notify() especially async IO would be much better to
avoid latency. It's also easier to do than in init, since it
wouldn't really even need to wait for the response. Even without
async IO does it really even need to read the answer from the server?
- also to avoid bursts of notifications in some situations, you
could use timeout_add(NOTIFY_DELAY_MSECS, ...) and have the timeout
function send the notification. The delay being for example 0,5
seconds or maybe configurable.

Hi Timo,
Thank you for these hints. I appreciate your input. This all sounds
pretty good and I will try to make these changes soon and push out a
new release.
For people who are interested in trying our this project, I have
made considerable improvements to the daemon side of things. It is
now a proper Twisted application that you can start using twistd, so
that it goes in the background, writes a pid file, etc.
See
https://github.com/st3fan/dovecot-xaps-daemon/blob/master/README.md#installing-and-running-the-daemon
S.