[Dovecot] LDA vs. LMTP

[Martin Burgraf]

Hi there,

I'm using Dovecot together with Postfix; as I understand it, there are two ways to transfer the mail from Postfix to Dovecot.
1.) by using LDA with mailbox_command = /usr/libexec/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"
2.) by using LMTP with mailbox_transport = lmtp:unix:private/dovecot-lmtp

(currently using number 1)
I'm interessted in the differences and the advantages/disadvantages of each of those solutions.

According to http://wiki2.dovecot.org/LDA the recommended way is to use LMTP, since it's supposed to have a better performance.
On the other hand, http://wiki2.dovecot.org/LMTP says, that LMTP is a backgound process, while LDA is only called when needed. I've also read, that LDA only uses the users privileges, which both means, that LDA should be better.
I've also noticed, that LMTP adds an additional Recieved:-Header to the mail.
Are there any other differences?

Thank you
M.

[Reindl Harald]

Post by Martin Burgraf
I'm using Dovecot together with Postfix; as I understand it, there are two ways to transfer the mail from Postfix to Dovecot.
1.) by using LDA with mailbox_command = /usr/libexec/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"
2.) by using LMTP with mailbox_transport = lmtp:unix:private/dovecot-lmtp
(currently using number 1)
I'm interessted in the differences and the advantages/disadvantages of each of those solutions
According to http://wiki2.dovecot.org/LDA the recommended way is to use LMTP, since it's supposed to have a better performance
On the other hand, http://wiki2.dovecot.org/LMTP says, that LMTP is a backgound process, while LDA is only called when needed

and that is why LMTP is preferred

instead fire up a new process for each message with all the costs
you have *one* process running all the time waiting for a
message to deliver

you would no run SMTPD via xinetd and start the smtpd service
each time someone delivers a message to your server...

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 263 bytes
Desc: OpenPGP digital signature
URL: <http://dovecot.org/pipermail/dovecot/attachments/20130727/f601a17d/attachment.bin>

[Joseph Tam]

Post by Martin Burgraf
According to http://wiki2.dovecot.org/LDA the recommended way is to use
LMTP, since it's supposed to have a better performance.

The performance gains comes mostly from avoiding the overhead of invoking
an executable and spawning a new process for each delivery. If your mail
system isn't stressed, I don't think it matters much.

Post by Martin Burgraf
On the other hand, http://wiki2.dovecot.org/LMTP says, that LMTP is a
backgound process, while LDA is only called when needed. I've also
read, that LDA only uses the users privileges, which both means, that
LDA should be better.

I don't know why you would consider a background process inferior to a
run-on-demand executable.

Post by Martin Burgraf
I've also noticed, that LMTP adds an additional Recieved:-Header to the mail.
Are there any other differences?
From a past discussion on this topic, I think Timo stated that if you

use SIS (single-instance storage or de-duping), it's more efficient
using LMTP since it knows all message bodies to multiple recipients will
be identical.

Joseph Tam <jtam.home at gmail.com>

[Steffen Kaiser]

Post by Martin Burgraf
I'm using Dovecot together with Postfix; as I understand it, there are two ways to transfer the mail from Postfix to Dovecot.
1.) by using LDA with mailbox_command = /usr/libexec/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"
2.) by using LMTP with mailbox_transport = lmtp:unix:private/dovecot-lmtp
(currently using number 1)
I'm interessted in the differences and the advantages/disadvantages of each of those solutions.
According to http://wiki2.dovecot.org/LDA the recommended way is to use LMTP, since it's supposed to have a better performance.
On the other hand, http://wiki2.dovecot.org/LMTP says, that LMTP is a backgound process, while LDA is only called when needed. I've also read, that LDA only uses the users privileges, which both means, that LDA should be better.
I've also noticed, that LMTP adds an additional Recieved:-Header to the mail.
Are there any other differences?

LMTP also adds "Delivered-To", unless I'm mistaken.

There is one difference, that pops up on failure: The LDA has the exit
code only to return success/failure back to the MTA. LMTP uses the same
mechanisms as SMTP to return success / failure incl. descriptive
information.

There is another difference, if you need additional hacking: With the
LDA-method you can put a wrapper script between MTA and MDA, in order to
alter the message, recipient, just log something, ... . Actually that
self-made wrapper script [and I really mean script in the sense of bash,
perl, python, C, ruby, ...] can control the delivery fully. That would be
more sophisticated to do with LMTP.

- --
Steffen Kaiser

[Jan Behrend]

Post by Martin Burgraf
Hi there,
I'm using Dovecot together with Postfix; as I understand it, there are two ways to transfer the mail from Postfix to Dovecot.
1.) by using LDA with mailbox_command = /usr/libexec/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"
2.) by using LMTP with mailbox_transport = lmtp:unix:private/dovecot-lmtp
(currently using number 1)
I'm interessted in the differences and the advantages/disadvantages of each of those solutions.

You cannot use the LDA method if SMTP and IMAP services reside on
different machines, which would be the case in larger scale mail system
setups.

My advice is to go with LMTP anyway!

Cheers Jan
--
MAX-PLANCK-INSTITUT fuer Radioastronomie
Jan Behrend - Rechenzentrum
----------------------------------------
Auf dem Huegel 69, D-53121 Bonn
Tel: [phone], Fax: [phone]
jbehrend at mpifr-bonn.mpg.de http://www.mpifr-bonn.mpg.de

------------------------------------------------------------------------
Die digitale Unterschrift dieser Mail kann durch das Zertifikat der
DFN Global Hierarchie ?berpr?ft werden:
https://ca.mpg.de/certs/root-DGP/deutsche-telekom-ca2-root-cert.der
Weitere Informationen zur CA der MPG finden Sie unter: https://ca.mpg.de
------------------------------------------------------------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4553 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://dovecot.org/pipermail/dovecot/attachments/20130729/8bdc816d/attachment.bin>

[Noel Butler]

Post by Jan Behrend
You cannot use the LDA method if SMTP and IMAP services reside on
different machines, which would be the case in larger scale mail system
setups.

Sorry, that is incorrect.

Granted, it does mean putting dovecot on the SMTP servers as well, but
you certainly do not need to allow pop3/imap access.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: This is a digitally signed message part
URL: <http://dovecot.org/pipermail/dovecot/attachments/20130729/02db04e6/attachment.bin>

[Stan Hoeppner]

Post by Jan Behrend
You cannot use the LDA method if SMTP and IMAP services reside on
different machines, which would be the case in larger scale mail system
setups.

Which brings up an interesting point. With a single LMTP daemon on the
Dovecot server communicating via a single socket with the upstream MTA
over the wire, it would stand to reason that message throughput rate may
be limited by serialization in the LMTP request/reply chain. There is
no parallelism, and thus there is relatively high latency.

In the case of LDA with an SMTP MTA on the local box, the potential
exists for very high parallelism, and thus elimination of the latency in
serial delivery over a single socket with LMTP.

So in theory, while LDA in this scenario would consume far more
resources with a very high message load, one should be able to attain
much higher message throughput. I say in theory because I've not tested
this head to head.

[Ben Morrow]

Post by Stan Hoeppner

Post by Jan Behrend
You cannot use the LDA method if SMTP and IMAP services reside on
different machines, which would be the case in larger scale mail system
setups.

Which brings up an interesting point. With a single LMTP daemon on the
Dovecot server communicating via a single socket with the upstream MTA
over the wire, it would stand to reason that message throughput rate may
be limited by serialization in the LMTP request/reply chain. There is
no parallelism, and thus there is relatively high latency.

What makes you think an SMTP server delivering over LMTP only makes a
single connection to the LMTP server? I believe Postfix by default makes
a fresh connection for each delivery.

Ben

[Stan Hoeppner]

Post by Ben Morrow

Post by Stan Hoeppner

Post by Jan Behrend
You cannot use the LDA method if SMTP and IMAP services reside on
different machines, which would be the case in larger scale mail system
setups.

Which brings up an interesting point. With a single LMTP daemon on the
Dovecot server communicating via a single socket with the upstream MTA
over the wire, it would stand to reason that message throughput rate may
be limited by serialization in the LMTP request/reply chain. There is
no parallelism, and thus there is relatively high latency.

You snipped the text where I stated this is a theoretical discussion,
due to the high msg volume required to prove one over the other. That
said, I'll gladly continue to postulate on the theoretical.

Post by Ben Morrow
What makes you think an SMTP server delivering over LMTP only makes a
single connection to the LMTP server? I believe Postfix by default makes
a fresh connection for each delivery.

No, Postfix by default uses connection caching w/both SMTP and LMTP:
http://www.postfix.org/postconf.5.html#lmtp_cache_connection

If the load is sufficiently high it will open additional connections,
but it attempts to reuse existing connections as much as possible to
eliminate additional connection setup delays, which can be considerable
with SMTP servers. For instance some OPs insert 2 minute or longer
greet delays as a (very crude) anti spam bot measure. Connection
caching is an SMTP optimization, and not nearly as beneficial to LMTP.
The Postfix SMTP/LMTP clients are literally the same code.

[Martin Burgraf]

Post by Joseph Tam
I don't know why you would consider a background process inferior to a
run-on-demand executable.

Well, the background process is hogging CPU and RAM while it basically does nothing. And when it's running as root there is always the danger of privilege escalation.
LDA only runs when it's needed and since it uses only user rights it shoudbe more harmless.


bye
Martin

[Reindl Harald]

Post by Martin Burgraf
Well, the background process is hogging CPU

why should it do that if it is idle?

Post by Martin Burgraf
and RAM while it basically does nothing.

guess what takes more RAM

one long-running prcoess or 5 LDA processes because
you get 5 messages at the same time and guess what
takes more CPU - a idle process waiting on a new
connection or fireup a whole new process all day long

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 263 bytes
Desc: OpenPGP digital signature
URL: <http://dovecot.org/pipermail/dovecot/attachments/20130730/667db3b9/attachment.bin>

[Joseph Tam]

Post by Martin Burgraf

Post by Joseph Tam
I don't know why you would consider a background process inferior to a
run-on-demand executable.

Well, the background process is hogging CPU and RAM while it basically
does nothing.

"Hogging" CPU and memory is putting it strongly, as it is basically
suspended while blocked on waiting for a connection, and if left for a
long time in an idle state, might be swapped out to disk and not consuming
(real) memory, or consuming real memory that isn't in use otherwise.

As I stated before, the resource usage is small compared with all the
other stuff going on, so if you don't have a busy mail server, I don't
think you should sweat the difference.

The benefits of LMTP should increase with load, as having LMTP resident
will save you the overhead of repeatedly loading/unloading LDA, and
I'm sure the CPU and memory usage of servicing that overhead will be
non-trivial. If you have oodles of memory, then it's no problem keeping a
LMTP resident. If you don't have enough memory and are VM disk thrashing,
you'll have other problems and LDA/LMTP is the least of your worries.

Post by Martin Burgraf
And when it's running as root there is always the danger
of privilege escalation. LDA only runs when it's needed and since it
uses only user rights it shoudbe more harmless.

I didn't contest the privilege separation aspect, as it a necessary
design trade-off that one daemon doing things for all user will need
overriding access. However, if this is a concern, you can virtualize
all your users. LMTP can theoretically be subverted, but at least won't
be as root. (I'm assuming LMTP stays as root, and not spawning off user
processes to do the real work.)

Joseph Tam <jtam.home at gmail.com>

[Ben Morrow]

Post by Joseph Tam

Post by Martin Burgraf
And when it's running as root there is always the danger
of privilege escalation. LDA only runs when it's needed and since it
uses only user rights it shoudbe more harmless.

I didn't contest the privilege separation aspect, as it a necessary
design trade-off that one daemon doing things for all user will need
overriding access. However, if this is a concern, you can virtualize
all your users. LMTP can theoretically be subverted, but at least won't
be as root. (I'm assuming LMTP stays as root, and not spawning off user
processes to do the real work.)

It doesn't stay as root; Dovecot's LMTP switches down to the user's uid
to perform delivery, including sieve scripts. The security concerns are
in fact very similar to LDA: for LDA delivery with (say) Postfix, you
have local(8) running as root and switching down to the user to invoke
the LDA, while for LMTP the Postfix lmtp(8) process runs as an
unprivileged Postfix user and the LMTP server runs as root and switches
down.

AFAICS the LMTP conversation itself happens as root, though, which is a
shame; I might think twice about exposing it directly over the network.

Ben

[Stan Hoeppner]

Post by Ben Morrow

Post by Joseph Tam

Post by Martin Burgraf
And when it's running as root there is always the danger
of privilege escalation. LDA only runs when it's needed and since it
uses only user rights it shoudbe more harmless.

I didn't contest the privilege separation aspect, as it a necessary
design trade-off that one daemon doing things for all user will need
overriding access. However, if this is a concern, you can virtualize
all your users. LMTP can theoretically be subverted, but at least won't
be as root. (I'm assuming LMTP stays as root, and not spawning off user
processes to do the real work.)

It doesn't stay as root; Dovecot's LMTP switches down to the user's uid
to perform delivery, including sieve scripts. The security concerns are
in fact very similar to LDA: for LDA delivery with (say) Postfix, you
have local(8) running as root and switching down to the user to invoke
the LDA, while for LMTP the Postfix lmtp(8) process runs as an
unprivileged Postfix user and the LMTP server runs as root and switches
down.
AFAICS the LMTP conversation itself happens as root, though, which is a
shame; I might think twice about exposing it directly over the network.

Shouldn't a few iptables/pf rules be able to substantially mitigate this
potential problem? I.e. restrict which hosts a given host is allowed to
speak LMTP with.

[Joseph Tam]

(Weird: this message digest got dumped into Google's spam folder. Maybe
it didn't like the string in a later post (obfuscated here) master(dot)cf,
which in the context of this mailing list is a postfix configuration
file, but which Gmail interpret as a website. However, that domain
is a SURBL/DBL blacklisted URI).

Post by Ben Morrow

Post by Joseph Tam

Post by Martin Burgraf
And when it's running as root there is always the danger
of privilege escalation. LDA only runs when it's needed and since it
uses only user rights it shoudbe more harmless.

...
(I'm assuming LMTP stays as root, and not spawning off user
processes to do the real work.)

It doesn't stay as root; Dovecot's LMTP switches down to the user's uid
to perform delivery, including sieve scripts.

I stand corrected. This removes the other objection that the original
poster for running a peristent LMTP process.

Joseph Tam <jtam.home at gmail.com>