dovecot replication (active-active) - server specs

[Martin Schmidt]

Hello,

i have some questions about the new dovecot replication and mdbox format.

my company has currently 3 old dovecot 2.0.x fileserver/backend with ca. 120k mailboxes and ca. 6 TB data used.
They are synchronised per drbd/corosync.
Each fileserver/backend have ca. 40k mailboxes im Maildir format.

Our MX server is delivering ca. 30 GB new mails per day.
Two IMAP proxy server get the connections from the users. Atm. without dovecot director.
We've got around 700k connections per day (imap 200k / pop3 500k)

The system is getting issues because the fileserver still have old slow HDDs.
Users sometime get connection timeouts, because the fileserver can not answer fast enough due to I/O waiting lag.

So we want to make a new system.
We desire the new system to use mdbox format ( bigger files, less I/O)
and replication through dovecot replication (active/active) instead of drbd.
Each fileserver should know every mailbox/user and for the time being 2 dovecot proxies for the user connections (IMAP/POP).
(later after the migration from the old system to the new, dovecot director instead of proxies, for caching reasons).

we've got 2 new fileservers, they have each SSD HDDs for "new-storage"
and 7200rpm SATA HDDs on RAID 5 with 10 TB for "alt-storage"
32 GB RAM per Server

Do you have some tips for the system?
Do you believe 32 GB RAM are enough for one fileserver each and have you experience with the I/O Waiting problem with huge amounts of Data on the alt-storage?
Could there be issues with the RAM, if one fileserver has a downtime, so the second one has to take over all mailboxes for a short amount of time?

In general are only 2 new fileserver enough or should we think in bigger dimensions, like 4 fileserver
Storage expansion in the new servers should not be a problem (bigger HDDs and a few slots free, so we can expand the raid 5).


thank you
kind regards

Martin Schmidt

[Florian Pritz]

So we want to make a new system. We desire the new system to use
mdbox format ( bigger files, less I/O)

Be sure to enable compression when do do that move (you'd have to do
pretty much the same migration again if you enable it later) as per
http://wiki2.dovecot.org/Plugins/Zlib

Just in case: You'll probably also want to increase mdbox_rotate_size
from the default 2m to something along the lines of 10m to 100m (This
also only affects new files so should be done early)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://dovecot.org/pipermail/dovecot/attachments/20141009/7cba033c/attachment.sig>

[Urban Loesch]

Hi,

Post by Martin Schmidt
Our MX server is delivering ca. 30 GB new mails per day.
Two IMAP proxy server get the connections from the users. Atm. without dovecot director.
We've got around 700k connections per day (imap 200k / pop3 500k)

Are this the hole connections per day? How many concurrend connections
do you have at the same time on each server?

Post by Martin Schmidt
So we want to make a new system.
We desire the new system to use mdbox format ( bigger files, less I/O)
and replication through dovecot replication (active/active) instead of drbd.

I have no experience with dovecot replication (Still on our roadmap). We
are currently using drbd on a 10Gbit dedicated link. Works very well for us.

Post by Martin Schmidt
Each fileserver should know every mailbox/user and for the time being 2 dovecot proxies for the user connections (IMAP/POP).
(later after the migration from the old system to the new, dovecot director instead of proxies, for caching reasons).

As Florian said, enable zlib. This also decreases I/O, but needs a bit
more of CPU. But not that much.

Post by Martin Schmidt
we've got 2 new fileservers, they have each SSD HDDs for "new-storage"
and 7200rpm SATA HDDs on RAID 5 with 10 TB for "alt-storage"
32 GB RAM per Server

You also could move the INDEX files from mdbox to different SSDs. We are
doing so with 40k accounts and 2TB user data. Index partition has only
22GB used and is increasing not very fast.

Post by Martin Schmidt
Do you have some tips for the system?
Do you believe 32 GB RAM are enough for one fileserver each and have you experience with the I/O Waiting problem with huge amounts of Data on the alt-storage?
Could there be issues with the RAM, if one fileserver has a downtime, so the second one has to take over all mailboxes for a short amount of time?

I think memory is not the problem. On IMAP/POP3 servers the main
problem is I/O. But with dovecot mdbox and index files on SSD's we have
no problem at the moment.

Post by Martin Schmidt
In general are only 2 new fileserver enough or should we think in bigger dimensions, like 4 fileserver
Storage expansion in the new servers should not be a problem (bigger HDDs and a few slots free, so we can expand the raid 5).

We are using raid 10 hardware raid controller with cache and sata
7200rpm disks. OK, raid 10 needs more disks, but is much faster than
raid 5. Raid 5 is not very fast in my eyes.

Post by Martin Schmidt
thank you
kind regards
Martin Schmidt

Regards
Urban

[Martin Schmidt]

Hello

Post by Urban Loesch
Hi,

Post by Martin Schmidt
Our MX server is delivering ca. 30 GB new mails per day.
Two IMAP proxy server get the connections from the users. Atm. without dovecot director.
We've got around 700k connections per day (imap 200k / pop3 500k)

Are this the hole connections per day? How many concurrend connections
do you have at the same time on each server?

we've got 3 Fileserver with ca. 1200 concurrend IMAP connections and ca.
50 concurrend POP3 connections on each server.

Post by Urban Loesch

Post by Martin Schmidt
So we want to make a new system.
We desire the new system to use mdbox format ( bigger files, less I/O)
and replication through dovecot replication (active/active) instead of drbd.

I have no experience with dovecot replication (Still on our roadmap).
We are currently using drbd on a 10Gbit dedicated link. Works very
well for us.

Post by Martin Schmidt
Each fileserver should know every mailbox/user and for the time being
2 dovecot proxies for the user connections (IMAP/POP).
(later after the migration from the old system to the new, dovecot
director instead of proxies, for caching reasons).

As Florian said, enable zlib. This also decreases I/O, but needs a bit
more of CPU. But not that much.

Yes we have enabled it, estimated space saving is up to 40%

Post by Urban Loesch

Post by Martin Schmidt
we've got 2 new fileservers, they have each SSD HDDs for "new-storage"
and 7200rpm SATA HDDs on RAID 5 with 10 TB for "alt-storage"
32 GB RAM per Server

You also could move the INDEX files from mdbox to different SSDs. We
are doing so with 40k accounts and 2TB user data. Index partition has
only 22GB used and is increasing not very fast.

On our testsystem we've got it also on a raid 1 SSD, only alt-storage is
on raid 5. Looks good.

Post by Urban Loesch

Post by Martin Schmidt
Do you have some tips for the system?
Do you believe 32 GB RAM are enough for one fileserver each and have
you experience with the I/O Waiting problem with huge amounts of Data
on the alt-storage?
Could there be issues with the RAM, if one fileserver has a downtime,
so the second one has to take over all mailboxes for a short amount
of time?

I think memory is not the problem. On IMAP/POP3 servers the main
problem is I/O. But with dovecot mdbox and index files on SSD's we
have no problem at the moment.

On each of our 3 Fileserver we've got 16 GB RAM, 5-7 GB is used and rest
is cached. You might be right, the i/o is always the bottleneck.

Post by Urban Loesch

Post by Martin Schmidt
In general are only 2 new fileserver enough or should we think in
bigger dimensions, like 4 fileserver
Storage expansion in the new servers should not be a problem (bigger
HDDs and a few slots free, so we can expand the raid 5).

We are using raid 10 hardware raid controller with cache and sata
7200rpm disks. OK, raid 10 needs more disks, but is much faster than
raid 5. Raid 5 is not very fast in my eyes.

We've made tests with raid 10 and raid 5, on 4 sata 7200rpm disks, of
course raid 10 was faster, but overall not very much. And you can expand
raid 5 easier.
Can you tell me, if you have a high "Waiting" on your alt-storage?

Post by Urban Loesch

Post by Martin Schmidt
thank you
kind regards
Martin Schmidt

Regards
Urban

Thank you for your impressions.

kr
Martin Schmidt

[Ed W]

Post by Martin Schmidt
we've got 2 new fileservers, they have each SSD HDDs for "new-storage"
and 7200rpm SATA HDDs on RAID 5 with 10 TB for "alt-storage"

Friends don't let friends use Raid5...
http://www.baarf.com/

(Use Raid6 or something else...)

Note, a common counter argument is that someone has "full backups and
can survive a rebuild, so the RAID5 is really just to increase uptime".
I suggest you do the sums on silent corruption and compare with your
data size. Bit rot seems to be an observable problem now. Scrub your
arrays regularly and where possible use data integrity checks at higher
levels (not much for linux, but ZFS offers this for other OSs)

Good luck

Ed W