quota recalc problem for big maildirs

[Dawid Stawiarski - nazwa.pl]

hi,

we use dovecot 2.2.13, and we've lately started using maildir quota
(previously fs quota was used, right now it's "quota = maildir:User
quota" changed from "quota = fs:User quota:user:inode_per_mail").

When the change was applied we didn't manually forced generating
maildirsize files. So, when the first deliver happens (or the file grows
larger then 5KB) it starts quota recalc. However for users with big
maildirs, receiving email every couple of seconds, there are tens of
delivery processes each doing its own recalc (as they open file for
APPEND and there's no locking). What's even worse, after all the stats
and getdents (it takes a lot of IO for 60GB account with 2M emails),
deliver makes stat on 'new' dir and detects changes after which it
unlinks freshly created maildirsize. So possibly this will never be
finished as every process drops its own work.

The same thing happened when we checked the native dovecot LMTP server
(every connection has its own forked process - each running
independently of the others).

Is it a known unwanted "feature" of maildir quota (and we should use
dict quota instead) or a bug that will be fixed?

cheers,
Dawid Stawiarski

[Timo Sirainen]

we use dovecot 2.2.13, and we've lately started using maildir quota (previously fs quota was used, right now it's "quota = maildir:User quota" changed from "quota = fs:User quota:user:inode_per_mail").
When the change was applied we didn't manually forced generating maildirsize files. So, when the first deliver happens (or the file grows larger then 5KB) it starts quota recalc. However for users with big maildirs, receiving email every couple of seconds, there are tens of delivery processes each doing its own recalc (as they open file for APPEND and there's no locking). What's even worse, after all the stats and getdents (it takes a lot of IO for 60GB account with 2M emails), deliver makes stat on 'new' dir and detects changes after which it unlinks freshly created maildirsize. So possibly this will never be finished as every process drops its own work.
The same thing happened when we checked the native dovecot LMTP server (every connection has its own forked process - each running independently of the others).
Is it a known unwanted "feature" of maildir quota (and we should use dict quota instead) or a bug that will be fixed?

It's a feature that won't be changed. The stat()s are especially bad if you see them - they wouldn't happen if your Maildir filenames contain the ,S=size part. Switch to dict file, which won't be recalculated unless you manually do it.

[Dawid Stawiarski - nazwa.pl]

Post by Timo Sirainen

we use dovecot 2.2.13, and we've lately started using maildir quota (previously fs quota was used, right now it's "quota = maildir:User quota" changed from "quota = fs:User quota:user:inode_per_mail").
When the change was applied we didn't manually forced generating maildirsize files. So, when the first deliver happens (or the file grows larger then 5KB) it starts quota recalc. However for users with big maildirs, receiving email every couple of seconds, there are tens of delivery processes each doing its own recalc (as they open file for APPEND and there's no locking). What's even worse, after all the stats and getdents (it takes a lot of IO for 60GB account with 2M emails), deliver makes stat on 'new' dir and detects changes after which it unlinks freshly created maildirsize. So possibly this will never be finished as every process drops its own work.
The same thing happened when we checked the native dovecot LMTP server (every connection has its own forked process - each running independently of the others).
Is it a known unwanted "feature" of maildir quota (and we should use dict quota instead) or a bug that will be fixed?

It's a feature that won't be changed. The stat()s are especially bad if you see them - they wouldn't happen if your Maildir filenames contain the ,S=size part. Switch to dict file, which won't be recalculated unless you manually do it.

Ok, thank you kindly for clarification.

ps. Our email filenames contain the size part. The stats/getents are for
each folder in Maildir (there are folders with 100-200K emails, so even
listing it takes a long time - especially if there are tens of processes
doing it in parallel)