[Dovecot] mdbox-files over 2 MB

Discussion:

Hardy Flor

2014-03-17 06:38:55 UTC

Hello,

I received an e-mail, this includes 9 forwarding, each with a file.
These files are not removed from the e-mail and stored in the
attachments-directory, although they are larger than 16 KB. For all
files the "Content-Transfer-Encoding: base64" is set.
The email is stored in completely in "m.00000034"

------------------------------------- doveconf:

# 2.2.12: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.4 ext4
mail_attachment_dir = /var/mail/attachments
mail_attachment_hash = %{md5}
mail_attachment_min_size = 16 k
mail_location = mdbox:/var/mail/user/%n
mdbox_rotate_interval = 1 weeks
mdbox_rotate_size = 2 M

------------------------------------- directory:

root at xxx:/var/mail/user/xxx/storage# ls -al
...
-rw------- 1 vmail mail 14433784 M?r 16 13:51 m.00000034
...

Hardy Flor

2014-05-05 14:10:38 UTC

Permalink

Is there really no one with this problem?

Reindl Harald

2014-05-05 14:16:19 UTC

Permalink

Post by Hardy Flor
Is there really no one with this problem?

next time quote the problem instead demand
others to seek for you in the archives

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 246 bytes
Desc: OpenPGP digital signature
URL: <http://dovecot.org/pipermail/dovecot/attachments/20140505/16020204/attachment.sig>

Patrick Domack

2014-05-05 19:23:05 UTC

Permalink

Post by Reindl Harald

Post by Hardy Flor
Is there really no one with this problem?

next time quote the problem instead demand
others to seek for you in the archives

Even searching for him in the archives, I have no idea what he thinks
is a problem.

I know I have no issues using mdbox, with 2mb or 50mb rotate sizes.

Stan Hoeppner

2014-05-06 01:05:07 UTC

Permalink

Post by Reindl Harald

Post by Hardy Flor
Is there really no one with this problem?

next time quote the problem instead demand
others to seek for you in the archives

Even searching for him in the archives, I have no idea what he thinks is
a problem.
I know I have no issues using mdbox, with 2mb or 50mb rotate sizes.

He's wondering why many of his mdbox files are much smaller than 2 MB.

Hello,
there are copies with different size in 3 mailboxes of the user
sequentially about 3800 emails.
why not something 2MB files?
After the big file "m.00000034" with 14MB follow very many small ...
# 2.2.12: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.4 ext4
mail_attachment_dir = /var/mail/attachments
mail_attachment_hash = %{md5}
mail_attachment_min_size = 16 k
mail_location = mdbox:/var/mail/user/%n
mdbox_rotate_interval = 1 weeks
mdbox_rotate_size = 2 M
root at xxx:/var/mail/user/xxx/storage# ls -al
insgesamt 76532
drwx--S--- 2 vmail mail 4096 M?r 16 13:52 .
drwx--S--- 6 vmail mail 4096 M?r 16 12:10 ..
-rw------- 1 vmail mail 74360 M?r 16 12:39 dovecot.map.index
-rw------- 1 vmail mail 25204 M?r 16 13:52 dovecot.map.index.log
-rw------- 1 vmail mail 63544 M?r 16 12:39 dovecot.map.index.log.2
-rw------- 1 vmail mail 2092921 M?r 16 12:15 m.00000001
-rw------- 1 vmail mail 2086746 M?r 16 12:15 m.00000002
-rw------- 1 vmail mail 2084182 M?r 16 12:15 m.00000003
-rw------- 1 vmail mail 2096826 M?r 16 12:15 m.00000004
-rw------- 1 vmail mail 1882666 M?r 16 12:15 m.00000005
-rw------- 1 vmail mail 1903965 M?r 16 12:15 m.00000006
-rw------- 1 vmail mail 2091169 M?r 16 12:15 m.00000007
-rw------- 1 vmail mail 2086396 M?r 16 12:15 m.00000008
-rw------- 1 vmail mail 507205 M?r 16 12:16 m.00000009
-rw------- 1 vmail mail 2031456 M?r 16 12:17 m.00000010
-rw------- 1 vmail mail 2095697 M?r 16 12:17 m.00000011
-rw------- 1 vmail mail 1689071 M?r 16 12:17 m.00000012
-rw------- 1 vmail mail 2092124 M?r 16 12:17 m.00000013
-rw------- 1 vmail mail 1950602 M?r 16 12:17 m.00000014
-rw------- 1 vmail mail 2092215 M?r 16 12:17 m.00000015
-rw------- 1 vmail mail 2087463 M?r 16 12:17 m.00000016
-rw------- 1 vmail mail 2079795 M?r 16 12:17 m.00000017
-rw------- 1 vmail mail 2014121 M?r 16 12:17 m.00000018
-rw------- 1 vmail mail 2081893 M?r 16 12:17 m.00000019
-rw------- 1 vmail mail 2092088 M?r 16 12:17 m.00000020
-rw------- 1 vmail mail 2090508 M?r 16 12:17 m.00000021
-rw------- 1 vmail mail 1929296 M?r 16 12:17 m.00000022
-rw------- 1 vmail mail 2067685 M?r 16 12:17 m.00000023
-rw------- 1 vmail mail 1745743 M?r 16 12:20 m.00000024
-rw------- 1 vmail mail 866452 M?r 16 12:20 m.00000025
-rw------- 1 vmail mail 296379 M?r 16 12:20 m.00000026
-rw------- 1 vmail mail 433541 M?r 16 12:39 m.00000027
-rw------- 1 vmail mail 92526 M?r 16 12:39 m.00000028
-rw------- 1 vmail mail 19094 M?r 16 12:39 m.00000029
-rw------- 1 vmail mail 106837 M?r 16 12:39 m.00000030
-rw------- 1 vmail mail 162012 M?r 16 12:39 m.00000031
-rw------- 1 vmail mail 412080 M?r 16 13:51 m.00000032
-rw------- 1 vmail mail 1715868 M?r 16 13:51 m.00000033
-rw------- 1 vmail mail 14433784 M?r 16 13:51 m.00000034
-rw------- 1 vmail mail 3280 M?r 16 13:51 m.00000035
-rw------- 1 vmail mail 6308 M?r 16 13:51 m.00000036
-rw------- 1 vmail mail 534274 M?r 16 13:51 m.00000037
-rw------- 1 vmail mail 378719 M?r 16 13:51 m.00000038
-rw------- 1 vmail mail 458528 M?r 16 13:51 m.00000039
-rw------- 1 vmail mail 192504 M?r 16 13:51 m.00000040
-rw------- 1 vmail mail 1277766 M?r 16 13:51 m.00000041
-rw------- 1 vmail mail 33417 M?r 16 13:51 m.00000042
-rw------- 1 vmail mail 142866 M?r 16 13:51 m.00000043
-rw------- 1 vmail mail 113529 M?r 16 13:51 m.00000044
-rw------- 1 vmail mail 55888 M?r 16 13:51 m.00000045
-rw------- 1 vmail mail 673504 M?r 16 13:51 m.00000046
-rw------- 1 vmail mail 430738 M?r 16 13:51 m.00000047
-rw------- 1 vmail mail 218494 M?r 16 13:51 m.00000048
-rw------- 1 vmail mail 96165 M?r 16 13:51 m.00000049
-rw------- 1 vmail mail 8370 M?r 16 13:51 m.00000050
-rw------- 1 vmail mail 29251 M?r 16 13:51 m.00000051
-rw------- 1 vmail mail 394680 M?r 16 13:51 m.00000052
-rw------- 1 vmail mail 674892 M?r 16 13:51 m.00000053
-rw------- 1 vmail mail 1076919 M?r 16 13:51 m.00000054
-rw------- 1 vmail mail 258806 M?r 16 13:51 m.00000055
-rw------- 1 vmail mail 60069 M?r 16 13:51 m.00000056
-rw------- 1 vmail mail 297891 M?r 16 13:51 m.00000057
-rw------- 1 vmail mail 92772 M?r 16 13:51 m.00000058
-rw------- 1 vmail mail 129272 M?r 16 13:51 m.00000059
-rw------- 1 vmail mail 56073 M?r 16 13:51 m.00000060
-rw------- 1 vmail mail 528260 M?r 16 13:51 m.00000061
-rw------- 1 vmail mail 29468 M?r 16 13:51 m.00000062
-rw------- 1 vmail mail 3250 M?r 16 13:51 m.00000063
-rw------- 1 vmail mail 96445 M?r 16 13:51 m.00000064
-rw------- 1 vmail mail 337663 M?r 16 13:51 m.00000065
-rw------- 1 vmail mail 187505 M?r 16 13:51 m.00000066
-rw------- 1 vmail mail 98839 M?r 16 13:51 m.00000067
-rw------- 1 vmail mail 5294 M?r 16 13:51 m.00000068
-rw------- 1 vmail mail 28276 M?r 16 13:51 m.00000069
-rw------- 1 vmail mail 67950 M?r 16 13:52 m.00000070
-rw------- 1 vmail mail 704457 M?r 16 13:52 m.00000071
-rw------- 1 vmail mail 1410205 M?r 16 13:52 m.00000072
-rw------- 1 vmail mail 736330 M?r 16 13:52 m.00000073
-rw------- 1 vmail mail 616992 M?r 16 13:52 m.00000074

Jesus Cea

2014-10-01 13:43:18 UTC

Permalink

I keep the quoted message complete because the original thread was
posted six months ago. I am top-posting, no comments inside the quoted text.

I plan to write a blog about mdbox oddities but in the meantime I can
tell you that if you store attachments "detached" in your mdbox storage
the issue is this:

1. You have a 200Kbytes MB m.* file you are currently targeting.

2. You have a new email to store. Lets say it has a 3MB attachment.

3. You have a 2MB mdbox max file size .

4. Dovecot notices that 0.2+3 > 2, so it will create a new m.* file.

5. When storing the message, it will store the attachment outside of the
mbox, so the new m.* file will be, lets say, 5Kbytes only.

6. Now you have 2 m.* files: one of 200 Kbytes and other of 5Kbytes.

7. Dovecot tries to reuse undersized m.* files, but it is quite
forgiving. I have some private patches to increase its agressiveness.

If you don't detach attachment, you probably have the same issue: when a
new message makes a m.* file to oversize, dovecot creates a new m.* file
and leave the old one alone, so it will be undersized.

Post by Stan Hoeppner

Post by Reindl Harald

Post by Hardy Flor
Is there really no one with this problem?

next time quote the problem instead demand
others to seek for you in the archives

Even searching for him in the archives, I have no idea what he thinks is
a problem.
I know I have no issues using mdbox, with 2mb or 50mb rotate sizes.

He's wondering why many of his mdbox files are much smaller than 2 MB.

--
Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/
jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
Twitter: @jcea _/_/ _/_/ _/_/_/_/_/
jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://dovecot.org/pipermail/dovecot/attachments/20141001/06b40af2/attachment.sig>

Hardy Flor

2014-05-06 10:17:45 UTC

Permalink

Sorry, I thought the tree display of threads is sufficient. I did not
want to quote much.

Post by Reindl Harald

Post by Hardy Flor
Is there really no one with this problem?

next time quote the problem instead demand
others to seek for you in the archives

Larry Stone

2014-05-06 11:29:18 UTC

Permalink

Sorry, I thought the tree display of threads is sufficient. I did not want to quote much.

Believe it or not, not all of us save every message or have the same MUA capabilities. Don?t assume how you read is the same way everyone does. Sometimes, what starts as something uninteresting and deleted immediately turns interesting. Therefore, quoting the relevant text from previous message is needed.
--
Larry Stone
lstone19 at stonejongleux.com
http://www.stonejongleux.com/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4160 bytes
Desc: not available
URL: <http://dovecot.org/pipermail/dovecot/attachments/20140506/866e84e5/attachment-0001.p7s>

Hardy Flor

2014-07-12 06:46:02 UTC

Permalink

Post by Hardy Flor
Hello,
I received an e-mail, this includes 9 forwarding, each with a file.
These files are not removed from the e-mail and stored in the
attachments-directory, although they are larger than 16 KB. For all
files the "Content-Transfer-Encoding: base64" is set.
The email is stored in completely in "m.00000034"
# 2.2.12: /etc/dovecot/dovecot.conf
# OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.4 ext4
mail_attachment_dir = /var/mail/attachments
mail_attachment_hash = %{md5}
mail_attachment_min_size = 16 k
mail_location = mdbox:/var/mail/user/%n
mdbox_rotate_interval = 1 weeks
mdbox_rotate_size = 2 M
root at xxx:/var/mail/user/xxx/storage# ls -al
...
-rw------- 1 vmail mail 14433784 M?r 16 13:51 m.00000034
...

Has this really not a solution? I've found some e-mails where the
attachments are not stored separately.