Corrupted index cache after migrating from maildir to mdbox

[Tim]

Hi guys,

I'm running dovecot/postfix on my Debian Jessie maschine:

# dovecot --version
2.2.13
doveconf -n output: http://p.bastelfreak.de/HsVwS/

today I migrated from maildir(located at /var/vmail) to mdbox with these
commands:
rsync -av -f"+ */" -f"- *" /var/vmail/ /home/vmail/
rm /home/vmail/*/*/Maildir -rf
for i in $(doveadm user \*); do u=$(echo $i | cut -d@ -f1); d=$(echo $i
|cut -d@ -f2); dsync -v -u $i mirror mdbox:/home/vmail/$d/$u/mdbox; done

Then I changed the mail_location to mdbox:/home/vmail/%d/%n/mdbox and
did the following commands:

service dovecot restart
for i in $(doveadm user \*); do u=$(echo $i | cut -d@ -f1); d=$(echo $i
|cut -d@ -f2); dsync -v -u $i mirror maildir:/var/vmail/$d/$u/Mail; done
doveadm index -A all INBOX
for i in $(doveadm user \*); do doveadm -Dv fts rescan -u $i;done

unfortunately I'm getting the following two kinds of error messages for
all mail boxes in my mail.log:

Error: Corrupted index cache file
/home/vmail/%domain/%user/mdbox/mailboxes/%maildir/dbox-Mails/dovecot.index.cache:
Broken virtual size for mail UID 2564

and:

Error: read(/home/vmail/%domain/%user/mdbox/storage/m.108
): FETCH BODY[] for mailbox %maildir UID 2564 got too little data: 2417
vs 6319

please let me know if you have any suggestions for fixing this issue or
if you need more information.

kind regards, Tim

[Tim]

Hi guys,

does nobody has any thoughts on this error?

Post by Tim
Hi guys,
# dovecot --version
2.2.13
doveconf -n output: http://p.bastelfreak.de/HsVwS/
today I migrated from maildir(located at /var/vmail) to mdbox with these
rsync -av -f"+ */" -f"- *" /var/vmail/ /home/vmail/
rm /home/vmail/*/*/Maildir -rf
Then I changed the mail_location to mdbox:/home/vmail/%d/%n/mdbox and
service dovecot restart
doveadm index -A all INBOX
for i in $(doveadm user \*); do doveadm -Dv fts rescan -u $i;done
unfortunately I'm getting the following two kinds of error messages for
Error: Corrupted index cache file
Broken virtual size for mail UID 2564
Error: read(/home/vmail/%domain/%user/mdbox/storage/m.108
): FETCH BODY[] for mailbox %maildir UID 2564 got too little data: 2417
vs 6319
please let me know if you have any suggestions for fixing this issue or
if you need more information.
kind regards, Tim

[Toni Mattila]

Hi,

You have broken maildir files with wrong S= in them. Usually produced by
maildrop or courier imap's tools. You can fix them with
http://www.dovecot.org/tools/maildir-size-fix.pl

If you compare ls -la and S= in the files you will see the difference.

Cheers,
Toni

Post by Tim
Hi guys,
does nobody has any thoughts on this error?

Post by Tim
Hi guys,
# dovecot --version
2.2.13
doveconf -n output: http://p.bastelfreak.de/HsVwS/
today I migrated from maildir(located at /var/vmail) to mdbox with these
rsync -av -f"+ */" -f"- *" /var/vmail/ /home/vmail/
rm /home/vmail/*/*/Maildir -rf
Then I changed the mail_location to mdbox:/home/vmail/%d/%n/mdbox and
service dovecot restart
doveadm index -A all INBOX
for i in $(doveadm user \*); do doveadm -Dv fts rescan -u $i;done
unfortunately I'm getting the following two kinds of error messages for
Error: Corrupted index cache file
Broken virtual size for mail UID 2564
Error: read(/home/vmail/%domain/%user/mdbox/storage/m.108
): FETCH BODY[] for mailbox %maildir UID 2564 got too little data: 2417
vs 6319
please let me know if you have any suggestions for fixing this issue or
if you need more information.
kind regards, Tim

[Tim]

Post by Toni Mattila
Hi,
You have broken maildir files with wrong S= in them. Usually produced
by maildrop or courier imap's tools. You can fix them with
http://www.dovecot.org/tools/maildir-size-fix.pl

I migrated from maildir to mdbox. Seems like the tool only fixes maildir
format? possible to fix my mdbox? Also I never noticed any error
messages while using maildir.

Post by Toni Mattila
If you compare ls -la and S= in the files you will see the difference.
Cheers,
Toni

Post by Tim
Hi guys,
does nobody has any thoughts on this error?

Post by Tim
Hi guys,
# dovecot --version
2.2.13
doveconf -n output: http://p.bastelfreak.de/HsVwS/
today I migrated from maildir(located at /var/vmail) to mdbox with these
rsync -av -f"+ */" -f"- *" /var/vmail/ /home/vmail/
rm /home/vmail/*/*/Maildir -rf
Then I changed the mail_location to mdbox:/home/vmail/%d/%n/mdbox and
service dovecot restart
doveadm index -A all INBOX
for i in $(doveadm user \*); do doveadm -Dv fts rescan -u $i;done
unfortunately I'm getting the following two kinds of error messages for
Error: Corrupted index cache file
Broken virtual size for mail UID 2564
Error: read(/home/vmail/%domain/%user/mdbox/storage/m.108
): FETCH BODY[] for mailbox %maildir UID 2564 got too little data: 2417
vs 6319
please let me know if you have any suggestions for fixing this issue or
if you need more information.
kind regards, Tim

[Tim]

Post by Tim

Post by Toni Mattila
Hi,
You have broken maildir files with wrong S= in them. Usually produced
by maildrop or courier imap's tools. You can fix them with
http://www.dovecot.org/tools/maildir-size-fix.pl

I migrated from maildir to mdbox. Seems like the tool only fixes maildir
format? possible to fix my mdbox? Also I never noticed any error
messages while using maildir.

Hey guys, any further thoughts here?

Post by Tim

Post by Toni Mattila
If you compare ls -la and S= in the files you will see the difference.
Cheers,
Toni

Post by Tim
Hi guys,
does nobody has any thoughts on this error?

Post by Tim
Hi guys,
# dovecot --version
2.2.13
doveconf -n output: http://p.bastelfreak.de/HsVwS/
today I migrated from maildir(located at /var/vmail) to mdbox with these
rsync -av -f"+ */" -f"- *" /var/vmail/ /home/vmail/
rm /home/vmail/*/*/Maildir -rf
Then I changed the mail_location to mdbox:/home/vmail/%d/%n/mdbox and
service dovecot restart
doveadm index -A all INBOX
for i in $(doveadm user \*); do doveadm -Dv fts rescan -u $i;done
unfortunately I'm getting the following two kinds of error messages for
Error: Corrupted index cache file
Broken virtual size for mail UID 2564
Error: read(/home/vmail/%domain/%user/mdbox/storage/m.108
): FETCH BODY[] for mailbox %maildir UID 2564 got too little data: 2417
vs 6319
please let me know if you have any suggestions for fixing this issue or
if you need more information.
kind regards, Tim

[Daniel Parthey]

I think you could try to simply delete dovecot.index.cache file and have dovecot automatically recreate it on the next delivery or login. And then see if the error messages go away for this mailbox.

With mdbox the dovecot.index file must not be deleted however, you would otherwise lose "seen" flags.

Anyway, better have a backup prepared, before deleting any files ;)

Regards
Daniel

[Tim]

Hi,

I mv'ed all dovecot.index.cache files for one mailbox away but I still
get error messages:

Oct 12 22:47:12 mail01 dovecot: imap(foobar at example.com): Error:
read(/home/vmail/example.com/foobar/mdbox/storage/m.1):
FETCH BODY[] for mailbox INBOX UID 3 got too little data: 738 vs 1543
Oct 12
22:47:12 mail01 dovecot: imap(foobar at example.com): Error: Corrupted
index cache file
/home/vmail/example.com/foobar/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.cache:
Broken virtual size for mail UID 3

So dovecot recreates the cache file but they are instant broken again?
Any thoughts here?

Post by Daniel Parthey
I think you could try to simply delete dovecot.index.cache file and have dovecot automatically recreate it on the next delivery or login. And then see if the error messages go away for this mailbox.
With mdbox the dovecot.index file must not be deleted however, you would otherwise lose "seen" flags.
Anyway, better have a backup prepared, before deleting any files ;)
Regards
Daniel

[Daniel Parthey]

Maybe the index is corrupted too and the index.cache is a cached version of the corrupted index?

Are you using NFS with mdbox and accessing one nfs mailbox directory concurrently from several dovecot hosts?

[Tim]

all mails are stored on an xfs filesystem and only the local dovecot
accesses it. Possible to fix the index.cache files? I havn't found
anything suitable in the docs. The doveadm-index commands are only for
the fts plugin I think.

Post by Daniel Parthey
Maybe the index is corrupted too and the index.cache is a cached version of the corrupted index?
Are you using NFS with mdbox and accessing one nfs mailbox directory concurrently from several dovecot hosts?

[Tim Meusel]

Post by Daniel Parthey
Maybe the index is corrupted too and the index.cache is a cached version of the corrupted index?
Are you using NFS with mdbox and accessing one nfs mailbox directory concurrently from several dovecot hosts?

all mails are stored on an xfs filesystem and only the local dovecot
accesses it. Possible to fix the index.cache files? I havn't found
anything suitable in the docs. The doveadm-index commands are only for
the fts plugin I think.

[Timo Sirainen]

Post by Tim
unfortunately I'm getting the following two kinds of error messages for
Error: Corrupted index cache file
Broken virtual size for mail UID 2564
Error: read(/home/vmail/%domain/%user/mdbox/storage/m.108
): FETCH BODY[] for mailbox %maildir UID 2564 got too little data: 2417
vs 6319

If that happens to all the mails, something went wrong. Did you maybe use zlib plugin while writing the messages (compressed), but now you're trying to read the mails without zlib plugin?

The error messages may be somewhat confusing. The problem isn't dovecot.index.cache file, but that the message sizes in the m.* files don't match the contents. There's no easy way to fix those, because they should just about never be wrong.