Discussion:
v2.2.14 released
Timo Sirainen
2014-10-14 16:53:43 UTC
Permalink
http://dovecot.org/releases/2.2/dovecot-2.2.14.tar.gz
http://dovecot.org/releases/2.2/dovecot-2.2.14.tar.gz.sig

Looks like it took a bit longer to make this release after RC1 than I thought. I'm now in San Jose area for a few months and still jumping between apartments and trying to adjust to the new timezone.

Some of the more important fixes since RC1:

- Fixed several race conditions with dovecot.index.cache handling that
may have caused unnecessary "cache is corrupted" errors.
- auth: If auth client listed userdb and disconnected before finishing,
the auth worker process got stuck (and eventually all workers could
get used up and requests would start failing).

Some of the larger changes since v2.2.13:

* lmtp: Delivered-To: header no longer contains <> around the email
address. Other MDAs don't have it either.
* "Out of disk space" errors are now treated as temporary errors
(not the same as "Out of disk quota").
* replication plugin: Use replication only for users who have a
non-empty mail_replica setting.

+ lmtp proxy: Log a line about each mail delivery.
+ Added login_source_ips setting. This can be used to set the source IP
address round-robin from a pool of IPs (in case you run out of TCP
ports).
+ Rawlog settings can use tcp:<host>:<port> as the path.
+ virtual plugin: Don't keep more than virtual_max_open_mailboxes
(default 64) number of backend mailboxes open.
+ SSL/TLS compression can be disabled with ssl_options=no_compression
+ acl: Global ACL file now supports "quotes" around patterns.
+ Added last-login plugin to set user's last-login timestamp on login.
+ LDAP auth: Allow passdb credentials lookup also with auth_bind=yes
- IMAP: MODSEQ was sent in FETCH reply even if CONDSTORE/QRESYNC wasn't
enabled. This broke at least old Outlooks.
- passdb static treated missing password field the same as an empty
password field.
- mdbox: Fixed potential infinite looping when scanning a broken
mdbox file.
- imap-login, pop3-login: Fixed potential crashes when client
disconnected unexpectedly.
- imap proxy: The connection was hanging in some usage patterns. This
mainly affected older Outlooks.
- lmtp proxy: The proxy sometimes delivered empty mails in error
situations or potentially delivered truncated mails.
- fts-lucene: If whitespace_chars was set, we may have ended up
indexing some garbage words, growing the index size unnecessarily.
- -c and -i parameters for dovecot/doveadm commands were ignored if
the config socket was readable.
- quota: Quota recalculation didn't include INBOX in some setups.
- Mail headers were sometimes added to dovecot.index.cache in wrong
order. The main problem this caused was with dsync+imapc incremental
syncing when the second sync thought the local mailbox had changed.
- doveadm backup didn't notice if emails were missing from the middle
of the destination mailbox. Now it deletes and resyncs the mailbox.
The Doctor
2014-10-14 18:53:46 UTC
Permalink
Timo

can you help me with this error?

dovecot-2.2.14$ make
make all-recursive
Making all in .
/bin/bash ./update-version.sh . .
Making all in src
Making all in lib-test
Making all in lib
make all-am
Making all in lib-settings
Making all in lib-auth
Making all in lib-master
Making all in lib-charset
Making all in lib-dns
Making all in lib-dict
Making all in lib-sasl
Making all in lib-ssl-iostream
Making all in lib-http
Making all in lib-fs
Making all in lib-mail
Making all in lib-imap
Making all in lib-imap-storage
Making all in lib-imap-client
Making all in lib-imap-urlauth
Making all in lib-compression
Making all in lib-dovecot
Making all in lib-index
Making all in lib-storage
Making all in list
/bin/bash ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/lib -I../../../src/lib-mail -I../../../src/lib-index -I../../../src/lib-imap -I../../../src/lib-storage -I../../../src/lib-storage/index -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -I/usr/contrib/include -MT mailbox-list-index-iter.lo -MD -MP -MF .deps/mailbox-list-index-iter.Tpo -c -o mailbox-list-index-iter.lo mailbox-list-index-iter.c
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../.. -I../../../src/lib -I../../../src/lib-mail -I../../../src/lib-index -I../../../src/lib-imap -I../../../src/lib-storage -I../../../src/lib-storage/index -std=gnu99 -g -O2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -I/usr/contrib/include -MT mailbox-list-index-iter.lo -MD -MP -MF .deps/mailbox-list-index-iter.Tpo -c mailbox-list-index-iter.c -fPIC -DPIC -o .libs/mailbox-list-index-iter.o
In file included from mailbox-list-index-iter.c:8:
mailbox-list-index.h:89: field `last_refresh_timeval' has incomplete type
mailbox-list-index.h:112: confused by earlier errors, bailing out
*** Error code 1

Stop.
*** Error code 1

Stop.
*** Error code 1

Stop.
*** Error code 1

Stop.
*** Error code 1

Stop.
--
Member - Liberal International This is doctor@@nl2k.ab.ca Ici doctor@@nl2k.ab.ca
God,Queen and country!Never Satan President Republic!Beware AntiChrist rising!
http://www.fullyfollow.me/rootnl2k Look at Psalms 14 and 53 on Atheism
Danger comes when our feelings outweigh reality. -unknown
Timo Sirainen
2014-10-14 19:14:33 UTC
Permalink
Post by The Doctor
mailbox-list-index.h:89: field `last_refresh_timeval' has incomplete type
http://hg.dovecot.org/dovecot-2.2/rev/93a3335ec0d5 should help.
Luigi Rosa
2014-10-16 07:09:50 UTC
Permalink
Post by Timo Sirainen
+ Added last-login plugin to set user's last-login timestamp on login.
According to http://wiki2.dovecot.org/Plugins/LastLogin the plugin uses redis,
but can use other dictionaries

If I have MySQL as userbase backend what field(s) should be created to have
this information recordered in the database?






Ciao,
luigi

- --
/
+--[Luigi Rosa]--
\

Afternoon: that part of the day we spend worrying about
how we wasted the morning.
Alexander 'Leo' Bergolth
2014-10-16 07:45:42 UTC
Permalink
Post by Timo Sirainen
- Fixed several race conditions with dovecot.index.cache handling that
may have caused unnecessary "cache is corrupted" errors.
Rejoyed too soon! :-(
Unfortunately it doesn't fix my (maybe mbox related) index.cache errors reported back in May: http://markmail.org/message/xhxorbutl3encr2d

I know that Timo suggests to switch away from mbox but unfortunately this is rather challenging for me. Switching to maildir results in millions of files and this overwhelmes my backup system. mdbox also does not qualify since AFAIK there is no python or perl library support for manipulating mdbox mailboxes.
:-(

-------------------- 8< --------------------
Oct 15 15:08:34 samba dovecot: imap(USER1): Error: Corrupted index cache file /home/USER1/mail/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 75144
Oct 15 15:08:34 samba dovecot: imap(USER1): Error: copy: i_stream_read() failed: Invalid argument
Oct 15 15:08:34 samba dovecot: imap(USER1): Error: read(/var/mail/USER1) failed: Cached message size smaller than expected (55337 < 55338) (uid=51556, box=Trash)
Oct 15 15:20:02 samba dovecot: imap(USER2): Error: Cached message size smaller than expected (282553 < 282554)
Oct 15 15:20:02 samba dovecot: imap(USER2): Error: Corrupted index cache file /home/USER2/mail/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 12364
Oct 15 15:20:02 samba dovecot: imap(USER2): Error: read(/var/mail/USER2) failed: Cached message size smaller than expected (282553 < 282554) (FETCH BODY[2] for mailbox INBOX UID 12364)
Oct 15 15:50:30 samba dovecot: imap(USER3): Error: Cached message size smaller than expected (58902 < 58903)
Oct 15 15:50:30 samba dovecot: imap(USER3): Error: Corrupted index cache file /home/USER3/mail/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 19810
Oct 15 15:50:30 samba dovecot: imap(USER3): Error: read(/var/mail/USER3) failed: Cached message size smaller than expected (58902 < 58903) (FETCH BODY[] for mailbox INBOX UID 19810)
Oct 15 16:02:56 samba dovecot: imap(USER4): Error: Cached message size smaller than expected (181831 < 181832)
Oct 15 16:02:56 samba dovecot: imap(USER4): Error: Corrupted index cache file /home/USER4/mail/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 2178
Oct 15 16:02:56 samba dovecot: imap(USER4): Error: read(/var/mail/USER4) failed: Cached message size smaller than expected (181831 < 181832) (FETCH BODY[] for mailbox INBOX UID 2178)
Oct 15 16:18:24 samba dovecot: imap(USER1): Error: Cached message size smaller than expected (55339 < 55340)
Oct 15 16:18:24 samba dovecot: imap(USER1): Error: Corrupted index cache file /home/USER1/mail/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 75144
Oct 15 16:18:24 samba dovecot: imap(USER1): Error: copy: i_stream_read() failed: Invalid argument
Oct 15 16:18:24 samba dovecot: imap(USER1): Error: read(/var/mail/USER1) failed: Cached message size smaller than expected (55339 < 55340) (uid=51559, box=Trash)
Oct 15 16:44:15 samba dovecot: imap(USER5): Error: Cached message size smaller than expected (6951359 < 6951360)
Oct 15 16:44:15 samba dovecot: imap(USER5): Error: Corrupted index cache file /home/USER5/mail/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 5023
Oct 15 16:44:15 samba dovecot: imap(USER5): Error: read(/var/mail/USER5) failed: Cached message size smaller than expected (6951359 < 6951360) (FETCH BODY[3] for mailbox INBOX UID 5023)
Oct 15 21:27:45 samba dovecot: imap(USER1): Error: Cached message size smaller than expected (45743 < 45744)
Oct 15 21:27:45 samba dovecot: imap(USER1): Error: Corrupted index cache file /home/USER1/mail/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 75059
Oct 15 21:27:45 samba dovecot: imap(USER1): Error: copy: i_stream_read() failed: Invalid argument
Oct 15 21:27:45 samba dovecot: imap(USER1): Error: read(/var/mail/USER1) failed: Cached message size smaller than expected (45743 < 45744) (uid=51565, box=Trash)
Oct 16 05:10:41 samba dovecot: imap(USER1): Error: Cached message size smaller than expected (98120 < 98121)
Oct 16 05:10:41 samba dovecot: imap(USER1): Error: Corrupted index cache file /home/USER1/mail/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 75158
Oct 16 05:10:41 samba dovecot: imap(USER1): Error: copy: i_stream_read() failed: Invalid argument
Oct 16 05:10:41 samba dovecot: imap(USER1): Error: read(/var/mail/USER1) failed: Cached message size smaller than expected (98120 < 98121) (uid=51566, box=Trash)
Oct 16 07:11:12 samba dovecot: imap(USER6): Error: Cached message size smaller than expected (7991 < 7992)
Oct 16 07:11:12 samba dovecot: imap(USER6): Error: Corrupted index cache file /home/USER6/mail/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 925
Oct 16 07:11:12 samba dovecot: imap(USER6): Error: read(/var/mail/USER6) failed: Invalid argument
Oct 16 08:49:30 samba dovecot: imap(USER1): Error: Cached message size smaller than expected (68834 < 68835)
Oct 16 08:49:30 samba dovecot: imap(USER1): Error: Corrupted index cache file /home/USER1/mail/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 75168
Oct 16 08:49:30 samba dovecot: imap(USER1): Error: copy: i_stream_read() failed: Invalid argument
Oct 16 08:49:30 samba dovecot: imap(USER1): Error: read(/var/mail/USER1) failed: Cached message size smaller than expected (68834 < 68835) (uid=1198, box=A_Leitung)
Oct 16 09:07:56 samba dovecot: imap(USER7): Error: Cached message size smaller than expected (367995 < 367997)
Oct 16 09:07:56 samba dovecot: imap(USER7): Error: Corrupted index cache file /home/USER7/mail/.imap/INBOX/dovecot.index.cache: Broken physical size for mail UID 25775
Oct 16 09:07:56 samba dovecot: imap(USER7): Error: read(/var/mail/USER7) failed: Cached message size smaller than expected (367995 < 367997) (FETCH BODY[] for mailbox INBOX UID 25775)
-------------------- 8< --------------------

Cheers,
--leo
--
e-mail ::: Leo.Bergolth (at) wu.ac.at
fax ::: +43-1-31336-906050
location ::: IT-Services | Vienna University of Economics | Austria
Gene Heskett
2014-10-16 16:27:15 UTC
Permalink
On Thursday 16 October 2014 03:45:42 Alexander 'Leo' Bergolth did opine
Post by Alexander 'Leo' Bergolth
Post by Timo Sirainen
- Fixed several race conditions with dovecot.index.cache handling that
may have caused unnecessary "cache is corrupted" errors.
Rejoyed too soon! :-(
Unfortunately it doesn't fix my (maybe mbox related) index.cache errors
reported back in May: http://markmail.org/message/xhxorbutl3encr2d
I know that Timo suggests to switch away from mbox but unfortunately
this is rather challenging for me. Switching to maildir results in
millions of files and this overwhelmes my backup system.
I reset kmail 1.13.5 to use maildir (mdir) format message storage several
years ago.

My email corpus is north of 20Gb, and amanda has no problems backing it
up. Its been doing it well enough I can do a bare metal recovery and have.

Sounds like you need a better backup proggy. Amanda, FWIW, is a wrapper,
most commonly using the current version of tar, can also use dump, but I
consider tars file at a time granularity a huge advantage since it can
also recover something you accidentally deleted without overwriting
everything.
Post by Alexander 'Leo' Bergolth
mdbox also
does not qualify since AFAIK there is no python or perl library
support for manipulating mdbox mailboxes.
:-(
-------------------- 8< --------------------
Oct 15 15:08:34 samba dovecot: imap(USER1): Error: Corrupted index
cache file /home/USER1/mail/.imap/INBOX/dovecot.index.cache: Broken
imap(USER1): Error: copy: i_stream_read() failed: Invalid argument Oct
15 15:08:34 samba dovecot: imap(USER1): Error: read(/var/mail/USER1)
failed: Cached message size smaller than expected (55337 < 55338)
Error: Cached message size smaller than expected (282553 < 282554) Oct
15 15:20:02 samba dovecot: imap(USER2): Error: Corrupted index cache
file /home/USER2/mail/.imap/INBOX/dovecot.index.cache: Broken physical
Error: read(/var/mail/USER2) failed: Cached message size smaller than
expected (282553 < 282554) (FETCH BODY[2] for mailbox INBOX UID 12364)
Oct 15 15:50:30 samba dovecot: imap(USER3): Error: Cached message size
imap(USER3): Error: Corrupted index cache file
/home/USER3/mail/.imap/INBOX/dovecot.index.cache: Broken physical size
read(/var/mail/USER3) failed: Cached message size smaller than
expected (58902 < 58903) (FETCH BODY[] for mailbox INBOX UID 19810)
Oct 15 16:02:56 samba dovecot: imap(USER4): Error: Cached message size
imap(USER4): Error: Corrupted index cache file
/home/USER4/mail/.imap/INBOX/dovecot.index.cache: Broken physical size
read(/var/mail/USER4) failed: Cached message size smaller than
expected (181831 < 181832) (FETCH BODY[] for mailbox INBOX UID 2178)
Oct 15 16:18:24 samba dovecot: imap(USER1): Error: Cached message size
imap(USER1): Error: Corrupted index cache file
/home/USER1/mail/.imap/INBOX/dovecot.index.cache: Broken physical size
copy: i_stream_read() failed: Invalid argument Oct 15 16:18:24 samba
dovecot: imap(USER1): Error: read(/var/mail/USER1) failed: Cached
message size smaller than expected (55339 < 55340) (uid=51559,
box=Trash) Oct 15 16:44:15 samba dovecot: imap(USER5): Error: Cached
message size smaller than expected (6951359 < 6951360) Oct 15 16:44:15
samba dovecot: imap(USER5): Error: Corrupted index cache file
/home/USER5/mail/.imap/INBOX/dovecot.index.cache: Broken physical size
read(/var/mail/USER5) failed: Cached message size smaller than
expected (6951359 < 6951360) (FETCH BODY[3] for mailbox INBOX UID
5023) Oct 15 21:27:45 samba dovecot: imap(USER1): Error: Cached
message size smaller than expected (45743 < 45744) Oct 15 21:27:45
samba dovecot: imap(USER1): Error: Corrupted index cache file
/home/USER1/mail/.imap/INBOX/dovecot.index.cache: Broken physical size
copy: i_stream_read() failed: Invalid argument Oct 15 21:27:45 samba
dovecot: imap(USER1): Error: read(/var/mail/USER1) failed: Cached
message size smaller than expected (45743 < 45744) (uid=51565,
box=Trash) Oct 16 05:10:41 samba dovecot: imap(USER1): Error: Cached
message size smaller than expected (98120 < 98121) Oct 16 05:10:41
samba dovecot: imap(USER1): Error: Corrupted index cache file
/home/USER1/mail/.imap/INBOX/dovecot.index.cache: Broken physical size
copy: i_stream_read() failed: Invalid argument Oct 16 05:10:41 samba
dovecot: imap(USER1): Error: read(/var/mail/USER1) failed: Cached
message size smaller than expected (98120 < 98121) (uid=51566,
box=Trash) Oct 16 07:11:12 samba dovecot: imap(USER6): Error: Cached
message size smaller than expected (7991 < 7992) Oct 16 07:11:12 samba
dovecot: imap(USER6): Error: Corrupted index cache file
/home/USER6/mail/.imap/INBOX/dovecot.index.cache: Broken physical size
read(/var/mail/USER6) failed: Invalid argument Oct 16 08:49:30 samba
dovecot: imap(USER1): Error: Cached message size smaller than expected
Corrupted index cache file
/home/USER1/mail/.imap/INBOX/dovecot.index.cache: Broken physical size
copy: i_stream_read() failed: Invalid argument Oct 16 08:49:30 samba
dovecot: imap(USER1): Error: read(/var/mail/USER1) failed: Cached
message size smaller than expected (68834 < 68835) (uid=1198,
Cached message size smaller than expected (367995 < 367997) Oct 16
09:07:56 samba dovecot: imap(USER7): Error: Corrupted index cache file
/home/USER7/mail/.imap/INBOX/dovecot.index.cache: Broken physical size
read(/var/mail/USER7) failed: Cached message size smaller than
expected (367995 < 367997) (FETCH BODY[] for mailbox INBOX UID 25775)
-------------------- 8< --------------------
Cheers,
--leo
Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>
US V Castleman, SCOTUS, Mar 2014 is grounds for Impeaching SCOTUS
Timo Sirainen
2014-10-16 19:11:15 UTC
Permalink
Post by Alexander 'Leo' Bergolth
Post by Timo Sirainen
- Fixed several race conditions with dovecot.index.cache handling that
may have caused unnecessary "cache is corrupted" errors.
Rejoyed too soon! :-(
Unfortunately it doesn't fix my (maybe mbox related) index.cache errors reported back in May: http://markmail.org/message/xhxorbutl3encr2d
These aren't actually cache related problems. These are mbox bugs.
Post by Alexander 'Leo' Bergolth
I know that Timo suggests to switch away from mbox but unfortunately this is rather challenging for me. Switching to maildir results in millions of files and this overwhelmes my backup system. mdbox also does not qualify since AFAIK there is no python or perl library support for manipulating mdbox mailboxes.
:-(
You shouldn't need to manipulate mdboxes directly. You can manipulate them via doveadm binary or doveadm protocol. That way you could also later on switch between any mailbox formats without changing your manipulation tools.
Andy Dills
2014-10-16 12:28:24 UTC
Permalink
2.2.14 is working great in production for us. No more strange errors
with indexes[1] (which was a constant thing in 2.2.13), and I'm happy
that the clustered LDA issues are resolved.

Well done Timo, and congratulations on your relocation to the US!

Andy

[1] Panic: file mail-index-transaction-export.c: line 203
(log_append_ext_hdr_update): assertion failed: (u32.offset + u32.size <=
ext_hdr_size)
---

-----------------------------------------------------
ANDY DILLS - XECUNET, LLC

5744-R Industry Lane
Frederick MD 21704
www.xecu.net [1]
P: 301-682-9972
P: 1-877-XECUNET
F: 240-215-0351

Twitter [2]

Facebook [3]
-----------------------------------------------------

Links:
------
[1] http://www.xecu.net/
[2] https://twitter.com/Xecunet
[3] http://www.facebook.com/xecunet
Loading...