index problem with only 1 folder of 1 box

Discussion:

Guillaume

2014-10-07 10:23:52 UTC

Hello,

First of all, sorry for my poor english level.

I pass to solr indexing.

All is OK, except for the inbox of one of the mailboxes. For this inbox
(other folders of the same mailbox have no problems), when i do text
search, there is always no response.
Others mailboxes have no problems.

If i do a : "doveadm -Dv fts rescan -u mybox at domain.tld", i can do one
and only one search with the correct response. It index the folder, take
a few seconds and give me the good answer.

If i redo the same search, it's extremly quick with no answers. It's
like a cache problem but i don't know where and why...
Logs are not very verbose...

Here is an example of command line search :
----------------------------------------

$ nc 127.0.0.1 143
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Bienvenue sur le serveur IMAP de
Domain.tld
10 LOGIN "mybox at domain.tld" "MyBigSecret"
10 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS
THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN
NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH
ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY
MOVE SEARCH=FUZZY QUOTA] Logged in
20 SELECT INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-2 unknown-9
unknown-0 unknown-10 unknown-1 unknown-4 $MDNSent unknown-3 unknown-8
$Forwarded NonJunk)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-2
unknown-9 unknown-0 unknown-10 unknown-1 unknown-4 $MDNSent unknown-3
unknown-8 $Forwarded NonJunk \*)] Flags permitted.
* 8437 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1412669180] UIDs valid
* OK [UIDNEXT 8439] Predicted next UID
* OK [HIGHESTMODSEQ 20] Highest
20 OK [READ-WRITE] Select completed (0.000 secs).
30 SEARCH text "coucou"
* OK Indexed 14% of the mailbox, ETA 0:57
* OK Indexed 29% of the mailbox, ETA 0:47
* OK Indexed 72% of the mailbox, ETA 0:11
* OK Mailbox indexing finished
* SEARCH 35 108 153 175 176 182 348 357 360 431 472 479 520 574 575 576
577 578 586 949 984 1025 1032 1033 1034 1087 1099 1129 1284 1313 1314
1315 1366 1372 1465 1538 1711 1726 1758 1759 1969 1981 1983 1984 2036
2077 2104 2116 2162 2181 2193 2437 2504 2587 2663 2808 2809 2854 2861
2862 3076 3087 3088 3137 3188 3189 3190 3191 3192 3193 3196 3197 3250
3251 3252 3327 3328 3329 3330 3331 3332 3379 3464 3481 3494 3521 3528
3582 3589 3593 3596 3651 3667 3672 3695 3787 3789 3939 3940 3941 3942
3943 3944 3945 3946 4019 4020 4034 4043 4134 4146 4175 4241 4346 4379
4422 4490 4491 4537 4540 4542 4543 4544 4550 4627 4727 4742 4767 4790
4817 4828 4831 4839 4862 4888 4899 4901 4906 4911 4916 4957 4999 5003
5004 5008 5009 5010 5011 5012 5013 5039 5062 5081 5092 5119 5138 5145
5150 5152 5185 5186 5187 5216 5246 5247 5282 5283 5314 5316 5320 5322
5325 5337 5353 5360 5363 5385 5432 5447 5472 5485 5496 5498 5500 5528
5551 5564 5567 5595 5598 5642 5794 5841 5896 5933 5960 5961 5994 6001
6005 6016 6017 6051 6075 6125 6152 6163 6215 6238 6239 6240 6241 6287
6311 6342 6350 6353 6357 6369 6431 6472 6689 6928 6945 6951 7026 7030
7045 7046 7115 7210 7212 7214 7230 7256 7295 7298 7340 7489 7490 7492
7654 7655 7658 7660 7676 7767 7768 7800 7801 7913 7937 7956 8075 8085
8102 8111 8144 8266 8329 8333 8394 8395 8431
30 OK Search completed (36.029 secs).
40 SEARCH text "coucou"
* SEARCH
40 OK Search completed (0.002 secs).
50 SEARCH text "blabla"
* SEARCH
50 OK Search completed (0.002 secs).
60 LOGOUT
* BYE Logging out
60 OK Logout completed.

----------------------------------------

To close this question, here is dovecot -n:
----------------------------------------
# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.13.0-36-generic x86_64 Ubuntu 14.04.1 LTS ext4
auth_mechanisms = plain login
debug_log_path = /var/log/dovecot-debug.log
disable_plaintext_auth = no
first_valid_uid = 5000
hostname = mailserver.domain.tld
info_log_path = /var/log/dovecot-info.log
last_valid_uid = 5000
lda_mailbox_autocreate = yes
lda_original_recipient_header = X-Original-To
listen = *
log_path = /var/log/dovecot-err.log
login_greeting = Bienvenue sur le serveur IMAP de Domain.tld
mail_debug = yes
mail_gid = 5000
mail_location = maildir:/data/mails/%d/%n
mail_plugins = " quota fts fts_solr"
mail_uid = 5000
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
passdb {
driver = pam
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
fts = solr
fts_autoindex = yes
fts_solr = break-imap-search url=http://localhost:8080/solr/
quota = maildir:User quota
sieve = /home/vmail/managesieve/.%n@%d.sieve
sieve_dir = /home/vmail/managesieve/%d/%n
sieve_global_dir = /home/vmail/sieve/
}
postmaster_address = postmaster at domain.tld
protocols = " imap sieve pop3"
service auth {
unix_listener /var/spool/postfix/private/auth {
group = root
mode = 0660
user = postfix
}
unix_listener auth-userdb {
group = vmail
mode = 0660
user = vmail
}
}
service dict {
unix_listener dict {
group = vmail
mode = 0600
user = vmail
}
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
inet_listener sieve_deprecated {
port = 2000
}
}
ssl_ca = </data/certificats/web/domain.tld/ca.pem
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
userdb {
driver = passwd
}
userdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
verbose_proctitle = yes
protocol lmtp {
mail_plugins = " sieve"
}
protocol lda {
mail_plugins = " quota fts fts_solr sieve"
}
protocol imap {
mail_plugins = " quota fts fts_solr imap_quota"
}
-------------------------------

By the way, i try to clean solr cache with an :
echo 1 > /proc/sys/vm/drop_caches

and a restart of the server, but there is always the same problem...

Does anyone have an idea to explore?

sincerly yours,

Guillaume

Guillaume

2014-10-10 11:50:49 UTC

Permalink

Hello,

I retry. Nobody does have an idea about my index problem?

The biggest trouble for me is :
Is it a solr problem or a dovecot problem?

In my opinion, it's more a dovecot problem because the first research
after a solr reindex give the good answer.

Can someone explains to me how dovecot creates and uses cache please?

Once again, sorry for my poor english level.

Thank you in advance

Guillaume

Post by Guillaume
Hello,
First of all, sorry for my poor english level.
I pass to solr indexing.
All is OK, except for the inbox of one of the mailboxes. For this inbox
(other folders of the same mailbox have no problems), when i do text
search, there is always no response.
Others mailboxes have no problems.
If i do a : "doveadm -Dv fts rescan -u mybox at domain.tld", i can do one
and only one search with the correct response. It index the folder, take
a few seconds and give me the good answer.
If i redo the same search, it's extremly quick with no answers. It's
like a cache problem but i don't know where and why...
Logs are not very verbose...
----------------------------------------
$ nc 127.0.0.1 143
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN] Bienvenue sur le serveur IMAP de
Domain.tld
10 LOGIN "mybox at domain.tld" "MyBigSecret"
10 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS
THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN
NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH
ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY
MOVE SEARCH=FUZZY QUOTA] Logged in
20 SELECT INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-2 unknown-9
unknown-0 unknown-10 unknown-1 unknown-4 $MDNSent unknown-3 unknown-8
$Forwarded NonJunk)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-2
unknown-9 unknown-0 unknown-10 unknown-1 unknown-4 $MDNSent unknown-3
unknown-8 $Forwarded NonJunk \*)] Flags permitted.
* 8437 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1412669180] UIDs valid
* OK [UIDNEXT 8439] Predicted next UID
* OK [HIGHESTMODSEQ 20] Highest
20 OK [READ-WRITE] Select completed (0.000 secs).
30 SEARCH text "coucou"
* OK Indexed 14% of the mailbox, ETA 0:57
* OK Indexed 29% of the mailbox, ETA 0:47
* OK Indexed 72% of the mailbox, ETA 0:11
* OK Mailbox indexing finished
* SEARCH 35 108 153 175 176 182 348 357 360 431 472 479 520 574 575 576
577 578 586 949 984 1025 1032 1033 1034 1087 1099 1129 1284 1313 1314
1315 1366 1372 1465 1538 1711 1726 1758 1759 1969 1981 1983 1984 2036
2077 2104 2116 2162 2181 2193 2437 2504 2587 2663 2808 2809 2854 2861
2862 3076 3087 3088 3137 3188 3189 3190 3191 3192 3193 3196 3197 3250
3251 3252 3327 3328 3329 3330 3331 3332 3379 3464 3481 3494 3521 3528
3582 3589 3593 3596 3651 3667 3672 3695 3787 3789 3939 3940 3941 3942
3943 3944 3945 3946 4019 4020 4034 4043 4134 4146 4175 4241 4346 4379
4422 4490 4491 4537 4540 4542 4543 4544 4550 4627 4727 4742 4767 4790
4817 4828 4831 4839 4862 4888 4899 4901 4906 4911 4916 4957 4999 5003
5004 5008 5009 5010 5011 5012 5013 5039 5062 5081 5092 5119 5138 5145
5150 5152 5185 5186 5187 5216 5246 5247 5282 5283 5314 5316 5320 5322
5325 5337 5353 5360 5363 5385 5432 5447 5472 5485 5496 5498 5500 5528
5551 5564 5567 5595 5598 5642 5794 5841 5896 5933 5960 5961 5994 6001
6005 6016 6017 6051 6075 6125 6152 6163 6215 6238 6239 6240 6241 6287
6311 6342 6350 6353 6357 6369 6431 6472 6689 6928 6945 6951 7026 7030
7045 7046 7115 7210 7212 7214 7230 7256 7295 7298 7340 7489 7490 7492
7654 7655 7658 7660 7676 7767 7768 7800 7801 7913 7937 7956 8075 8085
8102 8111 8144 8266 8329 8333 8394 8395 8431
30 OK Search completed (36.029 secs).
40 SEARCH text "coucou"
* SEARCH
40 OK Search completed (0.002 secs).
50 SEARCH text "blabla"
* SEARCH
50 OK Search completed (0.002 secs).
60 LOGOUT
* BYE Logging out
60 OK Logout completed.
----------------------------------------
----------------------------------------
# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 3.13.0-36-generic x86_64 Ubuntu 14.04.1 LTS ext4
auth_mechanisms = plain login
debug_log_path = /var/log/dovecot-debug.log
disable_plaintext_auth = no
first_valid_uid = 5000
hostname = mailserver.domain.tld
info_log_path = /var/log/dovecot-info.log
last_valid_uid = 5000
lda_mailbox_autocreate = yes
lda_original_recipient_header = X-Original-To
listen = *
log_path = /var/log/dovecot-err.log
login_greeting = Bienvenue sur le serveur IMAP de Domain.tld
mail_debug = yes
mail_gid = 5000
mail_location = maildir:/data/mails/%d/%n
mail_plugins = " quota fts fts_solr"
mail_uid = 5000
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
passdb {
driver = pam
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
fts = solr
fts_autoindex = yes
fts_solr = break-imap-search url=http://localhost:8080/solr/
quota = maildir:User quota
sieve_dir = /home/vmail/managesieve/%d/%n
sieve_global_dir = /home/vmail/sieve/
}
postmaster_address = postmaster at domain.tld
protocols = " imap sieve pop3"
service auth {
unix_listener /var/spool/postfix/private/auth {
group = root
mode = 0660
user = postfix
}
unix_listener auth-userdb {
group = vmail
mode = 0660
user = vmail
}
}
service dict {
unix_listener dict {
group = vmail
mode = 0600
user = vmail
}
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
inet_listener sieve_deprecated {
port = 2000
}
}
ssl_ca = </data/certificats/web/domain.tld/ca.pem
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
userdb {
driver = passwd
}
userdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
verbose_proctitle = yes
protocol lmtp {
mail_plugins = " sieve"
}
protocol lda {
mail_plugins = " quota fts fts_solr sieve"
}
protocol imap {
mail_plugins = " quota fts fts_solr imap_quota"
}
-------------------------------
echo 1 > /proc/sys/vm/drop_caches
and a restart of the server, but there is always the same problem...
Does anyone have an idea to explore?
sincerly yours,
Guillaume

--
Guillaume Buiret-F?dit
22, avenue de la Gare
B?t. C1
91570 Bi?vres
T?l : 06.61.33.65.75

Teemu Huovila

2014-10-10 12:15:32 UTC

Permalink

Hello

Post by Guillaume
Is it a solr problem or a dovecot problem?
In my opinion, it's more a dovecot problem because the first research after a solr reindex give the good answer.

If at all possible, you should try a newer version of Dovecot. There have been quite a few changes to FTS and the SOLR backend
since version 2.2.9. See attached log for HG log of changes. Hope this helps.

br,
Teemu Huovila
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fts-solr.log
Type: text/x-log
Size: 38068 bytes
Desc: not available
URL: <http://dovecot.org/pipermail/dovecot/attachments/20141010/b7cd51f1/attachment-0001.bin>

Guillaume

2014-10-10 12:46:08 UTC

Permalink

Thank you for your answer. I will try to upgrade dovecot.
By the way, I erase cache of solr ( <delete><query>*:*</query></delete>
) and the problem is always here...

When I will can upgrade dovecot, i will write back to say if its good or
not.

Sincerely yours,

Guillaume

Post by Teemu Huovila
Hello

Post by Guillaume
Is it a solr problem or a dovecot problem?
In my opinion, it's more a dovecot problem because the first research after a solr reindex give the good answer.

Guillaume

2014-10-16 07:46:27 UTC

Permalink

Hello,

Once again, sorry for my poor english skills...

I upgrade dovecot to 2.2.13, and... the problem is always here. :-(

So i purge (once again) solr entirely and reinstall it. But I forget to
install the schema.xml in solr. I have only the original schema which is
given with the package.
==> All is OK (but a bit slow : 5s against less than a second)

I reinstall the schema.xml of dovecot, it's very fast but... I have the
problem again...

Now i know the source of the problem! ;)
But i'm not able to configure correctly the schema at this moment... :-(

So i will preserve the original schema for the moment.

Good bye

Guillaume

Post by Guillaume
Thank you for your answer. I will try to upgrade dovecot.
By the way, I erase cache of solr ( <delete><query>*:*</query></delete>
) and the problem is always here...
When I will can upgrade dovecot, i will write back to say if its good or
not.
Sincerely yours,
Guillaume

Post by Teemu Huovila
Hello

Post by Guillaume
Is it a solr problem or a dovecot problem?
In my opinion, it's more a dovecot problem because the first research
after a solr reindex give the good answer.