Discussion:
Anvil client_limit reached
(too old to reply)
Juergen Obermann
2010-11-03 09:35:35 UTC
Permalink
Hello,

we have the following problem:

Nov 3 09:43:33 minerva dovecot: [ID 583609 local0.warning] master: Warning: service(anvil): client_limit reached, client connections are being dropped
Nov 3 09:51:33 minerva dovecot: [ID 583609 local0.error] imap-login: Error: net_connect_unix(anvil) failed: Connection refused
Nov 3 09:51:33 minerva dovecot: [ID 583609 local0.crit] imap-login: Fatal: Couldn't connect to anvil
Nov 3 09:52:42 minerva dovecot: [ID 583609 local0.error] pop3-login: Error: net_connect_unix(anvil) failed: Connection refused
Nov 3 09:52:42 minerva dovecot: [ID 583609 local0.crit] pop3-login: Fatal: Couldn't connect to anvil
Nov 3 09:52:42 minerva dovecot: [ID 583609 local0.error] imap-login: Error: net_connect_unix(anvil) failed: Connection refused
Nov 3 09:52:42 minerva dovecot: [ID 583609 local0.crit] imap-login: Fatal: Couldn't connect to anvil

At that time I counted 2030 imap processes from 995 users and 1106 imap-login processes. What can I do to set an appropriate client_limit for anvil?

The configuration is:

# doveadm config -n
# 2.0.5: /opt/local/etc/dovecot/dovecot.conf
# OS: SunOS 5.10 sun4v
auth_verbose = yes
default_client_limit = 3000
default_process_limit = 300
disable_plaintext_auth = no
listen = xxx.xxx.uni-giessen.de localhost
mail_fsync = always
mail_location = mbox:~/Mail:INBOX=/var/mail/%u
mail_nfs_index = yes
mail_nfs_storage = yes
mail_plugins = quota
mmap_disable = yes
namespace {
inbox = yes
location =
prefix =
separator = /
type = private
}
namespace {
hidden = yes
list = no
location =
prefix = Mail/
separator = /
subscriptions = yes
type = private
}
passdb {
driver = pam
}
postmaster_address = postmaster at hrz.uni-giessen.de
protocols = imap pop3
service auth {
client_limit = 5720
}
service imap {
process_limit = 4096
}
ssl_cert = </etc/ssl/server.crt
ssl_key = </etc/ssl/server.key
syslog_facility = local0
userdb {
driver = passwd
}
verbose_proctitle = yes
version_ignore = yes
protocol imap {
imap_client_workarounds = delay-newmail tb-extra-mailbox-sep
mail_max_userip_connections = 20
mail_plugins = quota imap_quota
}
protocol pop3 {
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
pop3_uidl_format = %08Xv%08Xu
}
minerva{root}2509:/
# grep anvil /var/log/locallog

minerva{root}2510:/
--
Hochschulrechenzentrum der | Mail: Juergen.Obermann at hrz.uni-giessen.de
Justus-Liebig-Universitaet | WWW: http://www.uni-giessen.de/obermann
Heinrich-Buff-Ring 44 | Tel: 0641-99-13054 (0641-99-13001)
D-35392 Giessen, Germany | Fax: 0641-99-13009
Timo Sirainen
2010-11-03 11:12:42 UTC
Permalink
Post by Juergen Obermann
Nov 3 09:43:33 minerva dovecot: [ID 583609 local0.warning] master: Warning: service(anvil): client_limit reached, client connections are being dropped
At that time I counted 2030 imap processes from 995 users and 1106 imap-login processes. What can I do to set an appropriate client_limit for anvil?
Well, it's easy to answer the question:

service anvil {
client_limit = 8000 # or something
}

But you would probably benefit from not having that many login processes: http://wiki2.dovecot.org/LoginProcess
Post by Juergen Obermann
default_client_limit = 3000
default_process_limit = 300
If the default process limit is 300 and you haven't overridden that (I didn't see service imap-login in your settings), how do you have 1106 imap-login processes?

Also you didn't mention how many pop3-login processes there were. With anvil's client_limit being 3000, this error shouldn't have happened unless there were a total of about 3000 imap/pop3-login processes.

So maybe there are some bugs related to this.. Also I should add a check to startup that it warns if anvil's client_limit is too low.
Jürgen Obermann
2010-11-03 13:12:34 UTC
Permalink
Post by Timo Sirainen
Post by Juergen Obermann
Warning: service(anvil): client_limit reached, client connections
are being dropped
At that time I counted 2030 imap processes from 995 users and 1106
imap-login processes. What can I do to set an appropriate
client_limit for anvil?
service anvil {
client_limit = 8000 # or something
}
But you would probably benefit from not having that many login
processes: http://wiki2.dovecot.org/LoginProcess
I now switched the login processes from high security to high performance mode,
because the problem appeared again.
Post by Timo Sirainen
Post by Juergen Obermann
default_client_limit = 3000
default_process_limit = 300
If the default process limit is 300 and you haven't overridden that
(I didn't see service imap-login in your settings), how do you have
1106 imap-login processes?
Perhaps the imap-login processes should have gone away but still hang araound and serve SSL sessions
Post by Timo Sirainen
Also you didn't mention how many pop3-login processes there were.
With anvil's client_limit being 3000, this error shouldn't have
happened unless there were a total of about 3000 imap/pop3-login
processes.
Normally only a few but if dovecot hangs there are up to 64 pop3-login processes (64 is the number of cpus)
Post by Timo Sirainen
So maybe there are some bugs related to this.. Also I should add a
check to startup that it warns if anvil's client_limit is too low.
--
Hochschulrechenzentrum der | Mail: Juergen.Obermann at hrz.uni-giessen.de
Justus-Liebig-Universitaet | WWW: http://www.uni-giessen.de/obermann/
Heinrich-Buff-Ring 44 | Tel: 0641-99-13054 (0641-99-13001)
D-35392 Giessen, Germany | Fax: 0641-99-13009

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
Charles Marcus
2010-11-03 13:26:30 UTC
Permalink
Post by Jürgen Obermann
Post by Timo Sirainen
But you would probably benefit from not having that many login
processes: http://wiki2.dovecot.org/LoginProcess
I now switched the login processes from high security to high
performance mode, because the problem appeared again.
Just curious - how much less secure is the high performance mode?
--
Best regards,

Charles
Timo Sirainen
2010-11-03 15:43:16 UTC
Permalink
Post by Charles Marcus
Post by Jürgen Obermann
Post by Timo Sirainen
But you would probably benefit from not having that many login
processes: http://wiki2.dovecot.org/LoginProcess
I now switched the login processes from high security to high
performance mode, because the problem appeared again.
Just curious - how much less secure is the high performance mode?
Copy&pasting (slightly just updated) from the wiki:

High-performance mode

It works by using a number of long running login processes, each
handling a number of connections. This loses much of the security
benefits of the login process design, because in case of a security hole
(in Dovecot or SSL library) the attacker is now able to see other users
logging in and steal their passwords, read their mails, etc.
Timo Sirainen
2010-11-03 15:35:10 UTC
Permalink
Post by Jürgen Obermann
Post by Timo Sirainen
Post by Juergen Obermann
default_client_limit = 3000
default_process_limit = 300
If the default process limit is 300 and you haven't overridden that
(I didn't see service imap-login in your settings), how do you have
1106 imap-login processes?
Perhaps the imap-login processes should have gone away but still hang araound and serve SSL sessions
No. Looks like default_process_limit was never used. The default was
unlimited number of processes. This is now fixed:
http://hg.dovecot.org/dovecot-2.0/rev/b4b6bf889044

Removed unnecessary anvil connection from each auth worker process:
http://hg.dovecot.org/dovecot-2.0/rev/3ce7879e3ac0

Added a warning to startup if anvil's client_limit is too low:
http://hg.dovecot.org/dovecot-2.0/rev/97b286b6ec45

And if verbose_proctitle=yes anvil now shows in ps output how many
connections it's currently serving:
http://hg.dovecot.org/dovecot-2.0/rev/4d91cec713cf
http://hg.dovecot.org/dovecot-2.0/rev/7b1aae5f7bd5

Hopefully those will get rid of this problem in future.
Jürgen Obermann
2010-11-04 19:07:41 UTC
Permalink
Post by Timo Sirainen
Post by Jürgen Obermann
Post by Timo Sirainen
Post by Juergen Obermann
default_client_limit = 3000
default_process_limit = 300
If the default process limit is 300 and you haven't overridden that
(I didn't see service imap-login in your settings), how do you have
1106 imap-login processes?
Perhaps the imap-login processes should have gone away but still
hang araound and serve SSL sessions
No. Looks like default_process_limit was never used. The default was
http://hg.dovecot.org/dovecot-2.0/rev/b4b6bf889044
http://hg.dovecot.org/dovecot-2.0/rev/3ce7879e3ac0
http://hg.dovecot.org/dovecot-2.0/rev/97b286b6ec45
And if verbose_proctitle=yes anvil now shows in ps output how many
http://hg.dovecot.org/dovecot-2.0/rev/4d91cec713cf
http://hg.dovecot.org/dovecot-2.0/rev/7b1aae5f7bd5
Hopefully those will get rid of this problem in future.
Yes, they do. Thank you!
How can I find out what the appropriate limits for our mail-server are?

Greetings, J?rgen
--
Hochschulrechenzentrum der | Mail: Juergen.Obermann at hrz.uni-giessen.de
Justus-Liebig-Universitaet | WWW: http://www.uni-giessen.de/obermann/
Heinrich-Buff-Ring 44 | Tel: 0641-99-13054 (0641-99-13001)
D-35392 Giessen, Germany | Fax: 0641-99-13009

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
Timo Sirainen
2010-11-04 19:21:20 UTC
Permalink
Post by Jürgen Obermann
Post by Timo Sirainen
Hopefully those will get rid of this problem in future.
Yes, they do. Thank you!
How can I find out what the appropriate limits for our mail-server are?
It should be at least:

service imap-login { process_limit } +
service pop3-login { process_limit } +
service managesieve-login { process_limit } +
3

+ a few more so that during full load it won't temporarily complain
about the the limit being reached.

Continue reading on narkive:
Loading...