vsz_limit is 18,447 PB

Discussion:

Dan Langille

2014-09-06 20:38:46 UTC

I encountered this error on FreeBSD 9.3 with dovecot2-2.2.13_3

Short version; vsz_limit is 18,447 PB and it ran out of RAM. Suggestions for change?

Sep 6 03:39:32 mailjail dovecot: imap(dan): Panic: file imap-fetch.c: line 556 (imap_fetch_more): assertion failed: (ctx->client->output_cmd_lock == NULL || ctx->client->output_cmd_lock == cmd)
Sep 6 03:39:32 mailjail dovecot: imap(dan): Fatal: master: service(imap): child 71153 killed with signal 6 (core not dumped - set service imap { drop_priv_before_exec=yes })
Sep 6 03:59:41 mailjail dovecot: imap(dan): Fatal: pool_system_realloc(2097152): Out of memory
Sep 6 03:59:41 mailjail dovecot: imap(dan): Fatal: master: service(imap): child 67732 returned error 83 (Out of memory (service imap { vsz_limit=256 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

Background: I?m in the only user on this system, but this server is accessed by my phone, my laptop, my tablet, and perhaps a web interface.

Here are the non-default values:

$ doveconf -n
# 2.2.13: /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 9.3-RELEASE amd64
auth_mechanisms = plain login
first_valid_gid = 1001
first_valid_uid = 1001
mail_location = maildir:~/Maildir
mail_max_userip_connections = 80
mail_privileged_group = mail
passdb {
args = scheme=SHA512-CRYPT /var/db/dovecot.users
driver = passwd-file
}
protocols = imap
service imap-login {
inet_listener imap {
address = 10.0.0.1
}
inet_listener imaps {
port = 0
}
}
ssl = required
ssl_ca = </usr/local/etc/ssl/ca.pem
ssl_cert = </usr/local/etc/ssl/server.pem
ssl_key = </usr/local/etc/ssl/mailjail.example.org.nopassword.key
userdb {
args = /var/db/dovecot.users
driver = passwd-file
}
verbose_proctitle = yes
verbose_ssl = yes

But there are some interesting values when I look at the output of doveconf. Specifically, vsz_limit is 18,447 PB? yeah, that?s pretty big.

service imap-login {
chroot = login
client_limit = 0
drop_priv_before_exec = no
executable = imap-login
extra_groups =
group =
idle_kill = 0
inet_listener imap {
address = 10.0.0.1
port = 143
reuse_port = no
ssl = no
}
inet_listener imaps {
address =
port = 0
reuse_port = no
ssl = yes
}
privileged_group =
process_limit = 0
process_min_avail = 0
protocol = imap
service_count = 1
type = login
user = $default_login_user
vsz_limit = 18446744073709551615 B
}

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 333 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://dovecot.org/pipermail/dovecot/attachments/20140906/468756f2/attachment.sig>

Jiri Bourek

2014-09-07 08:39:39 UTC

Permalink

Post by Dan Langille
I encountered this error on FreeBSD 9.3 with dovecot2-2.2.13_3
Short version; vsz_limit is 18,447 PB and it ran out of RAM. Suggestions for change?
Sep 6 03:39:32 mailjail dovecot: imap(dan): Panic: file imap-fetch.c: line 556 (imap_fetch_more): assertion failed: (ctx->client->output_cmd_lock == NULL || ctx->client->output_cmd_lock == cmd)
Sep 6 03:39:32 mailjail dovecot: imap(dan): Fatal: master: service(imap): child 71153 killed with signal 6 (core not dumped - set service imap { drop_priv_before_exec=yes })
Sep 6 03:59:41 mailjail dovecot: imap(dan): Fatal: pool_system_realloc(2097152): Out of memory
Sep 6 03:59:41 mailjail dovecot: imap(dan): Fatal: master: service(imap): child 67732 returned error 83 (Out of memory (service imap { vsz_limit=256 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

Check the message again - it says vsz_limit=256MB

Post by Dan Langille
Background: I?m in the only user on this system, but this server is accessed by my phone, my laptop, my tablet, and perhaps a web interface.
But there are some interesting values when I look at the output of doveconf. Specifically, vsz_limit is 18,447 PB? yeah, that?s pretty big.
service imap-login {
vsz_limit = 18446744073709551615 B
}

According to your log it was process "imap" what ran out of memory. You
are showing configuration for imap-login - they are not the same thing,
see http://wiki2.dovecot.org/Design/Processes

Not sure if this behaviour is a bug or something that is to be expected
when you run out of VSZ limit in imap process - that needs to be
answered by someone more knowledgeable

Dan Langille

2014-09-08 20:58:23 UTC

Permalink

Post by Jiri Bourek

Check the message again - it says vsz_limit=256MB

Yes, I saw that?. I can?t explain why I didn?t question the rest of the output.

Post by Jiri Bourek

According to your log it was process ?im ap" what ran out of memory. You are showing configuration for imap-login - they are not the same thing, see http://wiki2.dovecot.org/Design/Processes

My errror, sorry. OK, here?s imap:

service imap {
chroot =
client_limit = 1
drop_priv_before_exec = no
executable = imap
extra_groups =
group =
idle_kill = 0
privileged_group =
process_limit = 1024
process_min_avail = 0
protocol = imap
service_count = 1
type =
unix_listener login/imap {
group =
mode = 0666
user =
}
user =
vsz_limit = 18446744073709551615 B
}

Still the same value? why not 265MB?

Post by Jiri Bourek
Not sure if this behaviour is a bug or something that is to be expected when you run out of VSZ limit in imap process - that needs to be answered by someone more knowledgeable

It seems quite common:

$ doveconf | grep vsz_limit
default_vsz_limit = 256 M
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
vsz_limit = 18446744073709551615 B
[dan at mailjail:~] $

Thank you.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 333 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://dovecot.org/pipermail/dovecot/attachments/20140908/478822f3/attachment.sig>

Jiri Bourek

2014-09-08 22:38:08 UTC

Permalink

Post by Dan Langille

Post by Jiri Bourek

According to your log it was process ?im ap" what ran out of memory. You are showing configuration for imap-login - they are not the same thing, see http://wiki2.dovecot.org/Design/Processes

service imap {
chroot =
client_limit = 1
drop_priv_before_exec = no
executable = imap
extra_groups =
group =
idle_kill = 0
privileged_group =
process_limit = 1024
process_min_avail = 0
protocol = imap
service_count = 1
type =
unix_listener login/imap {
group =
mode = 0666
user =
}
user =
vsz_limit = 18446744073709551615 B
}
Still the same value? why not 265MB?

Hm, I can only guess (someone with more knowledge of dovecot source code
probably knows better), but it seems like that value is magic number for
"not set". My doveconf outputs the same thing on services which don't
have vsz_limit set explicitly, but when I list /proc/[pid]/limits on
such a service, the row called "Max address space" (which is - AFAIK -
limit on vsz) shows 2GB, a value of default_vsz_limit setting.

I'd say don't overthink it, just put explicit vsz_limit = 512M into
service imap section (if your machine's RAM allows) and be done with it.

Robert L Mathews

2014-09-08 23:41:18 UTC

Permalink

On 9/8/14 3:38 PM, Jiri Bourek wrote:> Hm, I can only guess (someone
with more knowledge of dovecot source code

Post by Jiri Bourek
probably knows better), but it seems like that value is magic number for
"not set". My doveconf outputs the same thing on services which don't
have vsz_limit set explicitly, but when I list /proc/[pid]/limits on
such a service, the row called "Max address space" (which is - AFAIK -
limit on vsz) shows 2GB, a value of default_vsz_limit setting.
I'd say don't overthink it, just put explicit vsz_limit = 512M into
service imap section (if your machine's RAM allows) and be done with it.

Right. According to my notes from when I upgraded to 2.x,
"18446744073709551615" means it's not set and that the default_vsz_limit
will apply, which is 256 MB.

To avoid the problem the original poster had, I used:

default_vsz_limit = 1 G

--
Robert L Mathews, Tiger Technologies, http://www.tigertech.net/

Darac Marjal

2014-09-09 12:59:58 UTC

Permalink

Post by Robert L Mathews
On 9/8/14 3:38 PM, Jiri Bourek wrote:> Hm, I can only guess (someone
with more knowledge of dovecot source code

Right. According to my notes from when I upgraded to 2.x,
"18446744073709551615" means it's not set and that the default_vsz_limit
will apply, which is 256 MB.

That seems sensible to me. 2^64 bytes is 18446744073709551616 (18.45EB),
so the number shown is (2^64 - 1). As a *limit* for a 64-bit program,
that seems about as close to "no limit" as you're going to get :)

Post by Robert L Mathews
default_vsz_limit = 1 G
--
Robert L Mathews, Tiger Technologies, http://www.tigertech.net/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://dovecot.org/pipermail/dovecot/attachments/20140909/0c56b948/attachment.sig>

Dan Langille

2014-09-15 14:20:15 UTC

Permalink

Post by Dan Langille

Post by Jiri Bourek

According to your log it was process ?im ap" what ran out of memory. You are showing configuration for imap-login - they are not the same thing, see http://wiki2.dovecot.org/Design/Processes

Hm, I can only guess (someone with more knowledge of dovecot source code probably knows better), but it seems like that value is magic number for "not set". My doveconf outputs the same thing on services which don't have vsz_limit set explicitly, but when I list /proc/[pid]/limits on such a service, the row called "Max address space" (which is - AFAIK - limit on vsz) shows 2GB, a value of default_vsz_limit setting.
I'd say don't overthink it, just put explicit vsz_limit = 512M into service imap section (if your machine's RAM allows) and be done with it.

For the record, the error has recurred a few times, so after today?s incident, I added this to dovecot.conf, and restarted dovecot:

service imap {
vsz_limit = 512MB
}

FWIW, the server contains 8192 MB of RAM.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 333 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://dovecot.org/pipermail/dovecot/attachments/20140915/99e461e5/attachment.sig>

Dan Langille

2014-09-25 02:04:58 UTC

Permalink

Post by Dan Langille

Post by Jiri Bourek

According to your log it was process ?im ap" what ran out of memory. You are showing configuration for imap-login - they are not the same thing, see http://wiki2.dovecot.org/Design/Processes

Hm, I can only guess (someone with more knowledge of dovecot source code probably knows better), but it seems like that value is magic number for "not set". My doveconf outputs the same thing on services which don't have vsz_limit set explicitly, but when I list /proc/[pid]/limits on such a service, the row called "Max address space" (which is - AFAIK - limit on vsz) shows 2GB, a value of default_vsz_limit setting.
I'd say don't overthink it, just put explicit vsz_limit = 512M into service imap section (if your machine's RAM allows) and be done with it.

service imap {
vsz_limit = 512MB
}
FWIW, the server contains 8192 MB of RAM.

For the record, the problem has not recurred.

?
Dan Langille

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 333 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://dovecot.org/pipermail/dovecot/attachments/20140924/1c1ca699/attachment.sig>

Joseph Tam

2014-09-09 22:37:49 UTC

Permalink

Post by Dan Langille
vsz_limit = 18446744073709551615 B
}
Still the same value? why not 265MB?

I guess this is Dovecot's approximation of infinity. It's the largest value
that fits into a 64-bit signed integer (2^64-1).

Joseph Tam <jtam.home at gmail.com>