Dovecot segfaulting (Ubuntu 14.04 LTS, CentOS 6 and 7)
We're currently installing a solution including dovecot for a company to go into production in April. So we kick this off with a development box for integrating parts that different suppliers to our customer are working on.
But after installing dovecot on the new joint development machine it just didn't start. It worked on our local development boxes but the install for the customer has been scripted with (what we call) "poor man's puppet" so it is somewhat hard to compare the setups. Same Ubuntu 14.04 LTS under the hood but on top of that things (like config layout, directory structures etc.) are quite different.
Back on topic: ps aux | grep d[o]ve
returned empty.
Looking at /var/log/mail.log
did not show anything relevant.
But syslog (/var/log/syslog
) had some worrying lines like:
Mar 16 03:16:17 dev-new kernel: [ 3222.339365] doveconf[6420]: segfault at 200 ip 00007fa041b25a03 sp 00007ffe7881e070 error 4 in libc-2.19.so[7fa041ada000+1bb000]
Manually running the daemon resulted in:
# /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
Segmentation fault
So
mkdir /var/core
chmod 1777 /var/core
echo "/var/core/%p" > /proc/sys/kernel/core_pattern
ulimit -c unlimited
/usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
Segmentation fault (core dumped)
Better. We have a core file now.
But:
# gdb /usr/sbin/dovecot /var/core/2032 GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1 [..] Reading symbols from /usr/sbin/dovecot...(no debugging symbols found)...done. [..] warning: core file may not match specified executable file. [New LWP 2032] Core was generated by `/usr/bin/doveconf -f service=master -c /etc/dovecot/dovecot.conf -m master -p -'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007fb0deb94a03 in ?? () (gdb) bt full #0 0x00007fb0deb94a03 in ?? () No symbol table info available. #1 0x0000001000000004 in ?? () No symbol table info available. [..] #15 0x0000000000000000 in ?? () No symbol table info available. (gdb) q
So apt-get install dovecot-dbg
and run this in gdb with debugging symbols:
# gdb --args /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1 [..] Reading symbols from /usr/sbin/dovecot...Reading symbols from /usr/lib/debug//usr/sbin/dovecot...done. [..] (gdb) r Starting program: /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf process 6073 is executing new program: /usr/bin/doveconf Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7798a03 in _IO_vfprintf_internal (s=s@entry=0x7fffffffdf80, format=, format@entry=0x55555556be60 "Error in configuration file %s: %s", ap=ap@entry=0x7fffffffe178) at vfprintf.c:1661 1661 vfprintf.c: No such file or directory. (gdb) bt full #0 0x00007ffff7798a03 in _IO_vfprintf_internal (s=s@entry=0x7fffffffdf80, format= , format@entry=0x55555556be60 "Error in configuration file %s: %s", ap=ap@entry=0x7fffffffe178) at vfprintf.c:1661 len = string_malloced = [..] #8 0x000055555555ff6a in main (argc=13, argv=0x55555577f390) at doveconf.c:795 scope = CONFIG_DUMP_SCOPE_ALL orig_config_path = config_path = 0x555555780000 "/etc/dovecot/dovecot.conf" module = 0x5555557770b0 "master" [..] (gdb) quit A debugging session is active. Inferior 1 [process 6073] will be killed. Quit anyway? (y or n) y
Aah, yes.
So it segfaults reading its own config. Defensive programming++.
A quick Google run shows this has been well known since 2014, e.g. reported for Ubuntu here. Unfortunately the advice given on the Dovecot mailing-list is the one so often received from upstreams:
>> so don't report problems with a outdated version instead >> just upgrade - there where bugfixes between 2.2.9 and >> 2.2.13 in context of segfaults here and there >> > Thank you. This outdated version is the one packaged in Ubuntu > LTS 14.04.1, which I believe I'm not the only one using that's the problem with all that LTS packages nobody knows what fixes they may have backported and what are missing, so the version number no longer says anything which makes it also impossible for the upstream developer to know the patchlevel that's why i build packages for server software the last 7 years on my own infrastructure from upstream sources
"Just upgrade" ... that is not the solution. The reason for LTS distros is that you want to minimize the risk for finding the new bugs. I fully understand developers would like users to run the latest & greatest but that is a risk few people will take in production setups.
But backporting the old fixes would really have help here...
In our case Dovecot 2.2.9 (Ubuntu 14.04 LTS (Trusty) version) happily segfaults on
auth_verbose_passwords = yes
in /etc/dovecot/dovecot.conf
. Replacing it with
auth_verbose_passwords = plain
Dovecot starts working again.
That setting was added for the joint dev box to aid debugging. We did not have a auth_verbose_passwords
line on our systems or the intended prod configuration.
The segfaulting on "yes" instead of "plain" has been fixed in upstream Dovecot in commits #3190f12 and #a5bcc9f two years ago.
Ubuntu, CentOS, for your backporting pleasure:
* 3190f12 - auth: Allow auth_verbose_passwords=yes as an alias for "plain". (2 years, 1 month ago) [Timo Sirainen]| | diff --git a/src/auth/auth-settings.c b/src/auth/auth-settings.c | index 586ee13..994730c 100644 | --- a/src/auth/auth-settings.c | +++ b/src/auth/auth-settings.c | @@ -330,7 +330,7 @@ auth_settings_set_self_ips(struct auth_settings *set, pool_t pool, | } | | static bool | -auth_verify_verbose_password(const struct auth_settings *set, | +auth_verify_verbose_password(struct auth_settings *set, | const char **error_r) | { | const char *p, *value = set->verbose_passwords; | @@ -351,7 +351,11 @@ auth_verify_verbose_password(const struct auth_settings *set, | return TRUE; | else if (strcmp(value, "sha1") == 0) | return TRUE; | - else { | + else if (strcmp(value, "yes") == 0) { | + /* just use it as alias for "plain" */ | + set->verbose_passwords = "plain"; | + return TRUE; | + } else { | *error_r = "auth_verbose_passwords: Invalid value"; | return FALSE; | } * a5bcc9f - auth: Fixed crash/NULL error if auth_verbose_passwords had an invalid value. (2 years, 1 month ago) [Timo Sirainen]| | diff --git a/src/auth/auth-settings.c b/src/auth/auth-settings.c | index 3ebb70a..586ee13 100644 | --- a/src/auth/auth-settings.c | +++ b/src/auth/auth-settings.c | @@ -351,8 +351,10 @@ auth_verify_verbose_password(const struct auth_settings *set, | return TRUE; | else if (strcmp(value, "sha1") == 0) | return TRUE; | - else | + else { | + *error_r = "auth_verbose_passwords: Invalid value"; | return FALSE; | + } | } | | static bool auth_settings_check(void *_set, pool_t pool,
But: this is not a good design.
Dovecot should really implement a config reader that does not depend on manual strcmp's and throwing errors hand coded for each config setting.
And segfaulting in case where that was forgotten or not backported to the widely used versions in the popular distributions.
NB: Debian Jessie has Dovecot 2.2.13, Ubuntu 16.04 LTS (Xenial) has 2.2.18, both include the fix.
Comments
Display comments as Linear | Threaded
Raimondo Giammanco on :
Thank you very much for your post! As of this moment it seems centos 7.2 with 2.2.10-5 still has the yes instead of plain.
Really thanks for the help.