Skip to content

Dovecot segfaulting (Ubuntu 14.04 LTS, CentOS 6 and 7)

Open Source

We're currently installing a solution including dovecot for a company to go into production in April. So we kick this off with a development box for integrating parts that different suppliers to our customer are working on.

But after installing dovecot on the new joint development machine it just didn't start. It worked on our local development boxes but the install for the customer has been scripted with (what we call) "poor man's puppet" so it is somewhat hard to compare the setups. Same Ubuntu 14.04 LTS under the hood but on top of that things (like config layout, directory structures etc.) are quite different.

Back on topic: ps aux | grep d[o]ve returned empty.

Looking at /var/log/mail.log did not show anything relevant.

But syslog (/var/log/syslog) had some worrying lines like:

Mar 16 03:16:17 dev-new kernel: [ 3222.339365] doveconf[6420]: segfault at 200 ip 00007fa041b25a03 sp 00007ffe7881e070 error 4 in libc-2.19.so[7fa041ada000+1bb000]

Manually running the daemon resulted in:

# /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
Segmentation fault

So

mkdir /var/core
chmod 1777 /var/core
echo "/var/core/%p" > /proc/sys/kernel/core_pattern
ulimit -c unlimited
/usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
Segmentation fault (core dumped)

Better. We have a core file now.

But:

# gdb /usr/sbin/dovecot /var/core/2032
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
[..]
Reading symbols from /usr/sbin/dovecot...(no debugging symbols found)...done.
[..]
warning: core file may not match specified executable file.
[New LWP 2032]
Core was generated by `/usr/bin/doveconf -f service=master -c /etc/dovecot/dovecot.conf -m master -p -'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fb0deb94a03 in ?? ()
(gdb) bt full
#0  0x00007fb0deb94a03 in ?? ()
No symbol table info available.
#1  0x0000001000000004 in ?? ()
No symbol table info available.
[..]
#15 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) q

So apt-get install dovecot-dbg and run this in gdb with debugging symbols:

# gdb --args /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
[..]
Reading symbols from /usr/sbin/dovecot...Reading symbols from /usr/lib/debug//usr/sbin/dovecot...done.
[..]
(gdb) r
Starting program: /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
process 6073 is executing new program: /usr/bin/doveconf

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7798a03 in _IO_vfprintf_internal (s=s@entry=0x7fffffffdf80, format=, 
    format@entry=0x55555556be60 "Error in configuration file %s: %s", ap=ap@entry=0x7fffffffe178) at vfprintf.c:1661
1661    vfprintf.c: No such file or directory.
(gdb) bt full
#0  0x00007ffff7798a03 in _IO_vfprintf_internal (s=s@entry=0x7fffffffdf80, format=, 
    format@entry=0x55555556be60 "Error in configuration file %s: %s", ap=ap@entry=0x7fffffffe178) at vfprintf.c:1661
        len = 
        string_malloced = 
[..]
#8  0x000055555555ff6a in main (argc=13, argv=0x55555577f390) at doveconf.c:795
        scope = CONFIG_DUMP_SCOPE_ALL
        orig_config_path = 
        config_path = 0x555555780000 "/etc/dovecot/dovecot.conf"
        module = 0x5555557770b0 "master"
[..]
(gdb) quit
A debugging session is active.

    Inferior 1 [process 6073] will be killed.

Quit anyway? (y or n) y

Aah, yes.
So it segfaults reading its own config. Defensive programming++.

A quick Google run shows this has been well known since 2014, e.g. reported for Ubuntu here. Unfortunately the advice given on the Dovecot mailing-list is the one so often received from upstreams:

>> so don't report problems with a outdated version instead
>> just upgrade - there where bugfixes between 2.2.9 and
>> 2.2.13 in context of segfaults here and there
>>
> Thank you. This outdated version is the one packaged in Ubuntu
> LTS 14.04.1, which I believe I'm not the only one using

that's the problem with all that LTS packages

nobody knows what fixes they may have backported and
what are missing, so the version number no longer says
anything which makes it also impossible for the upstream
developer to know the patchlevel

that's why i build packages for server software the last
7 years on my own infrastructure from upstream sources

"Just upgrade" ... that is not the solution. The reason for LTS distros is that you want to minimize the risk for finding the new bugs. I fully understand developers would like users to run the latest & greatest but that is a risk few people will take in production setups.

But backporting the old fixes would really have help here...

In our case Dovecot 2.2.9 (Ubuntu 14.04 LTS (Trusty) version) happily segfaults on

auth_verbose_passwords = yes

in /etc/dovecot/dovecot.conf. Replacing it with

auth_verbose_passwords = plain

Dovecot starts working again.

That setting was added for the joint dev box to aid debugging. We did not have a auth_verbose_passwords line on our systems or the intended prod configuration.

The segfaulting on "yes" instead of "plain" has been fixed in upstream Dovecot in commits #3190f12 and #a5bcc9f two years ago.

Ubuntu, CentOS, for your backporting pleasure:

* 3190f12 - auth: Allow auth_verbose_passwords=yes as an alias for "plain". (2 years, 1 month ago) [Timo Sirainen]| 
| diff --git a/src/auth/auth-settings.c b/src/auth/auth-settings.c
| index 586ee13..994730c 100644
| --- a/src/auth/auth-settings.c
| +++ b/src/auth/auth-settings.c
| @@ -330,7 +330,7 @@ auth_settings_set_self_ips(struct auth_settings *set, pool_t pool,
|  }
|  
|  static bool
| -auth_verify_verbose_password(const struct auth_settings *set,
| +auth_verify_verbose_password(struct auth_settings *set,
|                            const char **error_r)
|  {
|       const char *p, *value = set->verbose_passwords;
| @@ -351,7 +351,11 @@ auth_verify_verbose_password(const struct auth_settings *set,
|               return TRUE;
|       else if (strcmp(value, "sha1") == 0)
|               return TRUE;
| -     else {
| +     else if (strcmp(value, "yes") == 0) {
| +             /* just use it as alias for "plain" */
| +             set->verbose_passwords = "plain";
| +             return TRUE;
| +     } else {
|               *error_r = "auth_verbose_passwords: Invalid value";
|               return FALSE;
|       }

* a5bcc9f - auth: Fixed crash/NULL error if auth_verbose_passwords had an invalid value. (2 years, 1 month ago) [Timo Sirainen]| 
| diff --git a/src/auth/auth-settings.c b/src/auth/auth-settings.c
| index 3ebb70a..586ee13 100644
| --- a/src/auth/auth-settings.c
| +++ b/src/auth/auth-settings.c
| @@ -351,8 +351,10 @@ auth_verify_verbose_password(const struct auth_settings *set,
|               return TRUE;
|       else if (strcmp(value, "sha1") == 0)
|               return TRUE;
| -     else
| +     else {
| +             *error_r = "auth_verbose_passwords: Invalid value";
|               return FALSE;
| +     }
|  }
|  
|  static bool auth_settings_check(void *_set, pool_t pool,

But: this is not a good design.

Dovecot should really implement a config reader that does not depend on manual strcmp's and throwing errors hand coded for each config setting.
And segfaulting in case where that was forgotten or not backported to the widely used versions in the popular distributions.

NB: Debian Jessie has Dovecot 2.2.13, Ubuntu 16.04 LTS (Xenial) has 2.2.18, both include the fix.

Trackbacks

No Trackbacks

Comments

Display comments as Linear | Threaded

Raimondo Giammanco on :

Thank you very much for your post! As of this moment it seems centos 7.2 with 2.2.10-5 still has the yes instead of plain.

Really thanks for the help.

Add Comment

Markdown format allowed
Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA

Form options

Submitted comments will be subject to moderation before being displayed.