Daniel Lange's blog

Apparently somebody managed to resell Seagate hard disks that have 2-5 years of operations on them as brand new.

They did this by using some new shrink wrap bags and resetting the used hard disk SMART attributes to factory-new values.

Luckily Seagate has a proprietary extension "Seagate FARM (Field Access Reliability Metrics)" implemented in their disks that ... the crooks did not reset.

Luckily ... because other manufacturers do not have that extension. And you think the crooks only re-sell used Seagate disks? Lol.

To get access to the Seagate FARM extension, you need smartctl from smartmontools v7.4 or later.

For Debian 12 (Bookworm) you can add the backports archive and then install with apt install smartmontools/bookworm-backports.

For Debian 11 (Bullseye) you can use a backport we created at my company:

You can also download static builds from https://builds.smartmontools.org/ which keeps the latest CI builds of the current development branch (v7.5 at the time of writing).

To check the state of your drives, compare the output from smartctl -x and smartctl -l farm. Double checking Power_On_Hours vs. "Power on Hours" is the obvious. But the other values around "Head Flight Hours" and "Power Cycle Count" should also roughly match what you expect from a hard disk of a certain age. All near zero, of course, for a factory-new hard disk.

This is what it looks like for a hard disk that has gracefully serviced 4 years and 8 months so far. The smartctl -x and smartctl -l farm data match within some small margins:

You may like to run the command below on your systems to capture the state. Remember FARM is only supported on Seagate drives.

In 2020 my company bought a DYMO LabelWriter Wireless. It is an awesome little device for thermal printing a wide variety of labels. The labels are easily available both from DYMO and from third parties so the pricing is quite acceptable.

Unfortunately DYMO supplies their DYMO Connect Software only for Microsoft Windows and MacOSX. A mobile app of the same name for Android and Apple iOS devices is available in the app stores.

There is a SDK for Linux and there are drivers published for Linux but the LabelWriter Wireless was not supported on Linux when I tried to get it running for Debian in 2020.

This year I have had a new look at the situation as we still use the LabelWriter Wireless printers a lot and the company runs fully on Linux. So it is always a chore to run a Windows VM just to run DYMO Connect.

Thunderbird in Debian stable (Bookworm) has received Thunderbird v115.3.1 as a security update.

With it comes "Supernova", a UI redesign. There is a Mozilla blogpost with a walk-through of the new UI.

Unfortunately it features a super eye-catching "New Message" button that - thankfully - can be disabled. Even the whole space above the email folder pane can be recovered by disabling the folder pane header at Burger Menu (☰) -> View -> Folders -> Folder Pane Header.

Unfortunately there is no way to remove the same eye-catching "New Event" button for the Calendar view via a UI setting.

This needs a user CSS file to override the button as non-visible.

To make it process the user CSS Thunderbird needs a config setting to be enabled:

1. Burger Menu (☰) -> Settings -> General
2. Scroll down all the way
3. Click the Config editor... button on the bottom right
4. Accept that hell will freeze over because you configure software
5. Search for toolkit.legacyUserProfileCustomizations.stylesheets
6. Toggle the value to true to enable the user CSS

You can manually add user_pref("toolkit.legacyUserProfileCustomizations.stylesheets", true); to ~/.thunderbird/abcdefgh.default/prefs.js to the same effect (do this while Thunderbird is not running; replace abcdefgh with your Thunderbird profile ID).

Now create a new directory ~/.thunderbird/abcdefgh.default/chrome/, again replacing abcdefgh with your profile ID.

Inside the new directory create a userChrome.css file with the following content:

Restart Thunderbird. And enjoy less visual obstruction when using the Calendar.

Installing a laptop with the shiny new Debian Bookworm release finds a few interesting things broken that I probably had fixed in the past already on the old laptop.

One, that was increadibly unintuitive to fix, was lots of applications (like xfce4-terminal or Telegram) opening links in Chromium despite Firefox being set as the preferred webbrowser everywhere.

update-alternatives --config x-www-browser was pointing at Firefox already, of course.
The Xfce4 preferred application from settings was Firefox, of course.
xdg-mime query default text/html delivered firefox-esr.desktop, of course.

Still nearly every link opens in ███████ Chromium...

As usually the answer is out there. In this case in a xfce4-terminal bug report from 2015.

The friendly "runkharr" has debugged the issue and provides the fix as well. As usually, all very easy once you know where to look. And why to hate GTK again a bit more:

The GTK function gtk_show_uri() uses glib's g_app_info_launch_default_for_uri() and that - of course - cannot respect the usual mimetype setting.

So quoting "runkharr" verbatim:

1. Create a file `exo-launch.desktop´ in your `~/.local/share/applications´ directory with something like the following content:

    [Desktop Entry]
    Name=Exo Launcher
    Type=Application
    Icon=gtk-open
    Categories=Desktop;
    Comment=A try to force 'xfce4-terminal' to use the preferred application(s)
    GenericName=Exo Launcher
    Exec=exo-open %u
    MimeType=text/html;application/xhtml+xml;x-scheme-handler/http;x-scheme-handler/https;x-scheme-handler/ftp;application/x-mimearchive;
    Terminal=false
    OnlyShowIn=XFCE;

2. Create (if not already existing) a local `defaults.list´ file, again in your `~/.local/share/applications´ directory. This file must start with a "group header" of

    [Default Applications]

3. Insert the following three lines somewhere below this `[Default Applications]´ group header [..]:

    x-scheme-handler/http=exo-launch.desktop;
    x-scheme-handler/https=exo-launch.desktop;
    x-scheme-handler/ftp=exo-launch.desktop;

And ... links open in Firefox again. Thank you "runkharr"!

After an upgrade of my PC's mainboard BIOS the boot would take a minute or more to complete and sometimes the lightdm login screen would sit there but not accept keyboard input for another minute or so. Then the keyboard got enabled and I could log in normally. Everything worked fine after that bootup struggle completed. This was fully reproducible and persisted across reboots. Weird.

The kernel dmesg log showed entries that looked suspicious:

Googleing these error -110 and error -71 is a bit hard. Now why the USB driver does not give useful error messages instead of archaic errno-style numbers escapes me. This is not the 80s anymore.

The wisdom of the crowd says error -110 is something around "the USB port power supply was exceeded" [source].

Now lsusb -tv shows device 1-7 ... to be my USB keyboard. I somehow doubt that wants more power than the hub is willing to provide.

The Archlinux BBS Forums recommend to piece together information from drivers/usb/host/ohci.h and (updated from their piece which is from 2012) /tools/include/uapi/asm-generic/errno.h. This is why some people then consider -110 to mean "Connection timed out". Nah, not likely either.

Reading through the kernel source around drivers/usb/host did not enlighten me either. To the contrary. Uuugly. There seems to be no comprehensive list what these error codes mean. And the numbers are assigned to errors conditions quite arbitrarily. And - of course - there is no documentation. "It was hard to do, so it should be hard to understand as well."

Luckily some of the random musings I read through contained some curious advice: power cycle the host. So I did and that did not make the error go away. Other people insisted on removing cables out of wall sockets, unplugging everything and conducting esoteric rituals. That made it dawn on me, the mainboard of course nicely powers the USB in "off" state, too. So switching the power supply off (yes, these have a separate switch, go find yours), waiting a bit for capacitors to drain and switching things back on and ... the errors were gone, the system booted within seconds again.

So the takeaway message: If you get random error messages like

on devices that previously worked fine ... completely remove power from the host, the hubs and the USB devices. So they forget they saw each other on the bus before. And when they see each other after that blackout, they will happily go through negotiating protocol details with each other again successfully.

Canonical has implemented a staged roll-out for some Ubuntu package updates.

I find that rather annoying at times, e.g. when preparing the laptop for traveling.

So for my memory and for the benefit of others:

This can - for permanent use - be put into a config file, e.g. Gerrit Heim puts it into /etc/apt/apt.conf.d/99-Phased-Updates [German]. Some other options around this staged roll-out feature are "documented" within a thread in the Ubuntu discourse forum.

Thunderbird, srsly?

5MB (or 4.8MiB) import limit. Sure. My modest pubring (111 keys) is 18MB. The Debian keyring is 28MB.

May be, just may be, add another 0 to that if statement?

So, until that happens, workarounds ...

Option 1:

Export each pubkey into a separate file. The import dialog allows to select them all in one go. But - of course - it will ask confirmation for each. So prepare some valerian tea.

Option 2:

Strip all the signatures, so Thunderbird gets a smaller file to chew on. This uses pgp-clean from signing-party.

Option 1 will retain the signatures on individual keys, Option 2 will not.

The GnuPG (gpg) ecosystem has been played with a bit in 2019 by adding fake signatures en masse to well known keys. The main result is that the SKS Keyserver network based on the OCaml software of the same name is basically history. A few other keyservers have come up like Hagrid (Rust) and Hockeypuck (Go) but there seems to be no clear winner yet. In case you missed it in 2019, see my take on cleaning these polluted keys.

Now the changed defaults in gpg to "mitigate" this issue are trickling down to even the conservative distributions. Debian Bullseye has self-sigs-only on gpg 2.2.27 and it looks like Debian Bookworm will get gpg 2.2.40. This would add import-clean but Daniel Kahn Gillmor patched it out. He argues correctly that this new default could delete data from good locally stored pubkeys.

This all ends in you getting some random combination of self-sigs-only and / or import-clean depending on which Linux distribution and version you happen to use.

Better be explicit. I recommend to add:

to your ~/.gnupg/gpg.conf to make sure you can manage signatures yourself and receive them from keyservers or local imports as intended.

In case you care: See info gnupg --index-search=keyserver-options for the fine documentation. Of course apt install info first to be able to read info pages. 'cause who would still used them in 2023? Oh, wait...

Thunar, the Xfce4 file manager, has a bug that it underflows the time remaining for a file copy since ten years now (bugzilla, gitlab). Happy birthday!

Update from 01.06.2023:

Good news, Debian Bookworm will also be released with this long term friend still present. Phew!

So, 2021 isn't bad enough yet, but don't despair, people are working to fix that:

Welcome to the Stallman wars

Team Cancel: https://rms-open-letter.github.io/ (repo)

Team Support: https://rms-support-letter.github.io/ (repo)

Current Final stats are:

Team Cancel:  3019 signers from 1415 individual commit authors
Team Support: 6853 signers from 5418 individual commit authors

Git shortlog (Top 10):

rms_cancel.git (Last update: 2021-08-16 00:11:15 (UTC))
  1230  Neil McGovern
   251  Joan Touzet
    99  Elana Hashman
    73  Molly de Blanc
    36  Shauna
    19  Juke
    18  Stefano Zacchiroli
    17  Alexey Mirages
    16  Devin Halladay
    14  Nader Jafari

rms_support.git (Last update: 2021-09-29 07:14:39 (UTC))
  1821  shenlebantongying
  1585  nukeop
  1560  Ivanq
  1057  Victor
   880  Job Bautista
   123  nekonee
   101  Victor Gridnevsky
    41  Patrick Spek
    25  Borys Kabakov
    17  KIM Taeyeob

(data as of 2021-10-01)

Technical info:
Signers are counted from their "Signed / Individuals" sections. Commits are counted with git shortlog -s.
Team Cancel also has organizational signatures with Mozilla, Suse and X.Org being among the notable signatories. The 16 original signers of the Cancel petition are added in their count. Neil McGovern, Juke and shenlebantongying need .mailmap support as they have committed with different names.

Further reading:

• An introductory Ars Technica article in case you wonder what this is all about.
• Debian vote mailing-list: March 2021, April 2021
• NYT Magazine on the history of cancel culture
• Ed Santos' commentary and analysis

12.04.2021 Statements from the accused

• Statement of the FSF board on the election
• Statement of Richard Stallman

18.04.2021 Debian General Resolution

The Debian General Resolution (GR) vote of the developers has concluded to not issue a public statement at all, see https://www.debian.org/vote/2021/vote_002#outcome for the results.

It is better to keep quiet and seem ignorant than to speak up and remove all doubt.

See Quote Investigator for the many people that rephrased these words over the centuries. They still need to be recalled more often as too many people in the FLOSS community have forgotten about that wisdom...

01.10.2021 Final stats

It seems enough dust has settled on this unfortunate episode of mob activity now. Hence I stopped the cronjob that updated the stats above regularly. Team Support has kept adding signature all the time while Team Cancel gave up very soon after the FSF decided to stand with Mr. Stallman. So this battle was decided within two months. The stamina of the accused and determined support from some dissenting web devs trumped the orchestrated outrage of well known community figures and their publicity power this time. But history teaches us that does not mean the war is over. There will a the next opportunity to call for arms. And people will call. Unfortunately.

01.11.2024 Team Cancel is opening a new round; Team Support responds with exposing the author of "The Stallman report"

I hate to be right. Three years later than the above:

An anonymous member of team Cancel has published https://stallman-report.org/ [local pdf mirror, 504kB] to "justify our unqualified condemnation of Richard Stallman". It contains a detailed collection of quotes that are used to allege supporting (sexual) misconduct. The demand is again that Mr. Stallman "step[s] down from all positions at the FSF and the GNU project". Addressing him: "the scope and extent of your misconduct disqualifies you from formal positions of power within our community indefinitely".

Team Support has not issues a rebuttal (yet?) but has instead identified the anonymous author as Drew "sircmpwn" DeVault, a gifted software developer, but also a vocal and controversial figure in the Open Source / Free Software space. Ironically quite similar to Richard "rms" Stallman. Their piece is published at https://dmpwn.info/ [local pdf mirror, 929kB]. They also allege a proximity of Mr. DeVault to questionable "Lolita" anime preferences and societal positions to disqualify him.

The first install of a Gentoo kernel needs to be somewhat manual if you want to optimize the kernel for the (virtual) system it boots on.

In part 1 I laid out how to improve the subsequent emerges of sys-kernel/gentoo-sources with a small drop in script to build the kernel as part of the ebuild.

Since end of last year Gentoo also supports a less manual way of emerging a kernel:

The following kernel blends are available:

• sys-kernel/gentoo-kernel (the Gentoo kernel you can configure and compile locally - typically this is what you want if you run Gentoo)
• sys-kernel/gentoo-kernel-bin (a pre-compiled Gentoo kernel similar to what genkernel would get you)
• sys-kernel/vanilla-kernel (the upstream Linux kernel, again configurable and locally compiled)

So a quick walk-through for the gentoo-kernel variant:

1. Set up the correct package USE flags

We do not want an initrd and we want our own config to be re-used so:

2. Preseed the saved config

The current kernel config needs to be saved as the initial savedconfig so it is found and applied for our emerge below:

3. Emerge the new kernel

4. Update grub and reboot

Unfortunately this ebuild does not update grub, so we have to run grub-mkconfig manually. This can again be automated via a post_pkg_postinst() script. See the step 7 below.

But for now, let's do it manually:

5. (Optional) Prepare for the next kernel build

Run etc-update and merge the new kernel config entries into your savedconfig.

The kernel should auto-build once new versions become available via portage.

Again the etc-update can be automated if you feel that is sufficiently safe to do in your environment. See step 7 below for details.

6. (Optional) Remove the old kernel sources

If you want to switch from the method based on gentoo-sources to the gentoo-kernel one, you can remove the kernel sources:

Be sure to update the /usr/src/linux symlink to the new kernel sources directory from gentoo-kernel, e.g.:

This may be a good time for a bit more house-keeping: Clean up a bit in /usr/src/ to remove old build artefacts, /boot/ to remove old kernels and /lib/modules/ to get rid of old kernel modules.

7. (Optional) Further automate the ebuild

In part 1 we automated the kernel compile, install and a bit more via a helper function for post_pkg_postinst().

We can do the similarly for what is (currently) missing from the gentoo-kernel ebuilds:

Create /etc/portage/env/sys-kernel/gentoo-kernel with the following:

The upside of gentoo-kernel over gentoo-sources is that you can put "config override files" in /etc/kernel/config.d/. That way you theoretically profit from config improvements made by the upstream developers. See the Gentoo distribution kernel documentation for a sample snippet. I am fine with savedconfig for now but it is nice that Gentoo provides the flexibility to support both approaches.

Gentoo emerges of sys-kernel/gentoo-sources will nicely install the current kernel into /usr/src/linux-* but it will not compile them.

The Gentoo wiki kernel documentation has a script snippet to automate the kernel build with genkernel.

I do not like to use genkernel as it brings in lots of firmware files to build initrds that are not needed on virtual hardware. It also makes building the kernel slower.

So, the plain approach:

Make emerge sys-kernel/gentoo-sources symlink the latest kernel to /usr/src/linux so we can find it easily:

Create /etc/portage/env/sys-kernel/gentoo-sources with the following:

This will compile the next kernel on the basis of the config of the currently running kernel, install the modules and the kernel bzImage and update grub so it knows about the new kernel for the next reboot.

If you forget to unset ARCH the Linux build system will complain like:

Makefile:583: arch/amd64/Makefile: No such file or directory
make: *** No rule to make target 'arch/amd64/Makefile'.  Stop.

You can test the new magic by re-emerging the latest kernel, e.g. currently emerge =sys-kernel/gentoo-sources-5.4.80-r1:

System Rescue, the project formerly known as System Rescue CD, has moved from being based on Gentoo to being built on Arch Linux packages.

With this their ISO layout changed substantially so when updating my trusty recue USB flash drive, I could not just update the kernel, initrd and the root filesystem image as I had typically done every other year before.

The "Installing on a USB memory stick" documentation is good for Windows (use Rufus, it's nice) but rather useless for Linux. They recommend a dd or the fancy graphical version of that, called usbimager.

I much prefer to have a flash drive that I can write to over an image of a CD (ISO) written 1:1 onto the flash media.

The basic idea is to use the bulk of the System Rescue ISO contents but amend these with your own grub and syslinux so they work as intended over the supplied ones that are bound to the ISO layout a bit too much.

I did this on Debian Buster but with some adjustments to paths and what packages to install, any recent Linux distribution should do:

Limesurvey is an online survey tool. It is very powerful and commonly used in academic environments because it is Free Software (GPLv2+), allows for local installations protecting the data of participants and allowing to comply with data protection regulations. This also means there are typically no load-balanced multi-server szenarios with HA databases. But simple VMs where Limesurvey runs and needs upgrading in place.

There's an LTS branch (currently 3.x) and a stable branch (currently 4.x). There's also a 2.06 LTS branch that is restricted to paying customers. The main developers behind Limesurvey offer many services from template design to custom development to support to hosting ("Cloud", "Limesurvey Pro"). Unfortunately they also charge for easy updates called "ComfortUpdate" (currently 39€ for three months) and the manual process is made a bit cumbersome to make the "ComfortUpdate" offer more attractive.

Due to Limesurvey being an old code base and UI elements not being clearly separated, most serious use cases will end up patching files and symlinking logos around template directories. That conflicts a bit with the opaque "ComfortUpdate" process where you push a button and then magic happens. Or you have downtime and a recovery case while surveys are running.

If you do not intend to use the "ComfortUpdate" offering, you can prevent Limesurvey from connecting to http://comfortupdate.limesurvey.org daily by adding the updatable stanza as in line 14 to limesurvey/application/config/config.php:

The comma on line 13 is placed like that in the current default limesurvey config.php, don't let yourself get confused. Every item in a php array must end with a comma. It can be on the next line.

The basic principle of low risk, near-zero downtime, in-place upgrades is:

1. Create a diff between the current release and the target release
2. Inspect the diff
3. Make backups of the application webroot
4. Patch a copy of the application in-place
5. (optional) stop the web server
6. Make a backup of the production database
7. Move the patched application to the production webroot
8. (if 5) Start the webserver
9. Upgrade the database (if needed)
10. Check the application

So, in detail:

Wiping hard disks is part of my company's policy when returning servers. No exceptions.

Good providers will wipe what they have received back from a customer, but we don't trust that as the hosting / cloud business is under constant budget-pressure and cutting corners (wipefs) is a likely consequence.

With modern SSDs there is "security erase" (man hdparm or see the - as always well maintained - Arch wiki) which is useful if the device is encrypt-by-default. These devices basically "forget" the encryption key but it also means trusting the devices' implementation security. Which doesn't seem warranted. Still after wiping and trimming, a secure erase can't be a bad idea .

Still there are three things to be aware of when wiping modern hard disks:

1. Don't forget to add bs=4096 (blocksize) to dd as it will still default to 512 bytes and that makes writing even zeros less than half the maximum possible speed. SSDs may benefit from larger block sizes matched to their flash page structure. These are usually 128kB, 256kB, 512kB, 1MB, 2MB and 4MB these days.¹
2. All disks can usually be written to in parallel. screen is your friend.
3. The write speed varies greatly by disk region, so use 2 hours per TB and wipe pass as a conservative estimate. This is better than extrapolating what you see initially in the fastest region of a spinning disk.
4. The disks have become huge (we run 12TB disks in production now) but the write speed is still somewhere 100 MB/s ... 300 MB/s. So wiping servers on the last day before returning is not possible anymore with disks larger than 4 TB each (and three passes). Or 12 TB and one pass (where e.g. fully encrypted content allows to just do a final zero-wipe).