Daniel Lange's blog

Canonical has implemented a staged roll-out for some Ubuntu package updates.

I find that rather annoying at times, e.g. when preparing the laptop for traveling.

So for my memory and for the benefit of others:

This can - for permanent use - be put into a config file, e.g. Gerrit Heim puts it into /etc/apt/apt.conf.d/99-Phased-Updates [German]. Some other options around this staged roll-out feature are "documented" on a thread in the Ubuntu discourse forum.

Thunderbird, srsly?

5MB (or 4.8MiB) import limit. Sure. My modest pubring (111 keys) is 18MB. The Debian keyring is 28MB.

May be, just may be, add another 0 to that if statement?

So, until that happens, workarounds ...

Option 1:

Export each pubkey into a separate file. The import dialog allows to select them all in one go. But - of course - it will ask confirmation for each. So prepare some valerian tea.

Option 2:

Strip all the signatures, so Thunderbird gets a smaller file to chew on. This uses pgp-clean from signing-party.

Option 1 will retain the signatures on individual keys, Option 2 will not.

The GnuPG (gpg) ecosystem has been played with a bit in 2019 by adding fake signatures en masse to well known keys. The main result is that the SKS Keyserver network based on the OCaml software of the same name is basically history. A few other keyservers have come up like Hagrid (Rust) and Hockeypuck (Go) but there seems to be no clear winner yet. In case you missed it in 2019, see my take on cleaning these polluted keys.

Now the changed defaults in gpg to "mitigate" this issue are trickling down to even the conservative distributions. Debian Bullseye has self-sigs-only on gpg 2.2.27 and it looks like Debian Bookworm will get gpg 2.2.40. This would add import-clean but Daniel Kahn Gillmor patched it out. He argues correctly that this new default could delete data from good locally store pubkeys.

This all ends in you getting some random combination of self-sigs-only and / or import-clean depending on which Linux distribution and version you happen to use.

Better be explicit. I recommend to add:

to your ~/.gnupg/gpg.conf to make sure you can manage signatures yourself and receive them from keyservers or local imports as intended.

In case you care: See info gnupg --index-search=keyserver-options for the fine documentation. Of course apt install info first to be able to read info pages. 'cause who still used them in 2023? Oh, wait...

Thunar, the Xfce4 file manager, has a bug that it underflows the time remaining for a file copy since ten years now (bugzilla, gitlab). Happy birthday!

There is significant effort involved to get your apt or dnf commands always have a consistent set of servers to talk to.

That's why running separate "software stores" is a bad idea:

That way more admins need to learn how to run high availability services for dubious business opportunities to "later" monetize services. Services that nobody cares to pay for and thus opportunities that never materialize. But every company wants to find that out again. Because if Apple could do it, why shouldn't Canonical be able to do it? $$$!1!!

So, can't update Firefox on Ubuntu 22.04 right now.

At least there is https://status.snapcraft.io/:

So I can check back tomorrow if I can update my web browser ...

Update

09.11.2022 12:15 CET

The Snapcraft distribution system seems quite flaky, this is the downtime log:

Bonus points for the bad client side implementation:

Postscriptum

GNOME and KDE join forces to sink another 100 .. 200k USD into the void:

https://github.com/PlaintextGroup/oss-virtual-incubator/[..]/proposals/flathub-linux-app-store.md

This is an application for funding from Schmidt Futures, which is one of the investment (as in philanthropic) funds from Eric Schmidt, former Google CEO.

The application text is worth reading. Very entertaining.

Robert McQueen (GNOME, Flathub) wrote on 07.03.2023 that the PlaintextGroup/Schmidt Futures application was denied for 2023.

The company got me a Chromebook for times when I want to ignore email and not get dragged into code reviews but still be available on IRC and chat. Nifty devices at great price points. I really like them.

These things are meant to be very consumer-style end-user devices. You log in with your Google account and everything works. Until it doesn't.

Just setting it up caused the first issue:

I was always thrown back to a black screen and then another login-screen despite having successfully logged in initially to create the "owner" user of the Chromebook. No error message, not useful UI feedback. Just logging in again and again and again.

The issue is ... not having a GMail account associated with my Google account. Duh! So add a GMail.com address as the primary to your Google account and the initial setup completes. Of course you cannot delete that GMail.com association again because the owner user is linked to the email and not the account. Well, you can delete it but then you cannot configure "owner" items of your Chromebook any more. Great job, Google. Not. Identity management 101 fail.

Kudos to Anurag Chawake for blogging about the issue. The Google support forum thead claims this is solved now. But it didn't work for me, so this may be needing to trickle down through ChromeOS releases or be deployed on more Google infra. Or whatever. We can't tell from outside the Googleplex as - of course - "Rebecca" sheds no light on what the identified "root cause" was:

Once I was able to login to the new Chromebook all worked fine until I started to use ssh sessions. These always hung for 30 seconds to 10 minutes and then resumed with lots of packets lost in between and the last minute or so coming in from buffering in a burst.

This was easy to see in ping as well. The connection essentially dropped dead while the WiFi icon was continuing to show full signal strength. The logs did not show anything useful. These are really hard to access on ChromeOS (JSON format and no useful UI on the Chromebook itself, Google provides a viewer on Google Apps Toolbox but that requires uploading the logs). Better than no logs at all but not really nice.

The ChromeOS bug tracker and its Google corporate counterpart are also not useful at this time.

For reference:
Google ... Device users randomly disconnect from Wi-Fi network
Chromium ... Device users randomly disconnect from Wi-Fi network
Google ... Constant connect and disconnect from WiFi source post-update
Chromium ... Constant connect and disconnect from WiFi source post-update

Playing around with the device on the network showed that it reduced sending power beyond being able to reach the access point any more. This is why disconnecting and re-connecting the WiFi fixes the issue for a few minutes, typically.

Still, there is a better way:

In crosh (the ChromeOS shell available when pressing Alt+Ctrl+T) type:

wifi_power_save disable

This unfortunately only lasts until the next reboot but it does persist over suspend cycles (aka closing the lid).

So, 2021 isn't bad enough yet, but don't despair, people are working to fix that:

Welcome to the Stallman wars

Team Cancel: https://rms-open-letter.github.io/ (repo)

Team Support: https://rms-support-letter.github.io/ (repo)

Current Final stats are:

Team Cancel:  3019 signers from 1415 individual commit authors
Team Support: 6853 signers from 5418 individual commit authors

Git shortlog (Top 10):

rms_cancel.git (Last update: 2021-08-16 00:11:15 (UTC))
  1230  Neil McGovern
   251  Joan Touzet
    99  Elana Hashman
    73  Molly de Blanc
    36  Shauna
    19  Juke
    18  Stefano Zacchiroli
    17  Alexey Mirages
    16  Devin Halladay
    14  Nader Jafari

rms_support.git (Last update: 2021-09-29 07:14:39 (UTC))
  1821  shenlebantongying
  1585  nukeop
  1560  Ivanq
  1057  Victor
   880  Job Bautista
   123  nekonee
   101  Victor Gridnevsky
    41  Patrick Spek
    25  Borys Kabakov
    17  KIM Taeyeob

(data as of 2021-10-01)

Technical info:
Signers are counted from their "Signed / Individuals" sections. Commits are counted with git shortlog -s.
Team Cancel also has organizational signatures with Mozilla, Suse and X.Org being among the notable signatories. The 16 original signers of the Cancel petition are added in their count. Neil McGovern, Juke and shenlebantongying need .mailmap support as they have committed with different names.

Further reading:

• An introductory Ars Technica article in case you wonder what this is all about.
• Debian vote mailing-list: March 2021, April 2021
• NYT Magazine on the history of cancel culture
• Ed Santos' commentary and analysis

12.04.2021 statements from the accused:

• Statement of the FSF board on the election
• Statement of Richard Stallman

18.04.2021 Debian General Resolution

The Debian General Resolution (GR) vote of the developers has concluded to not issue a public statement at all, see https://www.debian.org/vote/2021/vote_002#outcome for the results.

It is better to keep quiet and seem ignorant than to speak up and remove all doubt.

See Quote Investigator for the many people that rephrased these words over the centuries. They still need to be recalled more often as too many people in the FLOSS community have forgotten about that wisdom...

01.10.2021 Final stats

It seems enough dust has settled on this unfortunate episode of mob activity now. Hence I stopped the cronjob that updated the stats above regularly. Team Support has kept adding signature all the time while Team Cancel gave up very soon after the FSF decided to stand with Mr. Stallman. So this battle was decided within two months. The stamina of the accused and determined support from some dissenting web devs trumped the orchestrated outrage of well known community figures and their publicity power this time. But history teaches us that does not mean the war is over. There will a the next opportunity to call for arms. And people will call. Unfortunately.

The first install of a Gentoo kernel needs to be somewhat manual if you want to optimize the kernel for the (virtual) system it boots on.

In part 1 I laid out how to improve the subsequent emerges of sys-kernel/gentoo-sources with a small drop in script to build the kernel as part of the ebuild.

Since end of last year Gentoo also supports a less manual way of emerging a kernel:

The following kernel blends are available:

• sys-kernel/gentoo-kernel (the Gentoo kernel you can configure and compile locally - typically this is what you want if you run Gentoo)
• sys-kernel/gentoo-kernel-bin (a pre-compiled Gentoo kernel similar to what genkernel would get you)
• sys-kernel/vanilla-kernel (the upstream Linux kernel, again configurable and locally compiled)

So a quick walk-through for the gentoo-kernel variant:

1. Set up the correct package USE flags

We do not want an initrd and we want our own config to be re-used so:

2. Preseed the saved config

The current kernel config needs to be saved as the initial savedconfig so it is found and applied for our emerge below:

3. Emerge the new kernel

4. Update grub and reboot

Unfortunately this ebuild does not update grub, so we have to run grub-mkconfig manually. This can again be automated via a post_pkg_postinst() script. See the step 7 below.

But for now, let's do it manually:

5. (Optional) Prepare for the next kernel build

Run etc-update and merge the new kernel config entries into your savedconfig.

The kernel should auto-build once new versions become available via portage.

Again the etc-update can be automated if you feel that is sufficiently safe to do in your environment. See step 7 below for details.

6. (Optional) Remove the old kernel sources

If you want to switch from the method based on gentoo-sources to the gentoo-kernel one, you can remove the kernel sources:

Be sure to update the /usr/src/linux symlink to the new kernel sources directory from gentoo-kernel, e.g.:

This may be a good time for a bit more house-keeping: Clean up a bit in /usr/src/ to remove old build artefacts, /boot/ to remove old kernels and /lib/modules/ to get rid of old kernel modules.

7. (Optional) Further automate the ebuild

In part 1 we automated the kernel compile, install and a bit more via a helper function for post_pkg_postinst().

We can do the similarly for what is (currently) missing from the gentoo-kernel ebuilds:

Create /etc/portage/env/sys-kernel/gentoo-kernel with the following:

The upside of gentoo-kernel over gentoo-sources is that you can put "config override files" in /etc/kernel/config.d/. That way you theoretically profit from config improvements made by the upstream developers. See the Gentoo distribution kernel documentation for a sample snippet. I am fine with savedconfig for now but it is nice that Gentoo provides the flexibility to support both approaches.

Gentoo emerges of sys-kernel/gentoo-sources will nicely install the current kernel into /usr/src/linux-* but it will not compile them.

The Gentoo wiki kernel documentation has a script snippet to automate the kernel build with genkernel.

I do not like to use genkernel as it brings in lots of firmware files to build initrds that are not needed on virtual hardware. It also makes building the kernel slower.

So, the plain approach:

Make emerge sys-kernel/gentoo-sources symlink the latest kernel to /usr/src/linux so we can find it easily:

Create /etc/portage/env/sys-kernel/gentoo-sources with the following:

This will compile the next kernel on the basis of the config of the currently running kernel, install the modules and the kernel bzImage and update grub so it knows about the new kernel for the next reboot.

If you forget to unset ARCH the Linux build system will complain like:

Makefile:583: arch/amd64/Makefile: No such file or directory
make: *** No rule to make target 'arch/amd64/Makefile'.  Stop.

You can test the new magic by re-emerging the latest kernel, e.g. currently emerge =sys-kernel/gentoo-sources-5.4.80-r1:

System Rescue, the project formerly known as System Rescue CD, has moved from being based on Gentoo to being built on Arch Linux packages.

With this their ISO layout changed substantially so when updating my trusty recue USB flash drive, I could not just update the kernel, initrd and the root filesystem image as I had typically done every other year before.

The "Installing on a USB memory stick" documentation is good for Windows (use Rufus, it's nice) but rather useless for Linux. They recommend a dd or the fancy graphical version of that, called usbimager.

I much prefer to have a flash drive that I can write to over an image of a CD (ISO) written 1:1 onto the flash media.

The basic idea is to use the bulk of the System Rescue ISO contents but amend these with your own grub and syslinux so they work as intended over the supplied ones that are bound to the ISO layout a bit too much.

I did this on Debian Buster but with some adjustments to paths and what packages to install, any recent Linux distribution should do:

Show https://github.com/torvalds/linux/blob/b4061a10fc29010a610ff2b5b20160d7335e69bf/drivers/hid/hid-samsung.c#L113-L118 to a friend.

Oops 'eh? Yep, Linux has been backdoored.

Well, or not.

Konstantin Ryabitsev explains it nicely in a cgit mailing list email:

It is common for git hosting environments to configure all forks of the same repo to use an "object storage" repository. For example, this is what allows git.kernel.org's 600+ forks of linux.git to take up only 10GB on disk as opposed to 800GB. One of the side-effects of this setup is that any object in the shared repository can be accessed from any of the forks, which periodically confuses people into believing that something terrible has happened.

The hack was discussed on Github in Dec 2018 when it was discovered. I forgot about it again but Konstantin's mail brought the memory back and I think it deserves more attention.

I'm sure putting some illegal content into a fork and sending a made up "blob" URL to law enforcement would go quite far. Good luck explaining the issue. "Yes this is my repo" but "no, no that's not my data" ... "yes, it is my repo but not my data" ... "no we don't want that data either, really" ... "but, but there is nothing we can do, we host on github...¹".

Updates

05.11.20 Nate Friedman (CEO of Github) promises

[..] we are going to make it much more obvious when you're viewing an orphaned commit.

For context: The source code of Github (the product) had been leaked as a commit to Github's own DMCA repository. The repository has turned into a playground since Github took down the hosting for youtube-dl as the result of a DMCA complaint.

14.11.20 Seems Github now adds a warning to commits that are not in a reachable branch

28.01.22 Github currently fails to show the warning message, so https://github.com/torvalds/linux/tree/8bcab0346d4fcf21b97046eb44db8cf37ddd6da0 is making rounds now:

Limesurvey is an online survey tool. It is very powerful and commonly used in academic environments because it is Free Software (GPLv2+), allows for local installations protecting the data of participants and allowing to comply with data protection regulations. This also means there are typically no load-balanced multi-server szenarios with HA databases. But simple VMs where Limesurvey runs and needs upgrading in place.

There's an LTS branch (currently 3.x) and a stable branch (currently 4.x). There's also a 2.06 LTS branch that is restricted to paying customers. The main developers behind Limesurvey offer many services from template design to custom development to support to hosting ("Cloud", "Limesurvey Pro"). Unfortunately they also charge for easy updates called "ComfortUpdate" (currently 39€ for three months) and the manual process is made a bit cumbersome to make the "ComfortUpdate" offer more attractive.

Due to Limesurvey being an old code base and UI elements not being clearly separated, most serious use cases will end up patching files and symlinking logos around template directories. That conflicts a bit with the opaque "ComfortUpdate" process where you push a button and then magic happens. Or you have downtime and a recovery case while surveys are running.

If you do not intend to use the "ComfortUpdate" offering, you can prevent Limesurvey from connecting to http://comfortupdate.limesurvey.org daily by adding the updatable stanza as in line 14 to limesurvey/application/config/config.php:

The comma on line 13 is placed like that in the current default limesurvey config.php, don't let yourself get confused. Every item in a php array must end with a comma. It can be on the next line.

The basic principle of low risk, near-zero downtime, in-place upgrades is:

1. Create a diff between the current release and the target release
2. Inspect the diff
3. Make backups of the application webroot
4. Patch a copy of the application in-place
5. (optional) stop the web server
6. Make a backup of the production database
7. Move the patched application to the production webroot
8. (if 5) Start the webserver
9. Upgrade the database (if needed)
10. Check the application

So, in detail:

I've long said that the main tools in the Open Source security space, OpenSSL and GnuPG (gpg), are broken and only a complete re-write will solve this. And that is still pending as nobody came forward with the funding. It's not a sexy topic, so it has to get really bad before it'll get better.

Gpg has a UI that is close to useless. That won't substantially change with more bolted-on improvements.

Now Robert J. Hansen and Daniel Kahn Gillmor had somebody add ~50k signatures (read 1, 2, 3, 4 for the g{l}ory details) to their keys and - oops - they say that breaks gpg.

But does it?

I downloaded Robert J. Hansen's key off the SKS-Keyserver network. It's a nice 45MB file when de-ascii-armored (gpg --dearmor broken_key.asc ; mv broken_key.asc.gpg broken_key.gpg).

Now a friendly:

And the result is a nicely useable 3835 byte file of the clean public key. If you supply a keyring instead of --no-default-keyring it will also keep the non-self signatures that are useful for you (as you apparently know the signing party).

So it does not break gpg. It does break things that call gpg at runtime and not asynchronously. I heard Enigmail is affected, quelle surprise.

Now the main problem here is the runtime. 1h45min is just ridiculous. As Filippo Valsorda puts it:

Someone added a few thousand entries to a list that lets anyone append to it. GnuPG, software supposed to defeat state actors, suddenly takes minutes to process entries. How big is that list you ask? 17 MiB. Not GiB, 17 MiB. Like a large picture. https://dev.gnupg.org/T4592

If I were a gpg / SKS keyserver developer, I'd

• speed this up so the edit-key run above completes in less than 10 s (just getting rid of the lseek/read dance and deferring all time-based decisions should get close)
• (ideally) make the drop-sig import-filter syntax useful (date-ranges, non-reciprocal signatures, ...)
• clean affected keys on the SKS keyservers (needs coordination of sysops, drop servers from unreachable people)
• (ideally) use the opportunity to clean all keyserver filesystem and the message board over pgp key servers keys, too
• only accept new keys and new signatures on keys extending the strong set (rather small change to the existing codebase)

That way another key can only be added to the keyserver network if it contains at least one signature from a previously known strong-set key. Attacking the keyserver network would become at least non-trivial. And the web-of-trust thing may make sense again.

Updates

09.07.2019

GnuPG 2.2.17 has been released with another set of quickly bolted together fixes:

  * gpg: Ignore all key-signatures received from keyservers.  This
    change is required to mitigate a DoS due to keys flooded with
    faked key-signatures.  The old behaviour can be achieved by adding
    keyserver-options no-self-sigs-only,no-import-clean
    to your gpg.conf.  [#4607]
  * gpg: If an imported keyblocks is too large to be stored in the
    keybox (pubring.kbx) do not error out but fallback to an import
    using the options "self-sigs-only,import-clean".  [#4591]
  * gpg: New command --locate-external-key which can be used to
    refresh keys from the Web Key Directory or via other methods
    configured with --auto-key-locate.
  * gpg: New import option "self-sigs-only".
  * gpg: In --auto-key-retrieve prefer WKD over keyservers.  [#4595]
  * dirmngr: Support the "openpgpkey" subdomain feature from
    draft-koch-openpgp-webkey-service-07. [#4590].
  * dirmngr: Add an exception for the "openpgpkey" subdomain to the
    CSRF protection.  [#4603]
  * dirmngr: Fix endless loop due to http errors 503 and 504.  [#4600]
  * dirmngr: Fix TLS bug during redirection of HKP requests.  [#4566]
  * gpgconf: Fix a race condition when killing components.  [#4577]

Bug T4607 shows that these changes are all but well thought-out. They introduce artificial limits, like 64kB for WKD-distributed keys or 5MB for local signature imports (Bug T4591) which weaken the web-of-trust further.

I recommend to not run gpg 2.2.17 in production environments without extensive testing as these limits and the unverified network traffic may bite you. Do validate your upgrade with valid and broken keys that have segments (packet groups) surpassing the above mentioned limits. You may be surprised what gpg does. On the upside: you can now refresh keys (sans signatures) via WKD. So if your buddies still believe in limiting their subkey validities, you can more easily update them bypassing the SKS keyserver network. NB: I have not tested that functionality. So test before deploying.

10.08.2019

Christopher Wellons (skeeto) has released his pgp-poisoner tool. It is a go program that can add thousands of malicious signatures to a GNUpg key per second. He comments "[pgp-poisoner is] proof that such attacks are very easy to pull off. It doesn't take a nation-state actor to break the PGP ecosystem, just one person and couple evenings studying RFC 4880. This system is not robust." He also hints at the next likely attack vector, public subkeys can be bound to a primary key of choice.

Wiping hard disks is part of my company's policy when returning servers. No exceptions.

Good providers will wipe what they have received back from a customer, but we don't trust that as the hosting / cloud business is under constant budget-pressure and cutting corners (wipefs) is a likely consequence.

With modern SSDs there is "security erase" (man hdparm or see the - as always well maintained - Arch wiki) which is useful if the device is encrypt-by-default. These devices basically "forget" the encryption key but it also means trusting the devices' implementation security. Which doesn't seem warranted. Still after wiping and trimming, a secure erase can't be a bad idea .

Still there are three things to be aware of when wiping modern hard disks:

1. Don't forget to add bs=4096 (blocksize) to dd as it will still default to 512 bytes and that makes writing even zeros less than half the maximum possible speed. SSDs may benefit from larger block sizes matched to their flash page structure. These are usually 128kB, 256kB, 512kB, 1MB, 2MB and 4MB these days.¹
2. All disks can usually be written to in parallel. screen is your friend.
3. The write speed varies greatly by disk region, so use 2 hours per TB and wipe pass as a conservative estimate. This is better than extrapolating what you see initially in the fastest region of a spinning disk.
4. The disks have become huge (we run 12TB disks in production now) but the write speed is still somewhere 100 MB/s ... 300 MB/s. So wiping servers on the last day before returning is not possible anymore with disks larger than 4 TB each (and three passes). Or 12 TB and one pass (where e.g. fully encrypted content allows to just do a final zero-wipe).

Update from 28.04.2022: Do not use the packages below any more. There is Netatalk 3.1.13 out with fixes for multiple remote code execution (RCE) bugs. Use packages from recent Debian again, they have been updated.

Netatalk 3.1.12 has been released which fixes an 18 year old RCE bug. The Medium write up on CVE-2018-1160 by Jacob Baines is quite an entertaining read.

The full release notes for 3.1.12 are unfortunately not even half as interesting.

Be sure to read the original blog post if you are new to Netatalk3 on Debian Jessie or Stretch! You'll get nowhere if you install the .debs below and don't know about the upgrade path from 2.2.x which is still in the Debian archive. So RTFA.

For Debian Buster (Debian 10) we'll have Samba 4.9 which has learnt (from Samba 4.8.0 onwards) how to emulate a SMB time machine share. I'll make a write up how to install this once Buster stabilizes. This luckily means there will be no need to continue supporting Netatalk in normal production environments. So I guess bug #690227 won't see a proper fix anymore. Waiting out problems helps at times, too :/.

Update instructions and downloads: