Daniel Lange's blog

If you see Thunderbird hanging during startup for a minute and then continuing to load fine, you are probably running into an issue similar to what I saw when Debian migrated Icedove back to the "official" Mozilla Thunderbird branding and changed ~/.icedove to ~/.thunderbird in the process (one symlinked to the other).

Looking at the console log (=start Thunderbird from a terminal so you see its messages), I got:

console.log: foxclocks.bootstrap._loadIntoWindow(): got xul-overlay-merged - waiting for overlay-loaded
[.. one minute delay ..]
console.log: foxclocks.bootstrap._windowListener(): got window load chrome://global/content/commonDialog.xul

Stracing confirms it hangs because Thunderbird loops waiting for a FUTEX until that apparently gets kicked by a XUL core timeout.
(Thanks for defensive programming folks!)

So in my case uninstalling the Add-On Foxclocks easily solved the problem.

I assume other Thunderbird Add-Ons may cause the same issue, hence the more generic description above.

Last year I bought the wonderful Dell XPS 13 9360 as it is certified to work with Ubuntu Linux and is just all around an awesome device. Dell made me buy the Windows version as only that got a 1 TB NVMe-SSD option. Linux apparently is only worthy of the 512GB and below models. What product manager comes up with such a stupid idea? Are SKUs that precious? Anyways ... so I bought a Windows version and that got wiped with a Linux install immediately as that was and is its intended purpose.

I purchased a DA200 with the system which is Dell's USB-C to anything (HDMI/VGA/Ethernet/USB 3.0) dongle. When I got the laptop the Ethernet port and USB 3.0 via the DA200 were working right out of the box. The VGA and HDMI ports were detected by Ubuntu but there was no way to get connected screens working. They stayed black.

The device was shipped with Thunderbolt firmware NVM18 and we've been told rather quickly by Dell this would be fixed with an update. And lo and behold Dell published the firmware version NVM21 right for Christmas 2016. Now unfortunately while their BIOS updates are Windows / DOS executables that can be just shoved at the Dell UEFI flash updater and thus the main BIOS can be updated from any OS, including Linux, without any hassle, the Intel provided Thunderbolt update needs Windows to get installed. Or, well, there is a convoluted way to compile an out-of-tree Linux kernel module, download and compile a few sets of software and do it via Linux. That description read so lengthy, I didn't even try it. Additionally there seems to have been no progress at all in getting this more mainline in the last three months, so I chose the cheap route and installed Windows 10 on a USB thumb drive¹.

This is done via the (unfortunately Windows only) Win2USB software (the free version is sufficient).

Update: There's a new bash script windows2usb that looks good and should work to get you a bootable Windows USB thumb drive in Linux. WinUSB (that stopped working in the Win10 area some time) has also been forked and updated into WoeUSB. And there is WinToUSBLinux, yet another shell script. Give them a try.

Once Windows has rebooted often enough to finish its own installation, you can work with the USB thumb drive install as with any Windows 10. Nice.

Put all the files you downloaded from Dell to update your XPS 13 into a directory on the USB thumb drive. That way Windows does not need to have any network connectivity.

I first updated the TPM 1.2 firmware to a TPM 2.0 version (DellTpm2.0_Fw1.3.2.8_V1_64.exe at the time of writing this blog entry). Now this is quite hilarious as the Windows installer doesn't do anything but putting a UEFI firmware update into the EFI partition that runs on reboot. Duh. You do need to manually clear the TPM in the BIOS' security settings section (there's a clear checkbox) to be able to program new firmware onto it.

Now back in Windows install the Thunderbolt drivers (Chipset_Driver_J95RR_WN32_16.2.55.275_A01.exe at the time of writing this) and then run Intel_TBT3_FW_UPDATE_NVM21_0THFT_A00_3.21.00.008.exe, which is the NVM21 Thunderbolt firmware update (or a later version).

Reboot again (into Linux if you want to) and (drumroll) the VGA and HDMI ports are working. Awesome.

An update log can be found on the USB thumb drive at Dell\UpdatePackage\Log\Intel_TBT3_FW_UPDATE_NVM21_0THFT_A00_3.log:

*** Dell Thunderbolt firmware update started on 4/6/2017 at 12:56:56***
Command: C:\Install\Intel_TBT3_FW_UPDATE_NVM21_0THFT_A00_3.21.00.008.exe 

Starting FW Update....
***TBT GPIO Power is Turning On:  No Dock or DockInfo.
***TBT GPIO power is turned on.

Thunderbolt Firmware Update SUCCEEDED
TBT Items Registry creation is Success at \SOFTWARE\Dell\ManageableUpdatePackage\Thunderbolt Controller:
User selected OK for reboot
System TBT NVM Current Version:BCD:00000018: New Version:BCD:00000021

Exit Code = 0 (Success) 
***Thunderbolt Firmware flash finished at 4/6/2017 at 13:00:23***

If Windows has added its boot loader entry into your UEFI options, you can easily remove that again with the Dell UEFI BIOS or efibootmgr from within Linux.

The whole process took me less than 30 minutes. And most of that was creating the Windows USB thumb drive. I'll keep that for future updates until Intel and Dell have sorted out the Thunderbolt update process in Linux.

Updates:

18.05.17: Intel has published a large patchset on LKML to enable Thunderbolt security levels (thus preventing DMA attacks) and get NVM firmware upgrades mainlined. Yeah!

02.05.18: Added a link to the windows2usb bash script that should remove the need to create a bootable Windows USB thumb drive with a Windows only software.

11.06.18: Added a link to WoeUSB which is currently packaged for Ubuntu in a PPA.

08.04.20: Added a link to WinToUSBLinux. A recently released shell script to create a bootable Windows USB stick from Linux.

One thing I love about FreeBSD is the way the core team keeps the wider community updated about project news e.g. via their quarterly status reports. So while reading the FreeBSD Q4/2016 status report, I was quite surprised to find that a text filter converting English to "Jive speak" had been removed from the ports tree. FreeBSD Core members argue that "today the implicit approval implied by having it in the ports tree sends a message at odds with the project's aims."

Now this is bullshit as I'm sure FreeBSD core neither endorses Citrix (net/citrix_ica) nor Cisco (emulators/gna3, devel/libcli, graphics/py27-blockdiagcontrib-cisco and many more) but just hosts code to make living with them easier.

So the important thing here is:

Hosting is not endorsing. It is a purely technical act and by definition agnostic to the hosted content.

In every sane jurisdiction there is the requirement to remove hosted content that violates a law. And that makes sense. It reflects the societal consensus what is still acceptable and what is not. This changes over time but there is a proven process in place for these changes to become relevant: political discussion and consequential law making.

There is very deliberately never a law against bad taste and/or offensive humor. Where such a law still exists, you're in a somewhat underdeveloped jurisdiction. Because the hosting (pun intended) society has not matured sufficiently yet. This may happen due to overly conservative or self-protective ruling classes, ideological or religious blindness. None of these are desirable for society as a whole and the scissors in your head are paving the way to go back to darker ages. So don't. Be welcoming, be tolerant.

Tolerance means accepting things you do not like. Not accepting just what endorses your personal taste, beliefs or state of mind.

Does that mean, FreeBSD should continue to host the "Jive" filter? No, it's purely their choice. But their argument that hosting is endorsing is wrong. Inclusion into a FreeBSD media may be, like Debian strictly differentiates between the main archive, which it endorses, and contrib or non-free sections which it does not endorse. But still hosts regardless. So hosting is not endorsing.

That said, here you go:

The last file above is a mirror of files hosted by John B. Chambers. He has a "chef" cgi running there allowing the conversion of English text to "Swedish Chef", "Valley Girl" or "Pig Latin". And the "Jive" variant that uses the same Lex/Yacc/Flex files as the misc/jive that used to be part of the FreeBSD ports tree and is conserved above.

If you are interested in the public part of the discussion that happened after misc/jive was marked for removal from the ports tree, check out the freebsd-ports mailing list thread.

P.S.: Valspeak is still in the ports tree as misc/valspeak ... just sayin'.

P.P.S.: apt-cache show filters # Debian & Ubuntu. Awesome. ♡

During the 33C3 annual CCC Congress in Hamburg I gave a presentation on the developments in Dieselgate over 2016 and the VW strategy in Europe.

You can watch the video (40min) and/or browse the slides.

If you want to know why people laughed during Ijon's introduction of the talk ... click here. The video doesn't show this completely due to the camera being switched.

Nice neighbourhood, https://www.amsterdam.nl...

For your copy and paste pleasure:

Update

03.11.18: Cloudflare fixed this mess somewhat. They now look for SNI servernames and use customer-specific certs. See:

(notice the -servername in the above vs. the original command that will fail with something like 140246838507160:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769: now)

The BMW K1300S is only homologated to Euro 3 standards which means you can't buy a new one and get it on the road next year anymore. At least in Europe. Thus BMW has already removed it from its home page and sells off the remaining stock. Probably until the end of the year (they could license it and sell into 2017 but stock seems to be quite low).

I had the bike since 2009 as a company bike and when I returned to Germany that was the thing I found to have missed abroad the most. This is a bike from the time when the company strived to built the best motorbike technically possible. A time that has since passed in most of the automotive industry. Sad but true. Lifestyle products anyone?

So ... I've already bought mine last year.

It hasn't changed much from 2009 so there is no reason to buy a 2016 model year, buy any in good condition.

Double check that the handlebar switch units have been replaced with the improved versions (both sides). The stock ones don't like heat and stop working when it gets really hot. I got stuck at a friend's house after going for a long swim in the warm summer of 2015. BMW had a free replace-if-customer-complains (silent) recall until the end of last year. Prod your dealer to get some good will out of BMW Motorrad. Esp. before you buy. Also check the cardan drive for excess degrees of freedom. It doesn't take "binary" road racers too well. So make sure you get to know the previous owner.

As Motorrad 17/2016 put it:

In last years' sportstourer concept comparison the K 1300 S [..] still blew the competition away.
Brutal while well honed, that's the lasting impression. [..] Extra-ordinary stable, tight and still comfortable.
And still today the K 1300 [S] is the reference in breaking, because of the wide wheel base and because she stays up when breaking.
[The K-BMWs of the last generation] are cold perfection, executed into each detail. Just different.

That light we see is burning in my hall. How far that little candle throws his beams! So shines a good deed in a naughty world. William Shakespeare, The Merchant of Venice

The font size Mozilla chose for Firefox and Thunderbird menus looks awfully large on Netbook screens. It wastes space and is visually at odds with reasonably sized content. And for some weird reason you can set the content font and size via the menu but not the font and size for the drop-down menus themselves.

As the "Theme Font & Size Changer" Add-On doesn't work reliably and phones home way too often (showing a nag screen), I dug back into how to do this "manually". Probably a decade after I fixed this the first time...

You need to create the file ~/.mozilla/firefox/*/chrome/userChrome.css with * being your profile directory (<random_number>.default usually) and you most probably have to create the chrome directory first.

The same for Thunderbird resides in ~/.thunderbird/*/chrome/userChrome.css. Here again the chrome directory will most probably need to be created first.

needs to go into these files for Firefox or Thunderbird respectively. The curly braces are important. So copy & paste correctly. Symlinks or hardlinks are fine if those files do not need to differ between your web browser and your email client.

Restart Firefox and/or Thunderbird to see the effect.

Obviously you can choose any other font and font size in the snippet above to suit your taste and requirements.

If you are massively space-confined and don't mind a quite ugly UI, check out the Littlefox Add-on. Ugly but optimal use of the minimal screen estate with very small screens.

Netatalk 3.1.9 has been released with two interesting fixes / amendments:

• FIX: afpd: fix "admin group" option
• NEW: afpd: new options "force user" and "force group"

Here are the full release notes for 3.1.9 for your reading pleasure.

Due to upstream now differentiating between SysVinit and systemd packages I've followed that for simplicity's sake and built libgcrypt-only builds. If you need the openssl-based tools continue to use the 3.1.8 openssl build until you have finished your migration to a safer password storage.

Be sure to read the original blog post if you are new to Netatalk3 on Debian Jessie! You'll get nowhere if you install the .debs below and don't know about the upgrade path. So RTFA.

Now with that out of the way:

Updating to irssi 0.8.19 (which is a mainly a bugfix release to 0.8.18) proved a real issue. The enter key (return key) stopped working. Ctrl-J still worked but that's way too annoying to remember after each line. Searching the github issues turned up #327 Numeric keypad "Enter" key stopped working which didn't help much. Digging deeper it shows the irssi devs enabled "App key" mode in these releases which causes so many issues, they had to implement a switch to turn it off again.

So a hopeful: /set term_appkey_mode off followed by Ctrl+J, remember ...

and ... nothing changed.

So finally, after more digging and a quick consideration to go back to irssi 0.8.17 on Debian stable (Jessie) ...

/bind ^M key return

Yes, irssi 0.8.19 wants to be told what the enter key is, like, by default. Duh.
No idea what caused this in my configuration, I've been using irssi for more than a decade so much cruft has accumulated in my .irssi/config but ... in case you run into this as well, hopefully I helped you save a morning for something better to do.

If you want to fumble this into your .irssi/config (e.g. because Ctrl-J does not work for you):

keyboard = (
  [...]
  { key = "^M"; id = "key"; data = "return"; },
  [...]
);

We're currently installing a solution including dovecot for a company to go into production in April. So we kick this off with a development box for integrating parts that different suppliers to our customer are working on.

But after installing dovecot on the new joint development machine it just didn't start. It worked on our local development boxes but the install for the customer has been scripted with (what we call) "poor man's puppet" so it is somewhat hard to compare the setups. Same Ubuntu 14.04 LTS under the hood but on top of that things (like config layout, directory structures etc.) are quite different.

Back on topic: ps aux | grep d[o]ve returned empty.

Looking at /var/log/mail.log did not show anything relevant.

But syslog (/var/log/syslog) had some worrying lines like:

Mar 16 03:16:17 dev-new kernel: [ 3222.339365] doveconf[6420]: segfault at 200 ip 00007fa041b25a03 sp 00007ffe7881e070 error 4 in libc-2.19.so[7fa041ada000+1bb000]

Manually running the daemon resulted in:

# /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
Segmentation fault

So

mkdir /var/core
chmod 1777 /var/core
echo "/var/core/%p" > /proc/sys/kernel/core_pattern
ulimit -c unlimited
/usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
Segmentation fault (core dumped)

Better. We have a core file now.

But:

The Debian Netatalk3 saga continues at bug #685878. In season 4 of the epic the main issue still seems to be unclear license indications of a (very) few source files. And the usual "you go fix it", "no! you go fix it!". May be the fact that Firefox will be Firefox again in Debian [yeah!] could serve as an inspiration to the Netatalk maintainers?

Ah, well, until we have the eureka moment for Netatalk3 (4?) ...
<pragmatism style="priority-on-users:yes"> ... I'll post my .debs of the new 3.1.8 version of Netatalk as well.

Be sure to read the original blog post if you are new to Netatalk3 on Debian Jessie! You'll get nowhere if you install the .debs below and don't know about the upgrade path. So RTFA.

The release notes for 3.1.8 don't list anything that makes the update look mandatory but there is a nice compatibility fix for shares also exported via Samba (compatible xattrs handling). And it's faster.

The update instructions (assuming you have installed 3.1.7 before) are:

And here are the files:

Cygwin is a fantastic product for people that need to use Microsoft Windows and require some compatibility to Linux (or BSD or UNIX in general).

Unfortunately it is not trivial to keep it updated (and thus safe) as the update process requires downloading the latest installer and then clicking through the package list again and again on every update.
No apt-get update, no emerge --update @world, no dnf update.

But ... the people at Red Hat (who now own Cygwin) are not mean, they are just not good at documenting things ...

Be sure to have wget installed via cygwin, you'll need it to fetch the installer automatically.
Then drop the following batch file as cyg_update.bat into C:\cygwin64 (or where you have installed cygwin¹):

Once you start the batch (e.g. by double-clicking from Windows Explorer) it will download the latest installer from Cygwin.com and perform a silent update. You need to approve the Windows installer warning as you do with every manual install / update as well. So it's not a no-click update but a one-or-two-clicks update.
Still much better than the click fest without the batch file.

Update

01.06.2017: I've changed icacls cygwinSetup-x86_64.exe /grant Everyone:RX to icacls cygwinSetup-x86_64.exe /grant *S-1-1-0:RX which is the SID and not language dependent. 'cause otherwise German Windows would like to see "Jeder" and French "Tous publics", Chinese "任何人", etc. Looking them up on Microsoft's terminology search is quite nice but not really scalable.

Jake Edge of LWN wrote an excellent article on the presentation Felix and me gave at the 32C3 annual CCC congress in Hamburg, Germany on December 27th, 2015.

If you have an hour to spare, you can watch the video and/or browse the slides.

IPv6 is hard. It has many, many design flaws and the decade where we all ignored it and hoped for the better hasn't helped. So we're now all in on the protocol. Yeah.

One of the design principles is that it tries to be rather stateless in the configuration and "plug and play". But just like P&P in the good old ISA times, it just doesn't always work.

One of the common issues is that Linux bridges in IPv6 just don't work well with the router announcements that try to discover and configure the IPv6 neighbourhood.

The result is a sheer endless amount of "kernel: Neighbour table overflow." lines flooding dmesg and syslog (or journal for those on SystemD).

Oct  4 16:26:06 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: __ratelimit: 1832 callbacks suppressed
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: __ratelimit: 887 callbacks suppressed
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:23 host-260 kernel: __ratelimit: 803 callbacks suppressed

Lovely. Welcome to a storage DOS waiting to happen.

So first tip: cat /proc/sys/kernel/printk_ratelimit shows you the amount of seconds the ratelimiter suppresses messages. The default is 5 seconds and you can adjust it to more reasonable values in case you get heavily flooded like in the example above. Notice that this will mean your dmesg becomes rather useless as the kernel is not very selective about which messages to suppress.

Now when you google "Neighbour table overflow", you'll find thousands of pages suggesting to increase the arp / lladdr caches and garbage collection (gc) times like so:

# Set ARP cache garbage collection interval
net.ipv4.neigh.default.gc_interval = 3600
net.ipv6.neigh.default.gc_interval = 3600

# Set ARP cache entry timeout
net.ipv4.neigh.default.gc_stale_time = 3600
net.ipv6.neigh.default.gc_stale_time = 3600

# Setup cache threshold for ARP
net.ipv4.neigh.default.gc_thresh1 = 1024
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh3 = 4096

# And the same for IPv6
net.ipv6.neigh.default.gc_thresh1 = 1024
net.ipv6.neigh.default.gc_thresh2 = 2048
net.ipv6.neigh.default.gc_thresh3 = 4096

That helps if and only if you really have 500+ IPv6 neighbours. Unless you have a badly segmented network or run in a university lab, you don't.

Now ... you may be seeing messages like "kernel: vmbr0: Multicast hash table maximum of 512 reached, disabling snooping: eth0" or "kernel: vmbr0: Multicast hash table chain limit reached: eth0" in your dmesg / syslog / journal.

That hints at what is really happening here: The bridge confused the link-local router negotiation and so you get endless ff02:: neighbour routing entries added to your caches until they flow over. So increasing the caches as in the sysctl entries above is basically pasting band-aid over the problem.

ip route show cache table all will show you the tables. With all entries. See if you have too many ff02:: neighbours in there. If so, you should try to add change your /etc/network/interfaces on Debian / Ubuntu similar to this:

iface vmbr0 inet6 static
   address 2a02:0100:1:1::500:1
   netmask 64
   gateway 2a02:0100:1:1::1
   post-up echo 2048 > /sys/class/net/vmbr0/bridge/hash_max
   post-up echo 1 > /sys/class/net/vmbr0/bridge/multicast_snooping
   post-up echo 0 > /proc/sys/net/ipv6/conf/vmbr0/accept_ra

This obviously assumes your bridge is called vmbr0.

Red Hat/CentOS users will need to adjust the config spread throughout multiple files in /etc/sysconfig/network-scripts. The ifup-ipv6 script is a good one to look at and amend.

The increase of the hash_max entry makes your bridge survive the initial storm of (useless) router solicitations.
multicast_snooping is usually off when routing but you may need it to make sure your VMs on the bridge can be reached.
Finally we make sure the bridge does not accept router announcements. Because that is what the host system should handle.

Sometimes you may need to throw in a static route or two to reach the VMs. P&P, you remember ... ip -6 neigh add nud permanent proxy <VM:IPv6:goes::here> dev vmbr0 is your friend. Unfortunately the antidote for the dreaded "Neighbour table overflow" depends on the specific cause. So you'll have to poke around a bit. tcpdump -i eth0 -v ip6 will show you what is on the wire and tcpdump -i vmbr0 -v ip6 what's visible on the bridge.

Generating an indexed list of passwords without complex perl or python:

Explanation:

pwgen: -y = complex passwords (including symbols) ; 20 = length of password; 30 = number of passwords to generate

nl: -w 2 = zero pad to a width of two characters; -n rz = print right-justified; -s - = use dash as a separator