Skip to content

tail -S (truncating lines to terminal width)

Open Source

The tail command has a quite glaring omission in that it can't truncate lines. Thus it wraps long log line into multiple terminal lines regardless. Which makes them very hard to read.

I used to work around this using less -S and then hitting the [F] key but that's interactive. less +F <filename> is the little known work-around for the interactive issue but that still doesn't work well with pipes (tail -f logfile | grep "ERROR:" etc).

There is a bug report from 2004 against GNU coreutils but that went nowhere.

So we're not getting a tail -S anytime soon.

Bash to the rescue: tail -S → tails

Hence I wrote this little script, tails [1kB]:

  1. #!/bin/bash -i
  2. # v2 from 170712: introduce loop to work around GNU coreutils issues with pipe/fifo/isatty
  3.  
  4. if [[ -z "$COLUMNS" ]] ; then
  5.         MYCOL=$(tput cols)
  6. else
  7.         MYCOL=${COLUMNS}
  8. fi
  9. tail "$@" | while read line; do
  10.         echo "$line" | expand | cut -c1-${MYCOL:-80}
  11. done

Now, there are some interesting bits even in this tiny script:

The bash -i results in $COLUMNS being set within the script on sane Linux bash. Otherwise that variable wouldn't be available. Because it is a shell variable and not an environment variable. You knew that, right?

Unfortunately the bash -i doesn't get $COLUMNS set on either MacOS (X) or FreeBSD, so that's where the tput cols comes into play. It outputs the column width for the current terminal to stdout.

If all that fails tails will default to 80 columns.

So tails -f /var/log/apache/access.log will now look nice.

Corner case: color

If you use color codes somewhere (grep --color=always, dmesg --color=always) tails will just truncate lines too short so they will still not wrap.
There is a slight risk that it may cut into half a color code escape sequence and mess up the terminal a bit. You could change that by removing the -i from the shebang line and setting $COLUMNS explicitly. But that then needs manual adjustment for each combination of colored lines (=count of ANSI sequences) and terminal width. Better to color after the tails invocation then where possible, e.g. tails -f /var/log/httpd/error.log | grep ':error' to watch for PHP errors and the like.

Mended corner cases: inconsistent tail behaviour

A first version of the script didn't use a loop but just had

tail "$@" | expand | cut -c1-${MYCOL:-80}

This would break tails -f on Debian (coreutils 8.23) / Ubuntu (coreutils 8.26) while removing the |expand would make it work. On Fedora 25 (coreutils 8.25) I couldn't get tails -f to work at all with that v1. The cut (so just a single command chained) already broke the pipe :-(. And nope, stdbuf didn't help.

If you have a more simple solution to work around the isatty / isfifo mess, please leave a comment!

Alternatives

If you want to show multiple log tails in parallel, highlight strings etc. multitail is worth a look.

Depending on what you want to achieve you could also tell your terminal emulator to not wrap lines:

setterm -linewrap off; less -SR +F /var/log/apache/access.log; setterm -linewrap on

Prevent Ubuntu from phoning home

Linux

Ubuntu unfortunately has decided again to implement another "phone home" feature, this time transferring your lsb_release information, CPU model and speed (from /proc/cpuinfo), uptime output, most of uname -a and curl version to a Ubuntu news web-service.

Here is the Launchpad bug report #1637800 introducing this ... web bug.

This thing runs both systemd-timer based (via /lib/systemd/system/motd-news.service and /lib/systemd/system/motd-news.timer) and on request when you log in (via /etc/update-motd.d/50-motd-news).

Ubuntu news on ssh login

There has even been a bug filed about the motd advertising HBO's Silicon Valley show.

To prevent this from running (it is enabled by default on Ubuntu 17.04 and may probably propagate down to earlier versions as well), edit /etc/default/motd-news to include

ENABLED=0

so

sed -i "s/ENABLED=1/ENABLED=0/" /etc/default/motd-news # run as root

for your automated installs.

Update:

02.07.2017: Dustin Kirkland responded to a YC "hacker news" mention of his motd spam. He mentions:

You're welcome to propose your own messages for merging, if you have a well formatted, informative message for Ubuntu users.
We'll be happy to review and include them in the future.

What could possibly go wrong?

Thunderbird startup hang (hint: Add-Ons)

Debian

If you see Thunderbird hanging during startup for a minute and then continuing to load fine, you are probably running into an issue similar to what I saw when Debian migrated Icedove back to the "official" Mozilla Thunderbird branding and changed ~/.icedove to ~/.thunderbird in the process (one symlinked to the other).

Looking at the console log (=start Thunderbird from a terminal so you see its messages), I got:

console.log: foxclocks.bootstrap._loadIntoWindow(): got xul-overlay-merged - waiting for overlay-loaded
[.. one minute delay ..]
console.log: foxclocks.bootstrap._windowListener(): got window load chrome://global/content/commonDialog.xul

Stracing confirms it hangs because Thunderbird loops waiting for a FUTEX until that apparently gets kicked by a XUL core timeout.
(Thanks for defensive programming folks!)

So in my case uninstalling the Add-On Foxclocks easily solved the problem.

I assume other Thunderbird Add-Ons may cause the same issue, hence the more generic description above.

Updating the Dell XPS 13 9360 Thunderbolt firmware to get VGA and HDMI working

IT

Last year I bought the wonderful Dell XPS 13 9360 as it is certified to work with Ubuntu Linux and is just all around an awesome device. Dell made me buy the Windows version as only that got a 1 TB NVMe-SSD option. Linux apparently is only worthy of the 512GB and below models. What product manager comes up with such a stupid idea? Are SKUs that precious? Anyways ... so I bought a Windows version and that got wiped with a Linux install immediately as that was and is its intended purpose.

Dell DA200 USB-C to HDMI/VGA/Ethernet/USB 3.0 adapter

I purchased a DA200 with the system which is Dell's USB-C to anything (HDMI/VGA/Ethernet/USB 3.0) dongle. When I got the laptop the Ethernet port and USB 3.0 via the DA200 were working right out of the box. The VGA and HDMI ports were detected by Ubuntu but there was no way to get connected screens working. They stayed black.

The device was shipped with Thunderbolt firmware NVM18 and we've been told rather quickly by Dell this would be fixed with an update. And lo and behold Dell published the firmware version NVM21 right for Christmas 2016. Now unfortunately while their BIOS updates are Windows / DOS executables that can be just shoved at the Dell UEFI flash updater and thus the main BIOS can be updated from any OS, including Linux, without any hassle, the Intel provided Thunderbolt update needs Windows to get installed. Or, well, there is a convoluted way to compile an out-of-tree Linux kernel module, download and compile a few sets of software and do it via Linux. That description read so lengthy, I didn't even try it. Additionally there seems to have been no progress at all in getting this more mainline in the last three months, so I chose the cheap route and installed Windows 10 on a USB thumb drive1.

This is done via the (unfortunately Windows only) Win2USB software (the free version is sufficient).

Update: There's a new bash script windows2usb that looks good and should work to get you a bootable Windows USB thumb drive in Linux. WinUSB (that stopped working in the Win10 area some time) has also been forked and updated into WoeUSB. And there is WinToUSBLinux, yet another shell script. Give them a try.

Once Windows has rebooted often enough to finish its own installation, you can work with the USB thumb drive install as with any Windows 10. Nice.

Dell TPM 1.2 to 2.0 firmware update

Put all the files you downloaded from Dell to update your XPS 13 into a directory on the USB thumb drive. That way Windows does not need to have any network connectivity.

I first updated the TPM 1.2 firmware to a TPM 2.0 version (DellTpm2.0_Fw1.3.2.8_V1_64.exe at the time of writing this blog entry). Now this is quite hilarious as the Windows installer doesn't do anything but putting a UEFI firmware update into the EFI partition that runs on reboot. Duh. You do need to manually clear the TPM in the BIOS' security settings section (there's a clear checkbox) to be able to program new firmware onto it.
Thunderbolt firmware upgrade progress bar Thunderbolt firmware upgrade successful Now back in Windows install the Thunderbolt drivers (Chipset_Driver_J95RR_WN32_16.2.55.275_A01.exe at the time of writing this) and then run Intel_TBT3_FW_UPDATE_NVM21_0THFT_A00_3.21.00.008.exe, which is the NVM21 Thunderbolt firmware update (or a later version).

Reboot again (into Linux if you want to) and (drumroll) the VGA and HDMI ports are working. Awesome.

An update log can be found on the USB thumb drive at Dell\UpdatePackage\Log\Intel_TBT3_FW_UPDATE_NVM21_0THFT_A00_3.log:

*** Dell Thunderbolt firmware update started on 4/6/2017 at 12:56:56***
Command: C:\Install\Intel_TBT3_FW_UPDATE_NVM21_0THFT_A00_3.21.00.008.exe 

Starting FW Update....
***TBT GPIO Power is Turning On:  No Dock or DockInfo.
***TBT GPIO power is turned on.

Thunderbolt Firmware Update SUCCEEDED
TBT Items Registry creation is Success at \SOFTWARE\Dell\ManageableUpdatePackage\Thunderbolt Controller:
User selected OK for reboot
System TBT NVM Current Version:BCD:00000018: New Version:BCD:00000021

Exit Code = 0 (Success) 
***Thunderbolt Firmware flash finished at 4/6/2017 at 13:00:23***

If Windows has added its boot loader entry into your UEFI options, you can easily remove that again with the Dell UEFI BIOS or efibootmgr from within Linux.

The whole process took me less than 30 minutes. And most of that was creating the Windows USB thumb drive. I'll keep that for future updates until Intel and Dell have sorted out the Thunderbolt update process in Linux.

Updates:

18.05.17: Intel has published a large patchset on LKML to enable Thunderbolt security levels (thus preventing DMA attacks) and get NVM firmware upgrades mainlined. Yeah!

02.05.18: Added a link to the windows2usb bash script that should remove the need to create a bootable Windows USB thumb drive with a Windows only software.

11.06.18: Added a link to WoeUSB which is currently packaged for Ubuntu in a PPA.

08.04.20: Added a link to WinToUSBLinux. A recently released shell script to create a bootable Windows USB stick from Linux.


  1. If you go the Linux route please post a minimal image somewhere (kernel, initrd, squashfs or FAT16/32 raw image) and put a link into a comment below this blog post. Thanks. 

Saving misc/jive

BSD

One thing I love about FreeBSD is the way the core team keeps the wider community updated about project news e.g. via their quarterly status reports. So while reading the FreeBSD Q4/2016 status report, I was quite surprised to find that a text filter converting English to "Jive speak" had been removed from the ports tree. FreeBSD Core members argue that "today the implicit approval implied by having it in the ports tree sends a message at odds with the project's aims."

Now this is bullshit as I'm sure FreeBSD core neither endorses Citrix (net/citrix_ica) nor Cisco (emulators/gna3, devel/libcli, graphics/py27-blockdiagcontrib-cisco and many more) but just hosts code to make living with them easier.

So the important thing here is:

Important: Switch on brain and try to memorize. Hosting is not endorsing.
It is a purely technical act and by definition agnostic to the hosted content.

In every sane jurisdiction there is the requirement to remove hosted content that violates a law. And that makes sense. It reflects the societal consensus what is still acceptable and what is not. This changes over time but there is a proven process in place for these changes to become relevant: political discussion and consequential law making.

There is very deliberately never a law against bad taste and/or offensive humor. Where such a law still exists, you're in a somewhat underdeveloped jurisdiction. Because the hosting (pun intended) society has not matured sufficiently yet. This may happen due to overly conservative or self-protective ruling classes, ideological or religious blindness. None of these are desirable for society as a whole and the scissors in your head are paving the way to go back to darker ages. So don't. Be welcoming, be tolerant.

Tolerance means accepting things you do not like. Not accepting just what endorses your personal taste, beliefs or state of mind.

Does that mean, FreeBSD should continue to host the "Jive" filter? No, it's purely their choice. But their argument that hosting is endorsing is wrong. Inclusion into a FreeBSD media may be, like Debian strictly differentiates between the main archive, which it endorses, and contrib or non-free sections which it does not endorse. But still hosts regardless. So hosting is not endorsing.

That said, here you go:

File Function sha256
jive-1.1.tar.gz Source to the "Jive" filter 3463d80ad159a27d9fcf87f163a7be5eba39dbf15c5156f052798b81271523f2
ports_misc_jive.tar.gz ports files to build the "Jive" filter under FreeBSD 47dc7b660d499d671daa18f992cdd348bd95c34e02874addd2bcf3e5c3f90b59
swedishchef.zip mirror of swedishchef.zip d0830b81aec6ad6a6ff824e1d80c9fa97d3a5447bad9f8a2b32dbd0dfb8df709

The last file above is a mirror of files hosted by John B. Chambers. He has a "chef" cgi running there allowing the conversion of English text to "Swedish Chef", "Valley Girl" or "Pig Latin". And the "Jive" variant that uses the same Lex/Yacc/Flex files as the misc/jive that used to be part of the FreeBSD ports tree and is conserved above.

If you are interested in the public part of the discussion that happened after misc/jive was marked for removal from the ports tree, check out the freebsd-ports mailing list thread.

P.S.: Valspeak is still in the ports tree as misc/valspeak ... just sayin'.

P.P.S.: apt-cache show filters # Debian & Ubuntu. Awesome. ♡

Mozilla Firefox and Thunderbird Menu font sizes

Open Source

The font size Mozilla chose for Firefox and Thunderbird menus looks awfully large on Netbook screens. It wastes space and is visually at odds with reasonably sized content. And for some weird reason you can set the content font and size via the menu but not the font and size for the drop-down menus themselves.

As the "Theme Font & Size Changer" Add-On doesn't work reliably and phones home way too often (showing a nag screen), I dug back into how to do this "manually". Probably a decade after I fixed this the first time...

You need to create the file ~/.mozilla/firefox/*/chrome/userChrome.css with * being your profile directory (<random_number>.default usually) and you most probably have to create the chrome directory first.

The same for Thunderbird resides in ~/.thunderbird/*/chrome/userChrome.css. Here again the chrome directory will most probably need to be created first.


/* Global UI font */
* { font-size: 10pt !important;
  font-family: Ubuntu !important;
}
 

needs to go into these files for Firefox or Thunderbird respectively. The curly braces are important. So copy & paste correctly. Symlinks or hardlinks are fine if those files do not need to differ between your web browser and your email client.

Restart Firefox and/or Thunderbird to see the effect.

Obviously you can choose any other font and font size in the snippet above to suit your taste and requirements.

If you are massively space-confined and don't mind a quite ugly UI, check out the Littlefox Add-on. Ugly but optimal use of the minimal screen estate with very small screens.

Netatalk 3.1.9 .debs for Debian Jessie available (Apple Timemachine backup to Linux servers)

Debian

Netatalk 3.1.9 has been released with two interesting fixes / amendments:

  • FIX: afpd: fix "admin group" option
  • NEW: afpd: new options "force user" and "force group"

Here are the full release notes for 3.1.9 for your reading pleasure.

Due to upstream now differentiating between SysVinit and systemd packages I've followed that for simplicity's sake and built libgcrypt-only builds. If you need the openssl-based tools continue to use the 3.1.8 openssl build until you have finished your migration to a safer password storage.

Warning: Read the original blog post before installing for the first time. Be sure to read the original blog post if you are new to Netatalk3 on Debian Jessie!
You'll get nowhere if you install the .debs below and don't know about the upgrade path. So RTFA.

Now with that out of the way:

Continue reading "Netatalk 3.1.9 .debs for Debian Jessie available (Apple Timemachine backup to Linux servers)"

Irssi update to 0.8.19 from Debian jessie-backports may break enter / carriage return key / ↵ key

Debian

Updating to irssi 0.8.19 (which is a mainly a bugfix release to 0.8.18) proved a real issue. The enter key (return key) stopped working. Ctrl-J still worked but that's way too annoying to remember after each line. Searching the github issues turned up #327 Numeric keypad "Enter" key stopped working which didn't help much. Digging deeper it shows the irssi devs enabled "App key" mode in these releases which causes so many issues, they had to implement a switch to turn it off again.

So a hopeful: /set term_appkey_mode off followed by Ctrl+J, remember ...

and ... nothing changed.

So finally, after more digging and a quick consideration to go back to irssi 0.8.17 on Debian stable (Jessie) ...

/bind ^M key return

Yes, irssi 0.8.19 wants to be told what the enter key is, like, by default. Duh.
No idea what caused this in my configuration, I've been using irssi for more than a decade so much cruft has accumulated in my .irssi/config but ... in case you run into this as well, hopefully I helped you save a morning for something better to do.

If you want to fumble this into your .irssi/config (e.g. because Ctrl-J does not work for you):

keyboard = (
  [...]
  { key = "^M"; id = "key"; data = "return"; },
  [...]
);

Dovecot segfaulting (Ubuntu 14.04 LTS, CentOS 6 and 7)

Open Source

We're currently installing a solution including dovecot for a company to go into production in April. So we kick this off with a development box for integrating parts that different suppliers to our customer are working on.

But after installing dovecot on the new joint development machine it just didn't start. It worked on our local development boxes but the install for the customer has been scripted with (what we call) "poor man's puppet" so it is somewhat hard to compare the setups. Same Ubuntu 14.04 LTS under the hood but on top of that things (like config layout, directory structures etc.) are quite different.

Back on topic: ps aux | grep d[o]ve returned empty.

Looking at /var/log/mail.log did not show anything relevant.

But syslog (/var/log/syslog) had some worrying lines like:

Mar 16 03:16:17 dev-new kernel: [ 3222.339365] doveconf[6420]: segfault at 200 ip 00007fa041b25a03 sp 00007ffe7881e070 error 4 in libc-2.19.so[7fa041ada000+1bb000]

Manually running the daemon resulted in:

# /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
Segmentation fault

So

mkdir /var/core
chmod 1777 /var/core
echo "/var/core/%p" > /proc/sys/kernel/core_pattern
ulimit -c unlimited
/usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
Segmentation fault (core dumped)

Better. We have a core file now.

But:

Continue reading "Dovecot segfaulting (Ubuntu 14.04 LTS, CentOS 6 and 7)"

Netatalk 3.1.8 .debs for Debian Jessie available (Apple Timemachine backup to Linux servers)

Debian

The Debian Netatalk3 saga continues at bug #685878. In season 4 of the epic the main issue still seems to be unclear license indications of a (very) few source files. And the usual "you go fix it", "no! you go fix it!". May be the fact that Firefox will be Firefox again in Debian [yeah!] could serve as an inspiration to the Netatalk maintainers?

Ah, well, until we have the eureka moment for Netatalk3 (4?) ...
<pragmatism style="priority-on-users:yes"> ... I'll post my .debs of the new 3.1.8 version of Netatalk as well.

Warning: Read the original blog post before installing for the first time. Be sure to read the original blog post if you are new to Netatalk3 on Debian Jessie!
You'll get nowhere if you install the .debs below and don't know about the upgrade path. So RTFA.

The release notes for 3.1.8 don't list anything that makes the update look mandatory but there is a nice compatibility fix for shares also exported via Samba (compatible xattrs handling). And it's faster.

The update instructions (assuming you have installed 3.1.7 before) are:

# install new debs
dpkg -i libatalk17_3.1.8-1_amd64.deb netatalk_3.1.8-1_amd64.deb
# reboot the box (restart of netatalk may not be sufficient)
reboot
# After reboot: remove the obsolete libatalk16 (3.1.8 uses libatalk17)
dpkg -r libatalk16

And here are the files:

Continue reading "Netatalk 3.1.8 .debs for Debian Jessie available (Apple Timemachine backup to Linux servers)"

Cygwin automatic updates

IT

Cygwin is a fantastic product for people that need to use Microsoft Windows and require some compatibility to Linux (or BSD or UNIX in general).

Unfortunately it is not trivial to keep it updated (and thus safe) as the update process requires downloading the latest installer and then clicking through the package list again and again on every update.
No apt-get update, no emerge --update @world, no dnf update.

But ... the people at Red Hat (who now own Cygwin) are not mean, they are just not good at documenting things :-)...

Be sure to have wget installed via cygwin, you'll need it to fetch the installer automatically.
Then drop the following batch file as cyg_update.bat into C:\cygwin64 (or where you have installed cygwin1):

  1. @echo off
  2. cd /d C:\cygwin64
  3. del /Q cygwinSetup-x86_64.exe
  4. bin\wget.exe --progress=dot -S -N http://cygwin.com/setup-x86_64.exe
  5. move /y setup-x86_64.exe cygwinSetup-x86_64.exe
  6. REM S-1-1-0 is the SID for "Everyone"
  7. icacls cygwinSetup-x86_64.exe /grant *S-1-1-0:RX
  8. cygwinSetup-x86_64.exe --no-desktop --no-shortcuts --no-startmenu --quiet-mode

Once you start the batch (e.g. by double-clicking from Windows Explorer) it will download the latest installer from Cygwin.com and perform a silent update. You need to approve the Windows installer warning as you do with every manual install / update as well. So it's not a no-click update but a one-or-two-clicks update.
Still much better than the click fest without the batch file.

Cygwin update screenshot

Update

01.06.2017: I've changed icacls cygwinSetup-x86_64.exe /grant Everyone:RX to icacls cygwinSetup-x86_64.exe /grant *S-1-1-0:RX which is the SID and not language dependent. 'cause otherwise German Windows would like to see "Jeder" and French "Tous publics", Chinese "任何人", etc. Looking them up on Microsoft's terminology search is quite nice but not really scalable.


  1. If you have installed the 32bit version and/or used a different install path, adjust line 2 of the batch file accordingly. Did I need to say that? Hm, well, I did ... have a cookie. 

Security is hard, open source security unnecessarily harder

IT

Now it is a commonplace that security is hard. It involves advanced mathematics and a single, tiny mistake or omission in implementation can spoil everything.

And the only sane IT security can be open source security. Because you need to assess the algorithms and their implementation and you need to be able to completely verify the implementation. You simply can't if you don't have the code and can compile it yourself to produce a trusted (ideally reproducible) build. A no-brainer for everybody in the field.

But we make it unbelievably hard for people to use security tools. Because these have grown over decades fostered by highly intelligent people with no interest in UX.
"It was hard to write, so it should be hard to use as well."
And then complain about adoption.

PGP / gpg has received quite some fire this year and the good news is this has resulted in funding for the sole gpg developer. Which will obviously not solve the UX problem.

But the much worse offender is OpenSSL. It is so hard to use that even experienced hackers fail.

IRC wallop on hackint

Now, securely encrypting a mass communication media like IRC is not possible at all. Read Trust is not transitive: or why IRC over SSL is pointless1.
Still it makes wiretapping harder and that may be a good thing these days.

LibreSSL has forked the OpenSSL code base "with goals of modernizing the codebase, improving security, and applying best practice development processes". No UX improvement. A cleaner code for the chosen few. Duh.

I predict the re-implementations and gradual improvement scenarios will fail. The nearly-impossible-to-use-right situation with both gpg and (much more importantly) OpenSSL cannot be fixed by gradual improvements and however thorough code reviews.

Now the "there's an App for this" security movement won't work out on a grand scale either:

  1. Most often not open source. Notable exceptions: ChatSecure, TextSecure.
  2. No reference implementations with excellent test servers and well documented test suites but products. "Use my App.", "No, use MY App!!!".
  3. Only secures chat or email. So the VC-powered ("next WhatsApp") mass-adoption markets but not the really interesting things to improve upon (CA, code signing, FDE, ...).
  4. While everybody is focusing on mobile adoption the heavy lifting is still on servers. We need sane libraries and APIs. No App for that.

So we need a new development, a new code, a new open source product. Sadly so the Core Infrastructure Initiative so far only funds existing open source projects in dire needs and people bug hunting.

It basically makes the bad solutions of today a bit more secure and ensures maintenance of decade old crufty code bases. That way it extends the suffering of everybody using the inadequate solutions of today.

That's inevitable until we have a better stack but we need to look into getting rid of gpg and OpenSSL and replacing it with something new. Something designed well from the ground up, technically and from a user experience perspective.

Now who's in for a five year funding plan? $3m2 annually. ROCE 0. But a very good chance to get the OBE awarded.

Keep calm and enjoy the silence

Updates:

21.07.19: A current essay on "The PGP problem" is making rounds and lists some valid issues with the file format, RFCs and the gpg implementation. The GnuPG-users mailing list has a discussion thread on the issues listed in the essay.

19.01.19: Daniel Kahn Gillmor, a Senior Staff Technologist at the ACLU, tried to get his gpg key transition correct. He put a huge amount of thought and preparation into the transition. To support Autocrypt (another try to get GPG usable for more people than a small technical elite), he specifically created different identities for him as a person and his two main email addresses. Two days later he has to invalidate his new gpg key and back-off to less "modern" identity layouts because many of the brittle pieces of infrastructure around gpg from emacs to gpg signature management frontends to mailing list managers fell over dead.

28.11.18: Changed the Quakenet link on why encrypting IRC is useless to an archive.org one as they have removed the original content.

13.03.17: Chris Wellons writes about why GPG is a failure and created a small portable application Enchive to replace it for asymmetric encryption.

24.02.17: Stefan Marsiske has written a blog article: On PGP. He argues about adversary models and when gpg is "probably" 3 still good enough to use. To me a security tool can never be a sane choice if the UI is so convoluted that only a chosen few stand at least a chance of using it correctly. Doesn't matter who or what your adversary is.
Stefan concludes his blog article:

PGP for encryption as in RFC 4880 should be retired, some sunk-cost-biases to be coped with, but we all should rejoice that the last 3-4 years had so much innovation in this field, that RFC 4880 is being rewritten[Citation needed] with many of the above in mind and that hopefully there'll be more and better tools. [..]

He gives an extensive list of tools he considers worth watching in his article. Go and check whether something in there looks like a possible replacement for gpg to you. Stefan also gave a talk on the OpenPGP conference 2016 with similar content, slides.

14.02.17: James Stanley has written up a nice account of his two hour venture to get encrypted email set up. The process is speckled with bugs and inconsistent nomenclature capable of confusing even a technically inclined person. There has been no progress in the last ~two years since I wrote this piece. We're all still riding dead horses. James summarizes:

Encrypted email is nothing new (PGP was initially released in 1991 - 26 years ago!), but it still has a huge barrier to entry for anyone who isn't already familiar with how to use it.

04.09.16: Greg Kroah-Hartman ends an analysis of the Evil32 PGP keyid collisions with:

gpg really is horrible to use and almost impossible to use correctly.

14.11.15:
Scott Ruoti, Jeff Andersen, Daniel Zappala and Kent Seamons of BYU, Utah, have analysed the usability [local mirror, 173kB] of Mailvelope, a webmail PGP/GPG add-on based on a Javascript PGP implementation. They describe the results as "disheartening":

In our study of 20 participants, grouped into 10 pairs of participants who attempted to exchange encrypted email, only one pair was able to successfully complete the assigned tasks using Mailvelope. All other participants were unable to complete the assigned task in the one hour allotted to the study. Even though a decade has passed since the last formal study of PGP, our results show that Johnny has still not gotten any closer to encrypt his email using PGP.

  1. Quakenet has removed that article citing "near constant misrepresentation of the presented argument" sometime in 2018. The contents (not misrepresented) are still valid so I have added and archive.org Wayback machine link instead. 

  2. The estimate was $2m until end of 2018. The longer we wait, the more expensive it'll get. And - obviously - ever harder. E.g. nobody needed to care about sidechannel attacks on big-LITTLE five years ago. But now they start to hit servers and security-sensitive edge devices. 

  3. Stefan says "probably" five times in one paragraph. Probably needs an editor. The person not the application. 

Apple Timemachine backups on Debian 8 (Jessie)

Debian

Upgrading Debian 7 (Wheezy) servers to Debian 8 (Jessie) proves (unexpectedly) quite rough around the edges.

That's what you get for using a version x.0, we should have known better :-).

And - of course - the release notes follow the common practice of not even mentioning any of the issues we encountered so far.

Ah, well, let's go through the first one:

In Debian 7 (Wheezy) there was netatalk 2.2.2 (packages link). Now during upgrades that package may or may not get removed. There is no netatalk in Debian 8 (Jessie) anymore. Duh. There is in sid (aka Debian unstable) (packages link) so we may see a backport some time. Or not. In any case this is still 2.2.5 at the time of writing and as Adrian Knoth put it in the three year old bug asking for a upgrade to Netatalk 3:

Let's not ship another release without netatalk3, it's embarrassing.

Yes. It is. Removing a working version and not even mentioning it in the release notes is even worse though. So no cookies there.

Luckily the absolutely awesome Debian and greater FLOSS community have sorted 90% of the problem out for us already:

A quick Google search turns up an excellent article on netatalk's wiki that details installing Netatalk 3.1.7 on Debian 8 Jessie. There are two shortcomings to this: First it doesn't compile to .debs but installs besides apt and friends. And second it compiles with Spotlight search, courtesy of Gnome tracker, which doesn't really work well on servers yet. Hence we're lucky that Adrian Knoth's debified install has not yet added the tracker dependencies. It does compile for systemd use (the default for Debian 8 Jessie). If you want to continue using SysVInit, you need to modify debian/rules.

Compiling to .debs becomes as easy as:

# get build dependencies and a few helpers
apt-get install build-essential devscripts debhelper cdbs autotools-dev dh-buildinfo libdb-dev libwrap0-dev libpam0g-dev libcups2-dev libkrb5-dev libltdl3-dev libgcrypt11-dev libcrack2-dev libavahi-client-dev libldap2-dev libacl1-dev libevent-dev d-shlibs dh-systemd
# in case you want to try the tracker support (you need to ammend the debian/ build config as well)
# apt-get install tracker libtracker-sparql-1.0-dev libtracker-miner-1.0-dev  
git clone https://github.com/adiknoth/netatalk-debian
cd netatalk-debian
debuild -b -uc -us

This should leave you with (at the time of writing this):

File Function md5 sha1
libatalk-dev_3.1.7-1_amd64.deb Development files for the libatalk library (dev only) e5a465e39a8560c919d8db85c8e5a83b 0b924cf75f22ab42406289c6f18ae0243d6396a3
libatalk16_3.1.7-1_amd64.deb libatalk library (needed) 17a3d677ed0b3df1c2f4c1a8ab9045fd 7345ed3edd442716c99c2fe979140703204c0826
netatalk_3.1.7-1_amd64.deb netatalk daemons (needed) c694abca7f3cdc0070b2b3e7d528324a 932d1e3d5899958f29e79a7ba40e858d4ac272e8

Obviously you can download the files above if you run the AMD64 architecture and trust me enough to compile them for you.

Continue reading "Apple Timemachine backups on Debian 8 (Jessie)"

Managing a project consisting of multiple git repositories

IT

The core team organizing DebConf, the annual Debian developer conference, reached out to me two weeks ago to help support this year's effort a bit.

I'm very happy to do so as Debian is a cornerstone of everything I do in the Open Source/Free Software space.

Screenshot of git_pull_all with color

To get me started I got access to a lot of mailing lists and irc channels. And even more git repositories. So many that the DebConf team even has an instruction page on how the repositories all fit together.

It's unfortunately quite common to split a bigger project into many git repositories to ease access rights management and reduce the noise and data transfer volume for the average user. The downside is, everybody ends up with a dozen or more individual repositories to keep pulling. And then there's git annex for yet another level of indirection.

Joey Hess, a former Debian developer, has even written an extensive tool, myrepos, to meta-manage the different repositories and it can do quite some magic across different SCMSs1. In my case this is a bit of an overkill though.

And using myrepos may get you confused at some point whether to now run mr or git directly for each batch of repos you have inherited over some time of working on multiple projects.2 Thus I prefer the simple route:

Check out each repository into a common top-level directory (~/debconf/ in this case) and then put the following two lines into an executable script git_pull_all into that top level directory:

#!/bin/sh
find ~/debconf -mindepth 1 -maxdepth 1 -type d -exec sh -c "cd {}; test -r .git/config && git pull $*" \;

This will allow you to pull all git repos with one command and keep the normal syntax for everything else you do with each repo.

The --mindepth and --maxdepth will instruct find to just go and run your git pull only inside each direct child of the top level directory. So recursion depth = 1. That is the single trick there is to this.

Updates:

If you like to have some color and a bit of a spaced layout for improved readability, try:

#!/bin/sh
find ~/debconf -mindepth 1 -maxdepth 1 -type d -exec sh -c "cd {}; test -r .git/config && (printf \"\033[1m\033[34m%-50s\033[0m\" \"\${PWD}:\" ; git pull $*)" \;

When you have pull.rebase=true set in your .gitconfig, you can run ./git_pull_all --no-rebase to avoid rebases in case you work somewhere and want to have the merge commits.

P.S.: The DebConf15 Heidelberg registration just opened, please check the DebConf15 homepage for news, venue information and please register if you want to come around.


  1. Source Code Management Systems, like git, mercurial (hg) or subversion (svn). Or God forbid ... cvs. I don't like the (D)VCS (Distributed) Version Control Systems moniker. Because that's not really all these systems do. Not even the most important piece of what they do these days. 

  2. With myrepos you can still work with each individual repository via git. Just so nobody will write in "but...". 

scp (secure copy) a file to the same path on a remote system

IT

I've been copying files from one server to the symmetrical cluster partner a lot last week.

It's always

 scp /path/to/directory/file remote:/path/to/directory/

That gets boring after a while but does not really warrant setting up a full blown config management solution like salt, puppet or ansible.

So here is scpover[1.5kB].

It reduces the effort to

 scpover /path/to/directory/file

This will copy "file" from "/path/to/directory/" into exactly the same directory on the remote cluster partner. Which system to target is detected from the hostname of the local system and hard-coded into the script.

Scpover can also take multiple filepaths on one command line so you can beam over a few related config files from multiple locations in one go.
Not an atomic change but at least all within a reasonably short timespan.

Continue reading "scp (secure copy) a file to the same path on a remote system"