Skip to content

Summer Fun II

BMW

The BMW K1300S is only homologated to Euro 3 standards which means you can't buy a new one and get it on the road next year anymore. At least in Europe. Thus BMW has already removed it from its home page and sells off the remaining stock. Probably until the end of the year (they could license it and sell into 2017 but stock seems to be quite low).

I had the bike since 2009 as a company bike and when I returned to Germany that was the thing I found to have missed abroad the most. This is a bike from the time when the company strived to built the best motorbike technically possible. A time that has since passed in most of the automotive industry. Sad but true. Lifestyle products anyone?

So ... I've already bought mine last year.

Daniel's "new" K1300S

It hasn't changed much from 2009 so there is no reason to buy a 2016 model year, buy any in good condition.

Double check that the handlebar switch units have been replaced with the improved versions (both sides). The stock ones don't like heat and stop working when it gets really hot. I got stuck at a friend's house after going for a long swim in the warm summer of 2015. BMW had a free replace-if-customer-complains (silent) recall until the end of last year. Prod your dealer to get some good will out of BMW Motorrad. Esp. before you buy. Also check the cardan drive for excess degrees of freedom. It doesn't take "binary" road racers too well. So make sure you get to know the previous owner.

As Motorrad 17/2016 put it:

In last years' sportstourer concept comparison the K 1300 S [..] still blew the competition away.
Brutal while well honed, that's the lasting impression. [..] Extra-ordinary stable, tight and still comfortable.
And still today the K 1300 [S] is the reference in breaking, because of the wide wheel base and because she stays up when breaking.
[The K-BMWs of the last generation] are cold perfection, executed into each detail. Just different.

BMW K1300S candle

That light we see is burning in my hall. How far that little candle throws his beams! So shines a good deed in a naughty world. William Shakespeare, The Merchant of Venice

Mozilla Firefox and Thunderbird Menu font sizes

Open Source

The font size Mozilla chose for Firefox and Thunderbird menus looks awfully large on Netbook screens. It wastes space and is visually at odds with reasonably sized content. And for some weird reason you can set the content font and size via the menu but not the font and size for the drop-down menus themselves.

As the "Theme Font & Size Changer" Add-On doesn't work reliably and phones home way too often (showing a nag screen), I dug back into how to do this "manually". Probably a decade after I fixed this the first time...

You need to create the file ~/.mozilla/firefox/*/chrome/userChrome.css with * being your profile directory (<random_number>.default usually) and you most probably have to create the chrome directory first.

The same for Thunderbird resides in ~/.thunderbird/*/chrome/userChrome.css. Here again the chrome directory will most probably need to be created first.


/* Global UI font */
* { font-size: 10pt !important;
  font-family: Ubuntu !important;
}
 

needs to go into these files for Firefox or Thunderbird respectively. The curly braces are important. So copy & paste correctly. Symlinks or hardlinks are fine if those files do not need to differ between your web browser and your email client.

Restart Firefox and/or Thunderbird to see the effect.

Obviously you can choose any other font and font size in the snippet above to suit your taste and requirements.

If you are massively space-confined and don't mind a quite ugly UI, check out the Littlefox Add-on. Ugly but optimal use of the minimal screen estate with very small screens.

Netatalk 3.1.9 .debs for Debian Jessie available (Apple Timemachine backup to Linux servers)

Debian

Netatalk 3.1.9 has been released with two interesting fixes / amendments:

  • FIX: afpd: fix "admin group" option
  • NEW: afpd: new options "force user" and "force group"

Here are the full release notes for 3.1.9 for your reading pleasure.

Due to upstream now differentiating between SysVinit and systemd packages I've followed that for simplicity's sake and built libgcrypt-only builds. If you need the openssl-based tools continue to use the 3.1.8 openssl build until you have finished your migration to a safer password storage.

Warning: Read the original blog post before installing for the first time. Be sure to read the original blog post if you are new to Netatalk3 on Debian Jessie!
You'll get nowhere if you install the .debs below and don't know about the upgrade path. So RTFA.

Now with that out of the way:

Continue reading "Netatalk 3.1.9 .debs for Debian Jessie available (Apple Timemachine backup to Linux servers)"

Irssi update to 0.8.19 from Debian jessie-backports may break enter / carriage return key / ↵ key

Debian

Updating to irssi 0.8.19 (which is a mainly a bugfix release to 0.8.18) proved a real issue. The enter key (return key) stopped working. Ctrl-J still worked but that's way too annoying to remember after each line. Searching the github issues turned up #327 Numeric keypad "Enter" key stopped working which didn't help much. Digging deeper it shows the irssi devs enabled "App key" mode in these releases which causes so many issues, they had to implement a switch to turn it off again.

So a hopeful: /set term_appkey_mode off followed by Ctrl+J, remember ...

and ... nothing changed.

So finally, after more digging and a quick consideration to go back to irssi 0.8.17 on Debian stable (Jessie) ...

/bind ^M key return

Yes, irssi 0.8.19 wants to be told what the enter key is, like, by default. Duh.
No idea what caused this in my configuration, I've been using irssi for more than a decade so much cruft has accumulated in my .irssi/config but ... in case you run into this as well, hopefully I helped you save a morning for something better to do.

If you want to fumble this into your .irssi/config (e.g. because Ctrl-J does not work for you):

keyboard = (
  [...]
  { key = "^M"; id = "key"; data = "return"; },
  [...]
);

Dovecot segfaulting (Ubuntu 14.04 LTS, CentOS 6 and 7)

Open Source

We're currently installing a solution including dovecot for a company to go into production in April. So we kick this off with a development box for integrating parts that different suppliers to our customer are working on.

But after installing dovecot on the new joint development machine it just didn't start. It worked on our local development boxes but the install for the customer has been scripted with (what we call) "poor man's puppet" so it is somewhat hard to compare the setups. Same Ubuntu 14.04 LTS under the hood but on top of that things (like config layout, directory structures etc.) are quite different.

Back on topic: ps aux | grep d[o]ve returned empty.

Looking at /var/log/mail.log did not show anything relevant.

But syslog (/var/log/syslog) had some worrying lines like:

Mar 16 03:16:17 dev-new kernel: [ 3222.339365] doveconf[6420]: segfault at 200 ip 00007fa041b25a03 sp 00007ffe7881e070 error 4 in libc-2.19.so[7fa041ada000+1bb000]

Manually running the daemon resulted in:

# /usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
Segmentation fault

So

mkdir /var/core
chmod 1777 /var/core
echo "/var/core/%p" > /proc/sys/kernel/core_pattern
ulimit -c unlimited
/usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf
Segmentation fault (core dumped)

Better. We have a core file now.

But:

Continue reading "Dovecot segfaulting (Ubuntu 14.04 LTS, CentOS 6 and 7)"

Netatalk 3.1.8 .debs for Debian Jessie available (Apple Timemachine backup to Linux servers)

Debian

The Debian Netatalk3 saga continues at bug #685878. In season 4 of the epic the main issue still seems to be unclear license indications of a (very) few source files. And the usual "you go fix it", "no! you go fix it!". May be the fact that Firefox will be Firefox again in Debian [yeah!] could serve as an inspiration to the Netatalk maintainers?

Ah, well, until we have the eureka moment for Netatalk3 (4?) ...
<pragmatism style="priority-on-users:yes"> ... I'll post my .debs of the new 3.1.8 version of Netatalk as well.

Warning: Read the original blog post before installing for the first time. Be sure to read the original blog post if you are new to Netatalk3 on Debian Jessie!
You'll get nowhere if you install the .debs below and don't know about the upgrade path. So RTFA.

The release notes for 3.1.8 don't list anything that makes the update look mandatory but there is a nice compatibility fix for shares also exported via Samba (compatible xattrs handling). And it's faster.

The update instructions (assuming you have installed 3.1.7 before) are:

# install new debs
dpkg -i libatalk17_3.1.8-1_amd64.deb netatalk_3.1.8-1_amd64.deb
# reboot the box (restart of netatalk may not be sufficient)
reboot
# After reboot: remove the obsolete libatalk16 (3.1.8 uses libatalk17)
dpkg -r libatalk16

And here are the files:

Continue reading "Netatalk 3.1.8 .debs for Debian Jessie available (Apple Timemachine backup to Linux servers)"

Cygwin automatic updates

IT

Cygwin is a fantastic product for people that need to use Microsoft Windows and require some compatibility to Linux (or BSD or UNIX in general).

Unfortunately it is not trivial to keep it updated (and thus safe) as the update process requires downloading the latest installer and then clicking through the package list again and again on every update.
No apt-get update, no emerge --update @world, no dnf update.

But ... the people at Red Hat (who now own Cygwin) are not mean, they are just not good at documenting things :-)...

Be sure to have wget installed via cygwin, you'll need it to fetch the installer automatically.
Then drop the following batch file as cyg_update.bat into C:\cygwin64 (or where you have installed cygwin1):

  1. @echo off
  2. cd /d C:\cygwin64
  3. del /Q cygwinSetup-x86_64.exe
  4. bin\wget.exe --progress=dot -S -N http://cygwin.com/setup-x86_64.exe
  5. move /y setup-x86_64.exe cygwinSetup-x86_64.exe
  6. REM S-1-1-0 is the SID for "Everyone"
  7. icacls cygwinSetup-x86_64.exe /grant *S-1-1-0:RX
  8. cygwinSetup-x86_64.exe --no-desktop --no-shortcuts --no-startmenu --quiet-mode

Once you start the batch (e.g. by double-clicking from Windows Explorer) it will download the latest installer from Cygwin.com and perform a silent update. You need to approve the Windows installer warning as you do with every manual install / update as well. So it's not a no-click update but a one-or-two-clicks update.
Still much better than the click fest without the batch file.

Cygwin update screenshot

Update

01.06.2017: I've changed icacls cygwinSetup-x86_64.exe /grant Everyone:RX to icacls cygwinSetup-x86_64.exe /grant *S-1-1-0:RX which is the SID and not language dependent. 'cause otherwise German Windows would like to see "Jeder" and French "Tous publics", Chinese "任何人", etc. Looking them up on Microsoft's terminology search is quite nice but not really scalable.


  1. If you have installed the 32bit version and/or used a different install path, adjust line 2 of the batch file accordingly. Did I need to say that? Hm, well, I did ... have a cookie. 

Dieselgate Emissions cheating presentation at CCC Congress (32C3)

Strategy

Jake Edge of LWN wrote an excellent article on the presentation Felix and me gave at the 32C3 annual CCC congress in Hamburg, Germany on December 27th, 2015.

If you have an hour to spare, you can watch the video and/or browse the slides.

Video of Daniel and Felix' talk

IPv6: Getting rid of the dreaded "Neighbour table overflow"

Internet

IPv6 is hard. It has many, many design flaws and the decade where we all ignored it and hoped for the better hasn't helped. So we're now all in on the protocol. Yeah.

One of the design principles is that it tries to be rather stateless in the configuration and "plug and play". But just like P&P in the good old ISA times, it just doesn't always work.

One of the common issues is that Linux bridges in IPv6 just don't work well with the router announcements that try to discover and configure the IPv6 neighbourhood.

The result is a sheer endless amount of "kernel: Neighbour table overflow." lines flooding dmesg and syslog (or journal for those on SystemD).

Oct  4 16:26:06 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: __ratelimit: 1832 callbacks suppressed
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:11 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: __ratelimit: 887 callbacks suppressed
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:16 host-260 kernel: Neighbour table overflow.
Oct  4 16:26:23 host-260 kernel: __ratelimit: 803 callbacks suppressed

Grep -c(ount) on syslog

Lovely. Welcome to a storage DOS waiting to happen.

So first tip: cat /proc/sys/kernel/printk_ratelimit shows you the amount of seconds the ratelimiter suppresses messages. The default is 5 seconds and you can adjust it to more reasonable values in case you get heavily flooded like in the example above. Notice that this will mean your dmesg becomes rather useless as the kernel is not very selective about which messages to suppress.

Now when you google "Neighbour table overflow", you'll find thousands of pages suggesting to increase the arp / lladdr caches and garbage collection (gc) times like so:

# Set ARP cache garbage collection interval
net.ipv4.neigh.default.gc_interval = 3600
net.ipv6.neigh.default.gc_interval = 3600

# Set ARP cache entry timeout
net.ipv4.neigh.default.gc_stale_time = 3600
net.ipv6.neigh.default.gc_stale_time = 3600

# Setup cache threshold for ARP
net.ipv4.neigh.default.gc_thresh1 = 1024
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh3 = 4096

# And the same for IPv6
net.ipv6.neigh.default.gc_thresh1 = 1024
net.ipv6.neigh.default.gc_thresh2 = 2048
net.ipv6.neigh.default.gc_thresh3 = 4096

That helps if and only if you really have 500+ IPv6 neighbours. Unless you have a badly segmented network or run in a university lab, you don't.

Now ... you may be seeing messages like "kernel: vmbr0: Multicast hash table maximum of 512 reached, disabling snooping: eth0" or "kernel: vmbr0: Multicast hash table chain limit reached: eth0" in your dmesg / syslog / journal.

That hints at what is really happening here: The bridge confused the link-local router negotiation and so you get endless ff02:: neighbour routing entries added to your caches until they flow over. So increasing the caches as in the sysctl entries above is basically pasting band-aid over the problem.

ip route show cache table all will show you the tables. With all entries. See if you have too many ff02:: neighbours in there. If so, you should try to add change your /etc/network/interfaces on Debian / Ubuntu similar to this:

iface vmbr0 inet6 static
   address 2a02:0100:1:1::500:1
   netmask 64
   gateway 2a02:0100:1:1::1
   post-up echo 2048 > /sys/class/net/vmbr0/bridge/hash_max
   post-up echo 1 > /sys/class/net/vmbr0/bridge/multicast_snooping
   post-up echo 0 > /proc/sys/net/ipv6/conf/vmbr0/accept_ra

This obviously assumes your bridge is called vmbr0.

Red Hat/CentOS users will need to adjust the config spread throughout multiple files in /etc/sysconfig/network-scripts. The ifup-ipv6 script is a good one to look at and amend.

The increase of the hash_max entry makes your bridge survive the initial storm of (useless) router solicitations.
multicast_snooping is usually off when routing but you may need it to make sure your VMs on the bridge can be reached.
Finally we make sure the bridge does not accept router announcements. Because that is what the host system should handle.

Sometimes you may need to throw in a static route or two to reach the VMs. P&P, you remember ... ip -6 neigh add nud permanent proxy <VM:IPv6:goes::here> dev vmbr0 is your friend. Unfortunately the antidote for the dreaded "Neighbour table overflow" depends on the specific cause. So you'll have to poke around a bit. tcpdump -i eth0 -v ip6 will show you what is on the wire and tcpdump -i vmbr0 -v ip6 what's visible on the bridge.

Generate an indexed list of passwords

Other

Generating an indexed list of passwords without complex perl or python:

pwgen -y 20 30 | nl -w 2 -n rz -s -

Explanation:

pwgen: -y = complex passwords (including symbols) ; 20 = length of password; 30 = number of passwords to generate

nl: -w 2 = zero pad to a width of two characters; -n rz = print right-justified; -s - = use dash as a separator

screenshot of pwgen | nl

Security is hard, open source security unnecessarily harder

IT

Now it is a commonplace that security is hard. It involves advanced mathematics and a single, tiny mistake or omission in implementation can spoil everything.

And the only sane IT security can be open source security. Because you need to assess the algorithms and their implementation and you need to be able to completely verify the implementation. You simply can't if you don't have the code and can compile it yourself to produce a trusted (ideally reproducible) build. A no-brainer for everybody in the field.

But we make it unbelievably hard for people to use security tools. Because these have grown over decades fostered by highly intelligent people with no interest in UX.
"It was hard to write, so it should be hard to use as well."
And then complain about adoption.

PGP / gpg has received quite some fire this year and the good news is this has resulted in funding for the sole gpg developer. Which will obviously not solve the UX problem.

But the much worse offender is OpenSSL. It is so hard to use that even experienced hackers fail.

IRC wallop on hackint

Now, securely encrypting a mass communication media like IRC is not possible at all. Read Trust is not transitive: or why IRC over SSL is pointless1.
Still it makes wiretapping harder and that may be a good thing these days.

LibreSSL has forked the OpenSSL code base "with goals of modernizing the codebase, improving security, and applying best practice development processes". No UX improvement. A cleaner code for the chosen few. Duh.

I predict the re-implementations and gradual improvement scenarios will fail. The nearly-impossible-to-use-right situation with both gpg and (much more importantly) OpenSSL cannot be fixed by gradual improvements and however thorough code reviews.

Now the "there's an App for this" security movement won't work out on a grand scale either:

  1. Most often not open source. Notable exceptions: ChatSecure, TextSecure.
  2. No reference implementations with excellent test servers and well documented test suites but products. "Use my App.", "No, use MY App!!!".
  3. Only secures chat or email. So the VC-powered ("next WhatsApp") mass-adoption markets but not the really interesting things to improve upon (CA, code signing, FDE, ...).
  4. While everybody is focusing on mobile adoption the heavy lifting is still on servers. We need sane libraries and APIs. No App for that.

So we need a new development, a new code, a new open source product. Sadly so the Core Infrastructure Initiative so far only funds existing open source projects in dire needs and people bug hunting.

It basically makes the bad solutions of today a bit more secure and ensures maintenance of decade old crufty code bases. That way it extends the suffering of everybody using the inadequate solutions of today.

That's inevitable until we have a better stack but we need to look into getting rid of gpg and OpenSSL and replacing it with something new. Something designed well from the ground up, technically and from a user experience perspective.

Now who's in for a five year funding plan? $3m2 annually. ROCE 0. But a very good chance to get the OBE awarded.

Keep calm and enjoy the silence

Updates:

21.07.19: A current essay on "The PGP problem" is making rounds and lists some valid issues with the file format, RFCs and the gpg implementation. The GnuPG-users mailing list has a discussion thread on the issues listed in the essay.

19.01.19: Daniel Kahn Gillmor, a Senior Staff Technologist at the ACLU, tried to get his gpg key transition correct. He put a huge amount of thought and preparation into the transition. To support Autocrypt (another try to get GPG usable for more people than a small technical elite), he specifically created different identities for him as a person and his two main email addresses. Two days later he has to invalidate his new gpg key and back-off to less "modern" identity layouts because many of the brittle pieces of infrastructure around gpg from emacs to gpg signature management frontends to mailing list managers fell over dead.

28.11.18: Changed the Quakenet link on why encrypting IRC is useless to an archive.org one as they have removed the original content.

13.03.17: Chris Wellons writes about why GPG is a failure and created a small portable application Enchive to replace it for asymmetric encryption.

24.02.17: Stefan Marsiske has written a blog article: On PGP. He argues about adversary models and when gpg is "probably" 3 still good enough to use. To me a security tool can never be a sane choice if the UI is so convoluted that only a chosen few stand at least a chance of using it correctly. Doesn't matter who or what your adversary is.
Stefan concludes his blog article:

PGP for encryption as in RFC 4880 should be retired, some sunk-cost-biases to be coped with, but we all should rejoice that the last 3-4 years had so much innovation in this field, that RFC 4880 is being rewritten[Citation needed] with many of the above in mind and that hopefully there'll be more and better tools. [..]

He gives an extensive list of tools he considers worth watching in his article. Go and check whether something in there looks like a possible replacement for gpg to you. Stefan also gave a talk on the OpenPGP conference 2016 with similar content, slides.

14.02.17: James Stanley has written up a nice account of his two hour venture to get encrypted email set up. The process is speckled with bugs and inconsistent nomenclature capable of confusing even a technically inclined person. There has been no progress in the last ~two years since I wrote this piece. We're all still riding dead horses. James summarizes:

Encrypted email is nothing new (PGP was initially released in 1991 - 26 years ago!), but it still has a huge barrier to entry for anyone who isn't already familiar with how to use it.

04.09.16: Greg Kroah-Hartman ends an analysis of the Evil32 PGP keyid collisions with:

gpg really is horrible to use and almost impossible to use correctly.

14.11.15:
Scott Ruoti, Jeff Andersen, Daniel Zappala and Kent Seamons of BYU, Utah, have analysed the usability [local mirror, 173kB] of Mailvelope, a webmail PGP/GPG add-on based on a Javascript PGP implementation. They describe the results as "disheartening":

In our study of 20 participants, grouped into 10 pairs of participants who attempted to exchange encrypted email, only one pair was able to successfully complete the assigned tasks using Mailvelope. All other participants were unable to complete the assigned task in the one hour allotted to the study. Even though a decade has passed since the last formal study of PGP, our results show that Johnny has still not gotten any closer to encrypt his email using PGP.

  1. Quakenet has removed that article citing "near constant misrepresentation of the presented argument" sometime in 2018. The contents (not misrepresented) are still valid so I have added and archive.org Wayback machine link instead. 

  2. The estimate was $2m until end of 2018. The longer we wait, the more expensive it'll get. And - obviously - ever harder. E.g. nobody needed to care about sidechannel attacks on big-LITTLE five years ago. But now they start to hit servers and security-sensitive edge devices. 

  3. Stefan says "probably" five times in one paragraph. Probably needs an editor. The person not the application. 

Apple Timemachine backups on Debian 8 (Jessie)

Debian

Upgrading Debian 7 (Wheezy) servers to Debian 8 (Jessie) proves (unexpectedly) quite rough around the edges.

That's what you get for using a version x.0, we should have known better :-).

And - of course - the release notes follow the common practice of not even mentioning any of the issues we encountered so far.

Ah, well, let's go through the first one:

In Debian 7 (Wheezy) there was netatalk 2.2.2 (packages link). Now during upgrades that package may or may not get removed. There is no netatalk in Debian 8 (Jessie) anymore. Duh. There is in sid (aka Debian unstable) (packages link) so we may see a backport some time. Or not. In any case this is still 2.2.5 at the time of writing and as Adrian Knoth put it in the three year old bug asking for a upgrade to Netatalk 3:

Let's not ship another release without netatalk3, it's embarrassing.

Yes. It is. Removing a working version and not even mentioning it in the release notes is even worse though. So no cookies there.

Luckily the absolutely awesome Debian and greater FLOSS community have sorted 90% of the problem out for us already:

A quick Google search turns up an excellent article on netatalk's wiki that details installing Netatalk 3.1.7 on Debian 8 Jessie. There are two shortcomings to this: First it doesn't compile to .debs but installs besides apt and friends. And second it compiles with Spotlight search, courtesy of Gnome tracker, which doesn't really work well on servers yet. Hence we're lucky that Adrian Knoth's debified install has not yet added the tracker dependencies. It does compile for systemd use (the default for Debian 8 Jessie). If you want to continue using SysVInit, you need to modify debian/rules.

Compiling to .debs becomes as easy as:

# get build dependencies and a few helpers
apt-get install build-essential devscripts debhelper cdbs autotools-dev dh-buildinfo libdb-dev libwrap0-dev libpam0g-dev libcups2-dev libkrb5-dev libltdl3-dev libgcrypt11-dev libcrack2-dev libavahi-client-dev libldap2-dev libacl1-dev libevent-dev d-shlibs dh-systemd
# in case you want to try the tracker support (you need to ammend the debian/ build config as well)
# apt-get install tracker libtracker-sparql-1.0-dev libtracker-miner-1.0-dev  
git clone https://github.com/adiknoth/netatalk-debian
cd netatalk-debian
debuild -b -uc -us

This should leave you with (at the time of writing this):

File Function md5 sha1
libatalk-dev_3.1.7-1_amd64.deb Development files for the libatalk library (dev only) e5a465e39a8560c919d8db85c8e5a83b 0b924cf75f22ab42406289c6f18ae0243d6396a3
libatalk16_3.1.7-1_amd64.deb libatalk library (needed) 17a3d677ed0b3df1c2f4c1a8ab9045fd 7345ed3edd442716c99c2fe979140703204c0826
netatalk_3.1.7-1_amd64.deb netatalk daemons (needed) c694abca7f3cdc0070b2b3e7d528324a 932d1e3d5899958f29e79a7ba40e858d4ac272e8

Obviously you can download the files above if you run the AMD64 architecture and trust me enough to compile them for you.

Continue reading "Apple Timemachine backups on Debian 8 (Jessie)"

Managing a project consisting of multiple git repositories

IT

The core team organizing DebConf, the annual Debian developer conference, reached out to me two weeks ago to help support this year's effort a bit.

I'm very happy to do so as Debian is a cornerstone of everything I do in the Open Source/Free Software space.

Screenshot of git_pull_all with color

To get me started I got access to a lot of mailing lists and irc channels. And even more git repositories. So many that the DebConf team even has an instruction page on how the repositories all fit together.

It's unfortunately quite common to split a bigger project into many git repositories to ease access rights management and reduce the noise and data transfer volume for the average user. The downside is, everybody ends up with a dozen or more individual repositories to keep pulling. And then there's git annex for yet another level of indirection.

Joey Hess, a former Debian developer, has even written an extensive tool, myrepos, to meta-manage the different repositories and it can do quite some magic across different SCMSs1. In my case this is a bit of an overkill though.

And using myrepos may get you confused at some point whether to now run mr or git directly for each batch of repos you have inherited over some time of working on multiple projects.2 Thus I prefer the simple route:

Check out each repository into a common top-level directory (~/debconf/ in this case) and then put the following two lines into an executable script git_pull_all into that top level directory:

#!/bin/sh
find ~/debconf -mindepth 1 -maxdepth 1 -type d -exec sh -c "cd {}; test -r .git/config && git pull $*" \;

This will allow you to pull all git repos with one command and keep the normal syntax for everything else you do with each repo.

The --mindepth and --maxdepth will instruct find to just go and run your git pull only inside each direct child of the top level directory. So recursion depth = 1. That is the single trick there is to this.

Updates:

If you like to have some color and a bit of a spaced layout for improved readability, try:

#!/bin/sh
find ~/debconf -mindepth 1 -maxdepth 1 -type d -exec sh -c "cd {}; test -r .git/config && (printf \"\033[1m\033[34m%-50s\033[0m\" \"\${PWD}:\" ; git pull $*)" \;

When you have pull.rebase=true set in your .gitconfig, you can run ./git_pull_all --no-rebase to avoid rebases in case you work somewhere and want to have the merge commits.

P.S.: The DebConf15 Heidelberg registration just opened, please check the DebConf15 homepage for news, venue information and please register if you want to come around.


  1. Source Code Management Systems, like git, mercurial (hg) or subversion (svn). Or God forbid ... cvs. I don't like the (D)VCS (Distributed) Version Control Systems moniker. Because that's not really all these systems do. Not even the most important piece of what they do these days. 

  2. With myrepos you can still work with each individual repository via git. Just so nobody will write in "but...". 

scp (secure copy) a file to the same path on a remote system

IT

I've been copying files from one server to the symmetrical cluster partner a lot last week.

It's always

 scp /path/to/directory/file remote:/path/to/directory/

That gets boring after a while but does not really warrant setting up a full blown config management solution like salt, puppet or ansible.

So here is scpover[1.5kB].

It reduces the effort to

 scpover /path/to/directory/file

This will copy "file" from "/path/to/directory/" into exactly the same directory on the remote cluster partner. Which system to target is detected from the hostname of the local system and hard-coded into the script.

Scpover can also take multiple filepaths on one command line so you can beam over a few related config files from multiple locations in one go.
Not an atomic change but at least all within a reasonably short timespan.

Continue reading "scp (secure copy) a file to the same path on a remote system"

How much memory does a process use on Linux?

Linux

Sometimes the easy questions are the hardest to answer.

Memory can mean RSS (Resident Set Size) which is the memory of a process held in RAM (so not swapped out). That does include shared memory allocations. So if you add two RSS numbers, you're probably wrong already. Still this is usually the number we look for in most practical investigations.

Then there is VSZ (Virtual Set siZe) also called SIZE. The VSZ includes code, data and stack segments a process has allocated. And again that will count some shared address space. So usually bash will have a VSZ that's lower than its RSS.

man ps will also tell you:

   The SIZE and RSS fields don't count some parts of a process including the page tables, kernel stack, struct
   thread_info, and struct task_struct.  This is usually at least 20 KiB of memory that is always resident.

In most (if not all) practical scenarios that difference won't matter. If it were, you'd be using valgrind to look into the memory usage of your application in minute detail. Wouldn't you?

If you want to have an as-detailed-as-possible look into the memory allocations of a process pmap <pid> will give you the information. The summary at the end is a gross over-estimation of the total memory a process has allocated as it counts all mapped memory (and may still be wrong due to de-duplication and other factors). But that number may well serve as an upper bound if you need something like that.

For running processes


ps -eo 'pid user rss:8 size:8 cmd' --sort 'rss'
 

will give you a nice sorted list of processes and their RSS and VSZ (SIZE) in kiB (old school kB...).

For short running commands GNU time (not the bash build-in time command, apt install time on Debian-based systems) has a nice capability that's not widely known yet:


/usr/bin/time -f "RSS: %MkiB" <command>
 

will tell you the maximum RSS size the <command> has had during its lifetime. That's better than top or watch ps and trying to spot the process.