Skip to content

How much memory does a process use on Linux?

Linux

Sometimes the easy questions are the hardest to answer.

Memory can mean RSS (Resident Set Size) which is the memory of a process held in RAM (so not swapped out). That does include shared memory allocations. So if you add two RSS numbers, you're probably wrong already. Still this is usually the number we look for in most practical investigations.

Then there is VSZ (Virtual Set siZe) also called SIZE. The VSZ includes code, data and stack segments a process has allocated. And again that will count some shared address space. So usually bash will have a VSZ that's lower than its RSS.

man ps will also tell you:

   The SIZE and RSS fields don't count some parts of a process including the page tables, kernel stack, struct
   thread_info, and struct task_struct.  This is usually at least 20 KiB of memory that is always resident.

In most (if not all) practical scenarios that difference won't matter. If it were, you'd be using valgrind to look into the memory usage of your application in minute detail. Wouldn't you?

If you want to have an as-detailed-as-possible look into the memory allocations of a process pmap <pid> will give you the information. The summary at the end is a gross over-estimation of the total memory a process has allocated as it counts all mapped memory (and may still be wrong due to de-duplication and other factors). But that number may well serve as an upper bound if you need something like that.

For running processes


ps -eo 'pid user rss:8 size:8 cmd' --sort 'rss'
 

will give you a nice sorted list of processes and their RSS and VSZ (SIZE) in kiB (old school kB...).

For short running commands GNU time (not the bash build-in time command, apt install time on Debian-based systems) has a nice capability that's not widely known yet:


/usr/bin/time -f "RSS: %MkiB" <command>
 

will tell you the maximum RSS size the <command> has had during its lifetime. That's better than top or watch ps and trying to spot the process.

Encrypting files with openssl for synchronization across the Internet

Linux

Well, shortly after I wrote about encrypting files with a keyfile / passphrase with gpg people asked about a solution with openssl.

You should prefer to use the gpg version linked above, but if you can't, below is a script offering the same functionality with openssl.

You basically call crypt_openssl <file> [<files...>] to encrypt file to file.aes using the same keyfile as used in the gpg script (~/.gnupg/mykey001 per default).

A simple crypt_openssl -d <file.aes> [<files.aes...>] will restore the original files from the encrypted AES256 version that you can safely transfer over the Internet even using insecure channels.

Please note that you should feed compressed data to crypt_openssl whenever you can. So use preferably use it on .zip or .tar.gz files.

Continue reading "Encrypting files with openssl for synchronization across the Internet"

Encrypting files with gpg for synchronization across the Internet

Linux

Automatically transferring (syncing) files between multiple computers is easy these days. Dropbox, owncloud or bitpocket to name a few. You can imagine I use the latter (if you want a recommendation)1.

In any case you want to encrypt what you send to be stored in "the cloud" even if it is just for a short time. There are many options how to encrypt the "in flight" data. Symmetric ciphers are probably the safest and most widely researched cryptography these days and easier to use than asymmetric key pairs in this context as well.

Encryption is notoriously hard to implement correctly and worthless when the implementation is flawed. So I looked at gpg, a well known reference implementation, and was amazed that it can neither use a proper keyfile for symmetric encryption (you can just supply a passphrase via --passphrase-file) nor does it handle multiple files on the command line consistently. You can use --multifile (wondering...why does a command need that at all?) with --decrypt and --encrypt (asymmetric public/private key pair encryption) but not with --symmetric (symmetric shared key encryption). Duh!

With a bit of scripting around the gpg shortcomings, you end up with crypt_gpg that can nicely encrypt or decrypt multiple files (symmetric cipher) in one go.


  1. Dropbox is closed source so it cannot be assessed for its security. Owncloud needs a thorough code review before I would dare to run it on my systems. 

Continue reading "Encrypting files with gpg for synchronization across the Internet"

Securing the grub boot loader

Open Source

Since version 2.0 the behaviour of grub regarding passwords has changed quite substantially. It can be nicely used to secure the boot process so that a X display manager (gdm, kdm, lightdm, ...) or login prompt cannot be circumvented by editing the Linux kernel boot command line parameters. The documentation is concise but many old how-tos may lead you down the wrong GNU grub "legacy" (the pre-2.0 versions) path.

So this assumes you have a grub installed and working. I.e. if you press Shift during boot, you get a grub menu and can edit menu entries via the e key.

First you need to setup grub users and corresponding passwords:

Run grub-mkpasswd-pbkdf2 to encrypt every password you want to use for grub users (which are technically unrelated to Linux system users at this time).
You'll get a string like 'grub.pbkdf2.sha512.10000...'. It will replace the plain text passwords.

In '/etc/grub/40_custom' add lines like:

# These users can change the config at boot time and run any menuentry:
set superusers="root user1"
password_pbkdf2 root grub.pbkdf2.sha512.10000.aaa...
password_pbkdf2 user1 grub.pbkdf2.sha512.10000.bbb...
# This user can only run specifically designated menuentries (not a superuser):
password_pbkdf2 user2 grub.pbkdf2.sha512.10000.ccc...

Now once you did this grub v. 2.0+ will ask for a supervisor password every time you want to boot any menu item. This is a changed behavior from v. 1.9x which defaulted to allow all entries if no user restriction was specified. So you need to add '--unrestricted' to all 'menuentries' that any user shall be able to boot. You can edit '/boot/grub/grub.cfg' and add --unrestricted to (the default) menuentries. Or you can edit the 'linux_entry ()' function in '/etc/grub/10_linux' so that the 'echo "menuentry ..."' lines include --unrestricted by default:

[...]
echo "menuentry '$(echo "$title" | grub_quote)' --unrestricted ${CLASS} \$menuentry_id_option 'gnulinux-$version-$type-$boot_device_id' {" | sed "s/^/$submenu_indentation/"
else
echo "menuentry '$(echo "$os" | grub_quote)' --unrestricted ${CLASS} \$menuentry_id_option 'gnulinux-simple-$boot_device_id' {" | sed "s/^/$submenu_indentation/"
[...]

Make a backup of this file as it will be overwritten by grub updates. This way all Linux kernels detected by the script will be available to all users without identifying to grub via username / password.

Now issue update-grub to re-generate 'grub.cfg' with the amended menuentries.

If everything worked well, your system can now be booted unrestricted but the grub configuration can only be changed from the grub superusers after identifying with their username and password at the grub prompt.

Bonus point:

If you want to create menuentries that user2 (and any superuser) from the above example user list can run, add blocks like these to the end of '40_custom':

menuentry "Only user2 (or superuser) can run this Windows installation" --users user2 {
set root=(hd1,1)
chainloader +1
}

Update

16.12.2015:
Hector Marco and Ismael Ripoll have found a nearly unbelievable exploit in Grub2 that allows you to tap backspace 28 times to get a rescue shell and that way bypass a password prompt. Time to update!
Read the excellent analysis of the bug and the exploit vector in Hector Marco's blog post.

Creating iPhone/iPod/iPad notes from the shell

Open Source

I found a very nice script to create Notes on the iPhone from the command line by hossman over at Perlmonks.

For some weird reason Perlmonks does not allow me to reply with amendments even after I created an account. I can "preview" a reply at Perlmonks but after "create" I get "Permission Denied". Duh. vroom, if you want screenshots, contact me on IRC :-).

As I wrote everything up for the Perlmonks reply anyways, I'll post it here instead.

Against hossman's version 32 from 2011-02-22 I changed the following:

  • removed .pl from filename and documentation
  • added --list to list existing notes
  • added --hosteurope for Hosteurope mail account preferences and with it a sample how to add username and password into the script for unattended use
  • made the "Notes" folder the default (so -f Notes becomes obsolete)
  • added some UTF-8 conversions to make Umlauts work better (this is a mess in perl, see Jeremy Zawodny's writeup and Ivan Kurmanov's blog entry for some further solutions). Please try combinations of utf8::encode and ::decode, binmode utf8 for STDIN and/or STDOUT and the other hints from these linked blog entries in your local setup to get Umlauts and other non-7bit ASCII characters working. Be patient. There's more than one way to do it :-).

I /msg'd hossman the URL of this blog entry.

Continue reading "Creating iPhone/iPod/iPad notes from the shell"

Apple iPhone ring tones Linux style

Open Source

Apple has crippled the iPhone to not allow normal music files as ringtones. Business decision. Technically any sub 40 second MP4 audio file will do once you rename it to *.m4r and drag-and-drop it to the ringtones folder of your phone in iTunes. Longer ones will work, too. But you'd need a jailbroken iPhone for that as iTunes will refuse to transfer the ringtone file if it's too long. Not much of an issue imho, who keeps ringing your phone for 40 seconds or more?

There's a gazillion websites available telling you how to convert a single .mp3-file to a ringtone with or without iTunes help and there are hundreds of tools doing that for you if you can't find out how to do it with just iTunes itself. Still the ones I tried failed for me as I wanted to convert my 20 or so standard ringtones from the good old Motorola K3 to iPhone ringtones all in one go. Without having to edit each one by hand. They are already nice ringtones and have served me well for years, just too long for the iPhone and in .mp3 format.

The basic processing sequence needed is

  1. Cut the .mp3 down to 39s
  2. Convert the .mp3 -> .wav (with mplayer, normalize output gain while we're at it)
  3. Convert the .wav -> .mp4 (with facc)
  4. Clean up, GOTO 1 for next file

So below is the free shell script to create multiple ringtones in one go on any Linux system. You need to install cutmp3, mplayer and faac for it, so apt-get install cutmp3 mplayer faac on Debian or Ubuntu. cutmp3 is currently not in the portage tree for Gentoo, but you can download an ebuild from Polynomial-C's overlay (mirror). Or you just download the cutmp3 binary from Jochen Puchalla's homepage. There's no error checking in the script, so know your way around the shell before running it.

Without further ado:

#!/bin/sh
#
# convert_to_ringtone file1.mp3 [file2.mp3, ...]
# Placed into the public domain by Daniel Lange, 2011.
#

for arg
do
        echo "Processing $arg..."
        cutmp3 -c -a 0:0.0 -b 0:39.0 -i "$arg" -O "$arg.tmp"
        mplayer -vo null -vc null -af volnorm -ao pcm:fast:file=tmpfile.wav "$arg.tmp"
        faac -b 128 -c 44100 -w tmpfile.wav
        name=`echo $arg| sed 's/.mp3//g'| sed 's/ /_/g'`
        mv tmpfile.m4a $name.m4r
        rm tmpfile.wav
        rm "$arg.tmp"
        echo "$arg done."
done
 

Wikipedia article on Apple's 1984 ad.

Update

23.12.14 Apparently the faac package in Debian and Ubuntu has had the MP4 writing capability removed in v1.28-5 and later due to a minor license incompatibility. See the Debian Changelogs. Duh.

faac (1.28-5) unstable; urgency=low
  [ Andres Mejia ]
  * Disable mp4v2 support.
    This only disables mp4v2 for the faac utility program. The faac
    utility is GPL-2 but the mp4v2 library is MPL-1.1. The two licenses
    are incompatible with each other.

So ... unfortunately you have to built faac from source yourself or pin the v1.28-4 version which is identical except for the castration anyways.

Random distro dev: "Why oh why doesn't my distro ever head mainstream...?"
Hint: Because of stuff like this.

Binding applications to a specific IP

Linux

These days many systems are multi-homed in the sense that they have more than one IP address bound at the same time.
I.e. for different network cards, virtual IPs for shared servers or just using WiFi and a wired network connection at the same time on a laptop.

Murphy of course makes sure that your system will choose to worst IP (i.e. that on slow WiFi or the one reserved for admin access) when an application does not specifically supports binding to a selected IP address. And Mozilla Firefox for example doesn't.

The kernel chooses an outgoing IP from those in the routing table with the same metric:

daniel@server:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.0.2.1         0.0.0.0         U     0      0        0 eth0
0.0.0.0         192.0.2.2         0.0.0.0         U     0      0        0 eth1
0.0.0.0         192.0.2.3         0.0.0.0         U     0      0        0 eth2
0.0.0.0         192.0.2.4         0.0.0.0         U     0      0        0 eth3

You can obviously play around with the metric and make the kernel router prefer the desired interface above others. This will affect all applications though. Some people use the firewall to nat all packages to port 80 onto the network interface desired for web browsing. Gee, beware the http://somewebsite.tld:8080 links...

Thankfully Daniel Ryde has solved the problem via a LD_PRELOAD shim. With his code you can run

daniel@laptop:~$ BIND_ADDR="192.0.2.100" LD_PRELOAD=/usr/lib/bind.so firefox (*)

and happily surf away.

To compile his code (3.3kB, local copy, see note 1) you need to run

gcc -nostartfiles -fpic -shared bind.c -o bind.so -ldl -D_GNU_SOURCE
strip bind.so
cp -i bind.so /usr/lib/

and you're set to go.

If you don't have gcc available (and trust me) you can download pre-compiled 32bit and 64bit (glibc-2) bind.so libraries here (4.5kB).

I guess because Daniel Ryde hid his code so well on his webpage, Robert J. McKay wrote another LD_PRELOAD shim, called Bindhack (4.5kB, local mirror). This will - as is - only compile on 32bit machines. But YMMV.

Run the above command (*) with your desired (and locally bound) IP address in bash and visit MyIP.dk or DNStools.ch or any of the other services that show your external IP to see whether you've succeeded.

Notes:

  1. Daniel Ryde did not specify the -D_GNU_SOURCE in the comments section of bind.c. Modern glibc/gcc need that as he used RTLD_NEXT which is Unix98 and not POSIX. I amended the local copy of bind.c and sent him an email so he can update his.
  2. Both are IPv4 only, no IPv6 support.

Updates:

19.03.15 madmakz wrote in to clarify that all of the bind LD_PRELOAD shims only work with TCP connections. So not with UDP.
I'm not aware of a shim that manipulates UDP sockets.

14.01.14 Christian Pellegrin wrote a superb article on how to achieve per-application routing with the help of Linux network namespaces.

16.06.13 showip.be seems to be gone, so I replaced it with dnstools.ch in the text above. There are plenty of others as well.

22.06.12 Lennart Poettering has a IPv4 only version of a shim and a rather good readme available at his site.

29.11.10 Catalin M. Boie wrote another LD_PRELOAD shim, force_bind. I have not tested this one. It's capable of handling IPv6 binds.

11.01.09 Daniel Ryde has replied to my email and updated his local copy now as well.

Ubuntu Karmic 9.10 Bluetooth UMTS Dial-up (DUN)

Linux

Using a mobile phone's Bluetooth Dial-up network (DUN) to connect to the Internet (UMTS/GPRS) while on the road is quite convenient for me. Sadly so this is not supported out-of-the-box in Ubuntu Karmic 9.10 (Netbook Remix) as it uses Network-Manager to handle - well - network connections. And that is not quite there on Bluetooth managed devices yet.

While the default solution (rfcomm and Gnome-PPP) still works, it's ugly to set up. Sadly so, zillions of Ubuntu-Forum threads and blog entries still detail this solution - or the issues encountered with it along the way.

The much better solutions is using Blueman, an improved Gnome-Bluetooth primarily developed by Valmantas Palikša. It brings the right UDEV magic along to teach Network-Manager about the Bluetooth devices it handles.

Blueman Screenshot on Ubuntu Karmic 9.10 Netbook Edition

Just follow the steps on their downloads page to set up the Blueman PPA (Personal Package Archive) to get things working.

Kubuntu 9.10 (karmic) 64bit firefox java plugin

Linux

For some unknown reason the (K)Ubuntu developers did not update the Java plugin for firefox after jaunty (yet?).

The version that Karmic (9.10) pulls out of the multiverse repository is still jaunty's (9.04).

So when you try:

apt-get install sun-java6-plugin

you'll get something like

   Reading package lists... Done
   Building dependency tree
   Reading state information... Done
   Some packages could not be installed. This may mean that you have
   requested an impossible situation or if you are using the unstable
   distribution that some required packages have not yet been created
   or been moved out of Incoming.
   The following information may help to resolve the situation:
   
   The following packages have unmet dependencies:
     sun-java6-plugin: Depends: sun-java6-bin (= 6-15-1) but 6-16-0ubuntu1.9.04 is to be installed
   E: Broken packages

Duh.

Actually if you have the Java Runtime Environment (JRE, package name sun-java6-jre) installed all files needed are already present.
Just not put in the right place on the filesystem.

So, run:

sudo apt-get install sun-java6-jre   # install JRE if needed
sudo ln -s /usr/lib/jvm/java-6-sun/jre/lib/amd64/libnpjp2.so /usr/lib/mozilla/plugins/

This will install the JRE (if it's not already installed) and will symlink the firefox plugin for java in place so that it'll be found after a browser restart.

Fixing FreeNX / NoMachine NX keyboard glitches (e.g. ALTGr)

Linux

There is a add-on technology to X or VNC called NX by an Italian company called NoMachine. It's quite useful as it speeds up working on remote desktops via slow network connections (i.e. DSL pipes) substantially.

The libraries that implement NX are released under GPLv2 by that company. A server wrapping up the libraries' functionality is available as closed source from NoMachine or as a free product (GPLv2 again) by Fabian Franz, called FreeNX.

FreeNX itself is amazing as it is written in BASH (with a few helper functions in C). It's also able to mend some of the shortcomings of the NX architecture. E.g. stock NX requires a technical user called "nx" to able to ssh into the NX server with a public/private keypair. FreeNX can work around that for more secure set-ups.

One issue I bumped into quite regularly with Linux clients and Linux hosts from different distributions/localisations is that the keymaps are not compatible. This usually results in the ALTGr key not usable, so German keyboard users can't enter a pipe ("|"), tilde ("~") or a backslash ("\") character. Also the up and down keys are usually resulting in weird characters being pasted to the shell. Now all of that makes using a shell/terminal prompt quite interesting.

Continue reading "Fixing FreeNX / NoMachine NX keyboard glitches (e.g. ALTGr)"

Getting dual-screen (xinerama) to work with Matrox G450/550 graphics cards and Xorg 1.5

Gentoo

Gentoo finally decided to update Xorg to 1.5. Because this has very substantial changes against the previous version, some things break and there is a migration guide that you are nagged to read. After the upgrade I found that the Matrox card in one of my servers would not display xinerama anymore, i.e. I would get the same image on both screens only. This is the default behaviour for the stock Xorg mga driver. It needs a proprietary HALlib to get real dual-screen capabilities. Whilst there are a few unstable ebuilds for x11-drivers/xf86-video-mga none worked for me any better with Xinerama. The Gentoo Changelog is useless as usual. (Gentoo ebuild ChangeLogs tend to never really tell what is fixed, if you're lucky they reference a bug with a good description. But that's only if you're really lucky.)

Worse, that driver hasn't been updated by Matrox anymore since mammals took over the earth (figuratively ... 2005). This is the typical unmaintained-closed-source-drivers-make-hardware-obsolete-sooner-than-later story. Luckily the cards are quite widely used and clever people from the Open Source community have written guides (Tuxx-Home, Fkung) on how to dissect the proprietary driver and combine parts of it with the Open Source version so that it can be linked into recent X servers. Unfortunately because of the architectural changes in Xorg 1.5, following these guides will fail at the compile stage.

In the Matrox Forum of Alexander Griesser, the author of the first comprehensive Matrox driver install guide linked above, people currently mostly downgrade to previous Xorg versions to work around the issue.

But there is a better^Hworking solution already emerging :-P ...

Continue reading "Getting dual-screen (xinerama) to work with Matrox G450/550 graphics cards and Xorg 1.5"

httpdate - set local date and time from a web server

Linux

While ntp may be a great protocol, I find it quite bloated and slow for the simple purpose of just setting a local date and time to a reference clock. I do not need 20ms accuracy on a notebook's clock :-). Thus I use(d) rdate for a decade now but the public rdate servers are slowly dying out. So I'm replacing it more and more with htpdate which works quite nicely. It's written in C and a perl alternative is available on the author's site. There is also a forked windows version of it available.

Developing a bit larger bash script (which syncs a few servers), I wondered whether I could realize the time sync part in bash as well.

It's quite possible:

  1. # open a tcp connection to www.google.com
  2. exec 3<>/dev/tcp/www.google.com/80
  3. # say hello HTTP-style
  4. echo -e "GET / HTTP/1.0\n\n">&3
  5. # parse for a Date: line and with a bit of magic throw the date-string at the date command
  6. LC_ALL=C LANG=en date --rfc-2822 --utc -s "$(head <&3 | grep -i "Date: " | sed -e s/Date\:\ //I)"
  7. # close the tcp connection
  8. exec 3<&-

Simple, eh?

Continue reading "httpdate - set local date and time from a web server"

kloeri announces Exherbo, another source based Linux distribution

Linux

Bryan Østergaard (aka kloeri) announced Exherbo today. He assembled a team of (ex-)Gentoo developers including Ciaran McCreesh (ciaranm), Richard Brown (rbrown), Fernando J. Pereda (ferdy) and Alexander Færøy (eroyf) to build a new source based Linux distribution.

They would like to overcome some of the short-commings of Gentoo both from a technical as well as from a community perspective. Obviously this is easily said and hard to really achieve, so time will tell how successful that team can be. Renaming USE-Flags to OPTIONS and merging the platform KEYWORDS (like x86, ~x86) into the Options-logic is no big deal, but getting the thousands of ebuilds^Hpackages better supported and maintained than Gentoo will be the real deal{maker|breaker}.

Paludis, ciaranm's package manager, supports Gentoo ebuilds and can import them into Exherbo, so there is a potential migration path sketched out.*

They also add another init-system re-write ("Genesis") to the pool. An already quite crowded pool with rather shallow water, I may add.

Exherbo has nothing that is end-user-safe at the time of the announcement, so it's safe to assume kloeri's team wants to attract further development capacity :-).

Browse around the website or join folks in #exherbo if you're interested.

I asked in #exherbo what "exherbo" means ... latin for "uproot" was the answer. How fitting.

Updates

*19.04.08: Two friendly folks wrote in to clarify that Paludis currently can only import Ebuild-builds into Exherbo via importare, i.e. take a Gentoo build result and package it for importing into the Exherbo system through Paludis.
23.05.08: Ciaranm wrote a blog entry how to get build results into Exherbo/Paludis via importare.

Seredipity default event_s9ymarkup plugin breaking URLs that contain underscores

Serendipity

The default Serendipity mark-up plugin (event_s9ymarkup) currently breaks URLs that contain underscores.

So

http://en.wikipedia.org/wiki/Statler_%26_Waldorf

will end up

http://en.wikipedia.org/wiki/Statler</u>%26_Waldorf

because of a faulty regex. Garvin Hicking does not really want to fix this. (See this s9y support forum article for arguments pro/contra fixing it). So if you encounter this problem, your options are:

  • replace _ in URLs with %5F (aka manually urlencode it)
  • remove the plugin or disable it
  • patch the plugin

Patching is basically changing

plugins/serendipity_event_s9ymarkup/serendipity_event_s9ymarkup.php:

$text = preg_replace('/\b_([\S ]+?)_\b/','<u>\1</u>',$text);

to

$text = preg_replace('/\ _([\S ]+?)_\ /',' <u>\1</u> ',$text);

If you want to be writing things like "Haha[lol]" (which I have no real use for ...), extend the "\ " with whatever you'd like to be o.k. to delimit bolded words beyond blanks. It should only be symbols that are not valid in URLs (so none of "$-_.+!*'()," which are all valid in URLs according to RFC 1738).

You may also want to consider replacing one underscore ("_") with two or more ("__") to make the detection, that you actually wanted to write bold text, more reliable.

SSHd chroot and PAM

Gentoo

SSH with chroot patch has been working fine for a number of years. Since PAM v0.99 things have broken though, if users are chrooted with the "/home/username/./" syntax as their homedir.

SSH sessions will just terminate immediately after successful logon. Doh.

Two solutions exist:

  1. Put UsePAM no into /etc/ssh/sshd_config and use the chroot patch and /./ in users homedirs
  2. Keep UsePAM yes. Emerge sys-auth/pam_chroot and add session required pam_chroot.so to /etc/pamd.d/sshd setup /etc/security/chroot.conf or add a chroot_dir=/home/username/ to the pam_chroot.so line.
    This will currently not work for amd64 though as the Gentoo bug regarding pam_chroot has not cought any attention from the arch testers. Since July...

Bugging the arch testers in #Gentoo-amd64 didn't help either:

Continue reading "SSHd chroot and PAM"