Encrypting files with gpg for synchronization across the Internet
Automatically transferring (syncing) files between multiple computers is easy these days. Dropbox, owncloud or bitpocket to name a few. You can imagine I use the latter (if you want a recommendation)1.
In any case you want to encrypt what you send to be stored in "the cloud" even if it is just for a short time. There are many options how to encrypt the "in flight" data. Symmetric ciphers are probably the safest and most widely researched cryptography these days and easier to use than asymmetric key pairs in this context as well.
Encryption is notoriously hard to implement correctly and worthless when the implementation is flawed. So I looked at gpg, a well known reference implementation, and was amazed that it can neither use a proper keyfile for symmetric encryption (you can just supply a passphrase via --passphrase-file) nor does it handle multiple files on the command line consistently.
You can use --multifile (wondering...why does a command need that at all?) with --decrypt and --encrypt (asymmetric public/private key pair encryption) but not with --symmetric (symmetric shared key encryption). Duh!
With a bit of scripting around the gpg shortcomings, you end up with crypt_gpg that can nicely encrypt or decrypt multiple files (symmetric cipher) in one go.
-
Dropbox is closed source so it cannot be assessed for its security. Owncloud needs a thorough code review before I would dare to run it on my systems. ↩
#!/bin/bash
# crypt_gpg v0.1
# Copyright (c) 2013 Daniel Lange, http://daniel-lange.com.
# Released into the public domain. NO LIABILITY ACCEPTED WHATSOEVER. USE AT YOUR OWN RISK.
# CRYPT_KEY may be max. 2400 (+ a few) bytes
#
CRYPT_KEY="${HOME}/.gnupg/mykey001"
METHOD="--symmetric"
if (($# == 0)) || [[ "$1" == "--help" ]] || [[ "$1" == "-h" ]]; then
echo "Usage: $0 [--decrypt] <filename> [<filename> ...]"
exit 1
fi
if [[ ! -e "$CRYPT_KEY" ]]; then
echo "Error: Cryptographic key does not exist on this system."
exit 2
fi
if [[ "$1" == "--decrypt" ]] || [[ "$1" == "-d" ]]; then
METHOD="--multifile --decrypt"
shift
fi
# --multifile works for decrypt but not for encrypt...
if [[ "$METHOD" == "--symmetric" ]]; then
for FILE in "$@"; do
cat "$CRYPT_KEY" | tr -d "\n\r\000" | gpg $METHOD --cipher-algo AES256 --passphrase-fd 0 --batch --verbose $FILE
done
else
cat "$CRYPT_KEY" | tr -d "\n\r\000" | gpg $METHOD --cipher-algo AES256 --passphrase-fd 0 --batch --verbose $*
fi
Download crypt_gpg (1kB).
Now the file pointed to by CRYPT_KEY is your keyfile, your shared secret that the PCs you want to sync need to have available to be able to decode the .gpg files.
Gpg only reads the first line from the keyfile (as it considers it to be a passphrase). Thus we need to make sure it does not contain \n (line feed) and for the Mac and Windows folks \r (carriage return). The tr command inside the crypt_gpg script above takes care of that but it should be good practice to make sure your keyfile also works around those gpg limitations. I also added \000 (NUL) to the exception list. C code, you know...
So, to generate your keyfile:
First: Ensure your system has enough entropy available for generating such a key.
cat /proc/sys/kernel/random/entropy_avail needs to return >2000 bits or you need to fix your entropy source first. Do this by installing haveged (apt-get install haveged on Debian and Ubuntu) or running guchaos as a one time measure.
Second: Generate a 1024 byte keyfile (128 bytes would be more than enough if you run low on entropy or processing time matters).
It is again critically important to strip line ending and NUL characters from the raw key material (as gpg would consider them to indicate the end of the passphrase).
dd if=/dev/random bs=1 count=1200 | tr -d "\n\r\000" | head -c 1024 > ~/.gnupg/mykey001
chmod 0600 ~/.gnupg/mykey001
Obviously you can make as many keyfiles as you wish for different setups.
Third: transfer the keyfile to all PCs that ought to sync via a secure channel (ssh is fine). The server (dropbox, owncloud or your own ssh, web or ftp server) should not have a copy of the keyfile. This way, even if the .gpg files are compromised, they are impossible to decode with any known means. Notice: Everybody who has access to the keyfile can decode the .gpg files lying around on storage media somewhere, so keep it safe. It is your secret encryption key.
Fourth: Every time you now run crypt_gpg myfile.doc anotherfile.txt /path/to/yetanotherfile.ods this will result in *.gpg files in the same directories as the original files.
Make sure to only add *.gpg files to your sync profile(s) and run a manual synchronization to see the *.gpg files transfer.
Fifth: On the receiving end(s) run crypt_gpg --decrypt *.gpg /path/to/*.gpg after a synchronization has taken place and have the original files reconstructed.
Trackbacks
No Trackbacks
Comments
Display comments as Linear | Threaded
Terence Ezekiel on :
I've read a few good posts here. I certainly value my bookmark to keep your site for revisiting. I am surprise how much you manage to create such an excellent informative site just like that. Kudos.
Stefan Sorgenfrey on :
Very nice. It would be also cool to have a script which is synchronizing a whole local folder with a cloud folder on fly with encryption. Maybe this could be also done with scripting by checking the last modified date. Are there any solutions out there already that can do that? I'd love to do it with gpg though. I'll give it a try and post my solution here.
Daniel Lange on :
This is trivial with eCryptfs. You basically sync the encrypted (.ecryptfs/) folder and "the cloud" basically only sees and syncs the encrypted files. Obviously that does not allow for things like conflict detection. The last one always wins.
https://github.com/DanHerbert/CloudSync is a wrapper around duplicity which in turn uses gnupg. You may be able to extend this into two-way sync.
I'd prefer the eCryptfs option (if I were to use "the cloud" for any personal data at all) but you may have different preferences so give it a try.
If you need something more capable / user friendly, try Seafile or Pydio.
Taylah Hesson on :
Аwesome! It's a truly amazing article, I've got a much clear idea on the topic off from this post. Thank you.
Yassir A. P. on :
This is an interesting read, though I'm still quite new with good encryption workflow/practices. Would you mind to share your thoughts about Cryptomator? https://cryptomator.org/ Seems encrypting a whole directories are trivial with this and can be safely synced to cloud.
Daniel Lange on :
I have not looked at Cryptomator, so I can't share any insights.