random neuron misfires

Sunday, January 17th, 2010 at 3:19 pm | 36,021 views | trackback url
Tags: Debian, Encryption, linux, LUKS, Ubuntu

My daughter recently dropped my computer flat on the floor from a standing height (she’s only 5), catastrophically crashing both hard drives in the laptop. I limped along for a few days because I had to function with it for work, and only just today, decided to rip it apart, back up the data that could be read, put in two new 500GB/7200 drives and reinstall everything from scratch.

The painful part, was that my original configuration was a dm-crypted LUKS volume inside an encrypted LVM container, and mounting the volume without booting to it, is not straightforward.

The first piece was to back up the data on it, as best as I could. That was a bit trickier than I’d originally assumed, because I had to be sure I wasn’t going to “fix” the area of the drive that contained the encryption key that unlocked the encrypted LVM container.

After booting in single-user mode and putting in my passphrase to unlock the volume, I ran:

# tune2fs -l /dev/mapper/galaeus-root

It showed that the filesystem was shut down unclean, and had to be checked:

Filesystem state:         not clean with errors
Errors behavior:          Continue

To get it to continue to check the fs until the end, regardless of state (to mark it as 100% checked and “clean”), I had to change the error state with the following:

# tune2fs -e continue /dev/mapper/galaeus-root

This makes sure that even in the face of errors, it will continue to normal execution if/when I need to boot the volume.

Next, I ran a simple e2fsck on the volume, forcing it to fix anything that it found wrong:

# e2fsck -C0 -f -y /dev/mapper/galaeus-root

That took several hours, as it churned through the bad sectors of the drive, to mark the drive as 100% clean (with lots of damage).

Once that was done, I shut down and rebooted back into single-user mode, and ran the following:

# e2fsck -c -k  -p -f /dev/mapper/galaeus-root

This now goes through and attempts to mark the bad blocks as bad, unreadable, and sets them in the bad-blocks inode. It also attempts to repair the fs (“preen”), with no user intervention (“-f”).

Now I had enough where I could go and back up the data on the volume to an external drive. I tried to boot back into the OS on the drive itself, but the fsck damaged enough of the booting userland that it wasn’t possible, so I had to back the data up with the drive plugged into another machine, externally.

I plugged the drive into an external, USB SATA bridge on another Linux machine here is where the trickery begins. Because the drive contained an encrypted LVM, with an encrypted LUKS volume inside that, with my actual partitions inside that, my host computer could not correctly identify the volumes and the encrypted volumes they contained, so even trying to decrypt it through the Debian/Ubuntu popup dialog, failed:

I had to find out which drive it was with fdisk:

# fdisk -l /dev/sd?

...
Disk /dev/sde: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xc1317447

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1   *           1       60770   488134993+  83  Linux
/dev/sde2           60771       60801      249007+   5  Extended
/dev/sde5           60771       60801      248976   83  Linux

To mount it, I need to use cryptsetup:

# /sbin/cryptsetup luksOpen /dev/sde1 external
Enter passphrase for /dev/sde1:
device-mapper: remove ioctl failed: Device or resource busy
Key slot 0 unlocked.

Now the actual device is accessible under /dev/mapper/external

# stat /dev/mapper/external
  File: `/dev/mapper/external'
  Size: 0               Blocks: 0          IO Block: 4096   block special file
Device: 5h/5d   Inode: 224998      Links: 1     Device type: fc,6
Access: (0660/brw-rw----)  Uid: (    0/    root)   Gid: (    6/    disk)
Access: 2010-01-17 13:33:49.077659138 -0500
Modify: 2010-01-17 13:33:49.077659138 -0500
Change: 2010-01-17 13:33:49.077659138 -0500

Let’s create a place to mount it, such as /mnt/external/

# mkdir -p /mnt/external/

And now let’s mount that there:

# mount /dev/mapper/external /mnt/external/
mount: unknown filesystem type 'LVM2_member'

Oops! So the problem here, is that the encrypted LUKS volume contains an encrypted LVM, which we have to handle next. First, let’s make sure we have the appropriate LVM support and kernel module loaded:

# dpkg -l lvm2
||/ Name           Version        Description
+++-==============-==============-============================================
ii  lvm2           2.02.39-0ubunt The Linux Logical Volume Manager

$ sudo /sbin/modprobe dm-mod

$ /sbin/lsmod | grep dm
dm_crypt               13035  5

Now let’s scan for LVM volumes and choose the right volume group name that we’re looking for:

$ sudo vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "galaeus" using metadata type lvm2

We know the volume name on the laptop drive plugged into the external SATA bridge is “galaeus“, so let’s activate that volume:

$ sudo vgchange -ay galaeus
  2 logical volume(s) in volume group "galaeus" now active

Now we see two volumes, but we’re only really concerned with the one that has our data on it. Let’s list them to be sure:

$ sudo lvs
  LV     VG      Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  root   galaeus -wi-a- 443.65G
  swap_1 galaeus -wi-a-  21.87G

Now we can mount it:

$ sudo mount /dev/galaeus/root /mnt/external/

And now our data is visible once again! But that’s only the system data (/var, /etc, /usr, and so on), I still need to get into my own home directory. Since I couldn’t start X to log in as my normal username, my home directory was still not mounted, because it is encrypted.

Let’s look at the situation (as ‘root’ this time):

# cd /mnt/external/home/desrod/

In here, you’ll see some template data, but no user data. We need to make sure the .ecryptfs symlink points to the mounted directory, and not the local one on the host system that the drive is plugged into for recovery. Remember, we’re still in /mnt/external/home/desrod here:

# rm .ecryptfs && ln -s /mnt/external/var/lib/ecryptfs/desrod .ecryptfs

We need to have a mountpoint, so let’s put that in /mnt/ on the host also:

# mkdir /mnt/priv

And now let’s attempt to mount it. This will prompt you for several questions, all of which I’ve included below (with my own answers in bold; yours may differ):

# mount -t ecryptfs /mnt/external/home/desrod/.Private /mnt/priv
Passphrase:
Select cipher:
 1) aes: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)
 2) blowfish: blocksize = 16; min keysize = 16; max keysize = 56 (not loaded)
 3) des3_ede: blocksize = 8; min keysize = 24; max keysize = 24 (not loaded)
 4) twofish: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)
 5) cast6: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)
 6) cast5: blocksize = 8; min keysize = 5; max keysize = 16 (not loaded)
Selection [aes]: 1
Select key bytes:
 1) 16
 2) 32
 3) 24
Selection [16]: 3
Enable plaintext passthrough (y/n) [n]: n
Enable filename encryption (y/n) [n]: n
Attempting to mount with the following options:
  ecryptfs_unlink_sigs
  ecryptfs_key_bytes=24
  ecryptfs_cipher=aes
  ecryptfs_sig=6536228a683536a4
WARNING: Based on the contents of [/root/.ecryptfs/sig-cache.txt],
it looks like you have never mounted with this key
before. This could mean that you have typed your
passphrase wrong.

Would you like to proceed with the mount (yes/no)? : yes
Would you like to append sig [6536228a683536a4] to
[/root/.ecryptfs/sig-cache.txt]
in order to avoid this warning in the future (yes/no)? : no
Not adding sig to user sig cache file; continuing with mount.

And that’s it! Now my encrypted home directory is mounted under /mnt/priv, with the remainder of the encrypted backup drive mounted under /mnt/external. From here, I can rsync all of the data off to another external drive plugged into another port, and away I go!

When I’m done, I can unmount those mountpoints, and detach the crypted device. That’s the opposite of the luksOpen used before. You simply supply the device name to cryptsetup, and close it:

$ sudo cryptsetup luksClose external

p.s. I’m a bit of a cleanliness nit, so when I’m done and have unmounted those mountpoints, I get rid of /mnt/external and /mnt/priv to keep things tidy on the system. I don’t like temporary files and directories littering the system. I could have mounted this to /tmp and played with the data there, but /tmp/ is world-readable, and would have allowed any other users to access it during the backup phase, so I didn’t.

Last Modified: Sunday, March 6th, 2016 @ 02:37

This entry was posted on Sunday, January 17th, 2010 at 3:19 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.