Virtual Disk Image Tools

Written by NickWal

Virtualization software creates a lot of possibilities, in a wide variety of fields. There are a wide variety of solutions available which are free / open source. These include KVM, XEN, QEMU, User-mode Linux, VirtualBox and others. One can now easily create virtual appliances for specific purposes, a database server, a backup server, repository and or search.

Virtual Machines typically reside inside a virtual hard disk image, and knowing how to manipulate these images can allow one to create and update appliances outside the virtual machine environment, open even more possibilities to scripted creation, update and customization of virtual appliances. In this article I will provide a quick rundown of software I have found useful for preparing and manipulating virtual disk images; some of the tools mentioned below are common tools part of the the core Linux utilities you find in every distribution while others are a little more obscure or rarely used.

Image Creation

parted

Parted can be considered a more modern version of fdisk. Its a disk partitioning tool that can not only create partitions, but also move and resize them too. It also has the advantage that you can script the operations and it will work equally well on devices as well as files. So in your script, you can include something like this to create a virtual disk image with 4 primary partitions on:

#!/bin/bash  
dd if=/dev/zero of=centos.raw bs=1 count=1 seek=40G #create a 40G sparse file  
parted centos.raw mklabel msdos # create the partition table  
parted centos.raw mkpart primary ext2 0G 512m #swap  
parted centos.raw mkpart primary ext2 512m 24G #root  
parted centos.raw mkpart primary ext2 24G 38G #data  
parted centos.raw mkpart primary linux-swap 38G 40G #swap  
parted centos.raw print all #just check what you have done

You can then mount this image file using kpartx (described below)

dd

dd is a very versatile tool for copying large block of information between files, or between devices or between files and devices. Its a core unix tool, and very versatile. You can find an extremely complete guide on its usage by example here: http://www.linuxquestions.org/linux/answers/Applications_GUI_Multimedia/How_To_Do_Eveything_With_DD. This is a kind of dd cookbook.

When discussing virtual disk images, its extremely useful for creating or increasing in size sparse disk images, and also copying image files into loopback devices and so on. In the parted section there is an example of creating a sparse file. Below shows how you might use it to copy from a disk file into a loopback device.

sudo dd if=centos4/data.img of=/dev/mapper/loop0p3 bs=4096 conv=notrunc,noerror

Filesystem Tools

These tools will be extremely familiar to many linux users, and there is plenty of documentation on their use. They are only mentioned here for completeness sake. Since clearly they are necessary for the creation of virtual disk images. You can use any of them directly on disk images, loop back devices or device mapper devices.

  • mkfs - create a file system (e.g. mkfs -t ext3 /dev/mapper/loop0p2)
  • mkswap - set up a swap file or partition
  • e2fsck - check an ext2/3 file system
  • resize2fs - resize an ext2/3 file system

Mounting

kpartx

Kpartx requires Device mapper support in the kernel, and dm userspace tools should be present.

Kpartx is an extremely useful tool for being able to mount partitions that are within a virtual disk image. Virtual Disk immages, can either be files which contain file systems and no partition table. These can be mounted fairly easily with the loopback device using mount -t auto -o loop filename.raw /point or they can be images which contain a partition table and one or more partitions, or they can even contain LVM volumes. With the latter two, both can be read with kpartx. kpartx reads and understands the partition table, so it can know in the virtual image file where each partition begins and ends, and then maps each portion of the file to a different device mapper device using a naming convention like /dev/mapper/loop0p1, /dev/mapper/loop0p2 and so on. If they image file also contains LVM volumes, these will also be mapped in with the device mapper. These sections once mapped can be mounted like normal devices. So for example:

#!/bin/bash  
sudo kpartx -a centos.raw   
sudo mkdir -p /mnt/loop2  
sudo mount -t ext3 /dev/mapper/loop0p2 /mnt/loop2  
sudo mount -t ext3 /dev/mapper/loop0p1 /mnt/loop2/boot

will create the mapping for the partitions in a the file centos.raw which contains a boot partition and the root partition. These can then be mounted normally. The following script will do the reverse and unmount the partitions, remove the mapping in the device mapper and delete the loopback device.

#!/bin/bash  
sudo umount /mnt/loop2/boot/  
sudo umount /mnt/loop2  
sudo kpartx -d centos-oracle.raw

mount –bind

mount –bind is worth a mention, as its particular useful in setting up a chroot environment. mount is typically used to mount devices or loopback devices so the filesystem can be accessed however with mount –bind the intention is slightly different. You can make the same part of the directory tree appear in two different places. When setting up a chrooted environment to be able to update software on a disk image, you might typically do the following:

sudo mount -t auto /dev/mapper/loop0p2 /mnt/loop2  
sudo mount -t auto /dev/mapper/loop0p1 /mnt/loop2/boot  
sudo mount --bind /dev /mnt/loop2/dev  
sudo mkdir -p /mnt/loop2/raw  
sudo mount --bind /workarea/raw /mnt/loop2/raw  
chroot /mnt/loop2 /bin/bash

The first two mount statements are regular mount systems, to mount the root file system, and then the boot file system. The third mount statement used mount –bind to make the /dev file system of the host computer visible as part of the directory tree of the mounted loopback images. The second mount –mind statement, just makes a regular directory within the host filesystem visible also in the mounted loopback images file system. Finally when the chroot command is executed, the chosen parts of the host file system will still be visible. Finally a note of caution here, typically chrooted environments are used to create secure environments that software cannot break out of. Making the host systems /dev file system available in the chrooted environment is not a secure thing to do, and should only be done in special circumstances - such as these, the preparation of a virtual disk image. Even then, cautions should be taken, its still quite easy to hose your system by running grub incorrectly or whatever.

device mapper, lvm

These two tools are just worth a quick mentioned here, as they are sometimes used by the other tools mentioned (e.g. kpartx). However if unlikely you will need to use them directly.

Editing

chroot

linux32

sed - http://www.gnu.org/software/sed/

grub - http://www.gnu.org/software/grub/

Bootstrapping Distributions

debootstrap

rinse

Image Conversion

qemu-img

vboxmanage

Sparse files

zerofree - http://intgat.tigress.co.uk/rmy/uml/index.ht