Virtualization software creates a lot of possibilities, in a wide variety of fields. There are a wide variety of solutions available which are free / open source. These include KVM, XEN, QEMU, User-mode Linux, VirtualBox and others. One can now easily create virtual appliances for specific purposes, a database server, a backup server, repository and or search.
Virtual Machines typically reside inside a virtual hard disk image, and knowing how to manipulate these images can allow one to create and update appliances outside the virtual machine environment, open even more possibilities to scripted creation, update and customization of virtual appliances. In this article I will provide a quick rundown of software I have found useful for preparing and manipulating virtual disk images; some of the tools mentioned below are common tools part of the the core Linux utilities you find in every distribution while others are a little more obscure or rarely used.
Image Creation
parted
- Home page www.gnu.org/software/parted/index.shtml
Parted can be considered a more modern version of fdisk. Its a disk partitioning tool that can not only create partitions, but also move and resize them too. It also has the advantage that you can script the operations and it will work equally well on devices as well as files. So in your script, you can include something like this to create a virtual disk image with 4 primary partitions on:
#!/bin/bash
dd if=/dev/zero of=centos.raw bs=1 count=1 seek=40G #create a 40G sparse file
parted centos.raw mklabel msdos # create the partition table
parted centos.raw mkpart primary ext2 0G 512m #swap
parted centos.raw mkpart primary ext2 512m 24G #root
parted centos.raw mkpart primary ext2 24G 38G #data
parted centos.raw mkpart primary linux-swap 38G 40G #swap
parted centos.raw print all #just check what you have done
You can then mount this image file using kpartx (described below)
dd
dd is a very versatile tool for copying large block of information between files, or between devices or between files and devices. Its a core unix tool, and very versatile. You can find an extremely complete guide on its usage by example here: http://www.linuxquestions.org/linux/answers/Applications_GUI_Multimedia/How_To_Do_Eveything_With_DD. This is a kind of dd cookbook.
When discussing virtual disk images, its extremely useful for creating or increasing in size sparse disk images, and also copying image files into loopback devices and so on. In the parted section there is an example of creating a sparse file. Below shows how you might use it to copy from a disk file into a loopback device.
sudo dd if=centos4/data.img of=/dev/mapper/loop0p3 bs=4096 conv=notrunc,noerror
Filesystem Tools
These tools will be extremely familiar to many linux users, and there is plenty of documentation on their use. They are only mentioned here for completeness sake. Since clearly they are necessary for the creation of virtual disk images. You can use any of them directly on disk images, loop back devices or device mapper devices.
- mkfs - create a file system (e.g. mkfs -t ext3 /dev/mapper/loop0p2)
- mkswap - set up a swap file or partition
- e2fsck - check an ext2/3 file system
- resize2fs - resize an ext2/3 file system
Mounting
kpartx
- Home Page (Part of the Linux multipath-tools) christophe.varoqui.free.fr
Kpartx requires Device mapper support in the kernel, and dm userspace tools should be present.
Kpartx is an extremely useful tool for being able to mount partitions that are within a virtual disk image. Virtual Disk immages, can either be files which contain file systems and no partition table. These can be mounted fairly easily with the loopback device using mount -t auto -o loop filename.raw /point or they can be images which contain a partition table and one or more partitions, or they can even contain LVM volumes. With the latter two, both can be read with kpartx. kpartx reads and understands the partition table, so it can know in the virtual image file where each partition begins and ends, and then maps each portion of the file to a different device mapper device using a naming convention like /dev/mapper/loop0p1, /dev/mapper/loop0p2 and so on. If they image file also contains LVM volumes, these will also be mapped in with the device mapper. These sections once mapped can be mounted like normal devices. So for example:
#!/bin/bash
sudo kpartx -a centos.raw
sudo mkdir -p /mnt/loop2
sudo mount -t ext3 /dev/mapper/loop0p2 /mnt/loop2
sudo mount -t ext3 /dev/mapper/loop0p1 /mnt/loop2/boot
will create the mapping for the partitions in a the file centos.raw which contains a boot partition and the root partition. These can then be mounted normally. The following script will do the reverse and unmount the partitions, remove the mapping in the device mapper and delete the loopback device.
#!/bin/bash
sudo umount /mnt/loop2/boot/
sudo umount /mnt/loop2
sudo kpartx -d centos-oracle.raw
mount --bind
mount --bind is worth a mention, as its particular useful in setting up a chroot environment. mount is typically used to mount devices or loopback devices so the filesystem can be accessed however with mount --bind the intention is slightly different. You can make the same part of the directory tree appear in two different places. When setting up a chrooted environment to be able to update software on a disk image, you might typically do the following:
sudo mount -t auto /dev/mapper/loop0p2 /mnt/loop2
sudo mount -t auto /dev/mapper/loop0p1 /mnt/loop2/boot
sudo mount --bind /dev /mnt/loop2/dev
sudo mkdir -p /mnt/loop2/raw
sudo mount --bind /workarea/raw /mnt/loop2/raw
chroot /mnt/loop2 /bin/bash
The first two mount statements are regular mount systems, to mount the root file system, and then the boot file system. The third mount statement used mount --bind to make the /dev file system of the host computer visible as part of the directory tree of the mounted loopback images. The second mount --mind statement, just makes a regular directory within the host filesystem visible also in the mounted loopback images file system. Finally when the chroot command is executed, the chosen parts of the host file system will still be visible. Finally a note of caution here, typically chrooted environments are used to create secure environments that software cannot break out of. Making the host systems /dev file system available in the chrooted environment is not a secure thing to do, and should only be done in special circumstances - such as these, the preparation of a virtual disk image. Even then, cautions should be taken, its still quite easy to hose your system by running grub incorrectly or whatever.
device mapper, lvm
These two tools are just worth a quick mentioned here, as they are sometimes used by the other tools mentioned (e.g. kpartx). However if unlikely you will need to use them directly.
Editing
chroot
<dl><dd>linux32 </dd></dl>
sed - http://www.gnu.org/software/sed/
grub - http://www.gnu.org/software/grub/
Bootstrapping Distributions
debootstrap
rinse
Image Conversion
qemu-img
vboxmanage
Sparse files
zerofree - http://intgat.tigress.co.uk/rmy/uml/index.ht
