Archive for October 2009
Below a quick summary of the Ubuntu Xen articles available in this blog. These articles where written and tested on Ubuntu Juanty. With the upcoming release of Ubuntu Karmic, things will likely change. The biggest gap in getting Xen to work on Karmic is currently obtaining a functioning xen patched kernel. There are a number of features that Karmic uses in very recent kernels, and there are not patched kernels available yet. I will investigate this and post here when I find out a little bit more. If you really need Xen, stick to Jaunty for now. As a plan B option where I want to use Karmic, I will install virtualbox, below you can find some details about convirting your machines to Sun VirtualBox.
Xen on Ubuntu 9.04 Jaunty
- Running Xen on Ubuntu Intrepid and Jaunty Step by Step guide to using Xen with Ubuntu Intrepid and Juanty - How to Set Up Xen Dom0 on Jaunty Jakalope 9.04
- Installing and Running Xen DomU Jaunty on Dom0 Ubuntu Juanty Installing and Running Xen DomU Jaunty on Dom0 Ubuntu Juanty
- Compiling a Xen Dom0 Kernel for Ubuntu Jaunty Compiling a Xen Dom0 Kernel for Ubuntu 9.04 Jaunty Jakalope
- Deploying DomU Centos 5 with Xen on Ubuntu 9.04 Jaunty Jakalope Howto: Install and run Xen DomU Centos 5 on Dom0 Ubuntu Juanty using Rinse
- Using LVM2 Snapshots to provide rollback functionality for Xen Howto: LVM2 Snapshots to provide rollback functionality for Xen Domain loopback image
Xen Related Articles
Virtualization software creates a lot of possibilities, in a wide variety of fields. There are a wide variety of solutions available which are free / open source. These include KVM, XEN, QEMU, User-mode Linux, VirtualBox and others. One can now easily create virtual appliances for specific purposes, a database server, a backup server, repository and or search.
Virtual Machines typically reside inside a virtual hard disk image, and knowing how to manipulate these images can allow one to create and update appliances outside the virtual machine environment, open even more possibilities to scripted creation, update and customization of virtual appliances. In this article I will provide a quick rundown of software I have found useful for preparing and manipulating virtual disk images; some of the tools mentioned below are common tools part of the the core Linux utilities you find in every distribution while others are a little more obscure or rarely used.
Image Creation
parted
- Home page www.gnu.org/software/parted/index.shtml
Parted can be considered a more modern version of fdisk. Its a disk partitioning tool that can not only create partitions, but also move and resize them too. It also has the advantage that you can script the operations and it will work equally well on devices as well as files. So in your script, you can include something like this to create a virtual disk image with 4 primary partitions on:
#!/bin/bash
dd if=/dev/zero of=centos.raw bs=1 count=1 seek=40G #create a 40G sparse file
parted centos.raw mklabel msdos # create the partition table
parted centos.raw mkpart primary ext2 0G 512m #swap
parted centos.raw mkpart primary ext2 512m 24G #root
parted centos.raw mkpart primary ext2 24G 38G #data
parted centos.raw mkpart primary linux-swap 38G 40G #swap
parted centos.raw print all #just check what you have done
You can then mount this image file using kpartx (described below)
dd
dd is a very versatile tool for copying large block of information between files, or between devices or between files and devices. Its a core unix tool, and very versatile. You can find an extremely complete guide on its usage by example here: http://www.linuxquestions.org/linux/answers/Applications_GUI_Multimedia/How_To_Do_Eveything_With_DD. This is a kind of dd cookbook.
When discussing virtual disk images, its extremely useful for creating or increasing in size sparse disk images, and also copying image files into loopback devices and so on. In the parted section there is an example of creating a sparse file. Below shows how you might use it to copy from a disk file into a loopback device.
sudo dd if=centos4/data.img of=/dev/mapper/loop0p3 bs=4096 conv=notrunc,noerror
Filesystem Tools
These tools will be extremely familiar to many linux users, and there is plenty of documentation on their use. They are only mentioned here for completeness sake. Since clearly they are necessary for the creation of virtual disk images. You can use any of them directly on disk images, loop back devices or device mapper devices.
- mkfs - create a file system (e.g. mkfs -t ext3 /dev/mapper/loop0p2)
- mkswap - set up a swap file or partition
- e2fsck - check an ext2/3 file system
- resize2fs - resize an ext2/3 file system
Mounting
kpartx
- Home Page (Part of the Linux multipath-tools) christophe.varoqui.free.fr
Kpartx requires Device mapper support in the kernel, and dm userspace tools should be present.
Kpartx is an extremely useful tool for being able to mount partitions that are within a virtual disk image. Virtual Disk immages, can either be files which contain file systems and no partition table. These can be mounted fairly easily with the loopback device using mount -t auto -o loop filename.raw /point or they can be images which contain a partition table and one or more partitions, or they can even contain LVM volumes. With the latter two, both can be read with kpartx. kpartx reads and understands the partition table, so it can know in the virtual image file where each partition begins and ends, and then maps each portion of the file to a different device mapper device using a naming convention like /dev/mapper/loop0p1, /dev/mapper/loop0p2 and so on. If they image file also contains LVM volumes, these will also be mapped in with the device mapper. These sections once mapped can be mounted like normal devices. So for example:
#!/bin/bash
sudo kpartx -a centos.raw
sudo mkdir -p /mnt/loop2
sudo mount -t ext3 /dev/mapper/loop0p2 /mnt/loop2
sudo mount -t ext3 /dev/mapper/loop0p1 /mnt/loop2/boot
will create the mapping for the partitions in a the file centos.raw which contains a boot partition and the root partition. These can then be mounted normally. The following script will do the reverse and unmount the partitions, remove the mapping in the device mapper and delete the loopback device.
#!/bin/bash
sudo umount /mnt/loop2/boot/
sudo umount /mnt/loop2
sudo kpartx -d centos-oracle.raw
mount --bind
mount --bind is worth a mention, as its particular useful in setting up a chroot environment. mount is typically used to mount devices or loopback devices so the filesystem can be accessed however with mount --bind the intention is slightly different. You can make the same part of the directory tree appear in two different places. When setting up a chrooted environment to be able to update software on a disk image, you might typically do the following:
sudo mount -t auto /dev/mapper/loop0p2 /mnt/loop2
sudo mount -t auto /dev/mapper/loop0p1 /mnt/loop2/boot
sudo mount --bind /dev /mnt/loop2/dev
sudo mkdir -p /mnt/loop2/raw
sudo mount --bind /workarea/raw /mnt/loop2/raw
chroot /mnt/loop2 /bin/bash
The first two mount statements are regular mount systems, to mount the root file system, and then the boot file system. The third mount statement used mount --bind to make the /dev file system of the host computer visible as part of the directory tree of the mounted loopback images. The second mount --mind statement, just makes a regular directory within the host filesystem visible also in the mounted loopback images file system. Finally when the chroot command is executed, the chosen parts of the host file system will still be visible. Finally a note of caution here, typically chrooted environments are used to create secure environments that software cannot break out of. Making the host systems /dev file system available in the chrooted environment is not a secure thing to do, and should only be done in special circumstances - such as these, the preparation of a virtual disk image. Even then, cautions should be taken, its still quite easy to hose your system by running grub incorrectly or whatever.
device mapper, lvm
These two tools are just worth a quick mentioned here, as they are sometimes used by the other tools mentioned (e.g. kpartx). However if unlikely you will need to use them directly.
Editing
chroot
<dl><dd>linux32 </dd></dl>
sed - http://www.gnu.org/software/sed/
grub - http://www.gnu.org/software/grub/
Bootstrapping Distributions
debootstrap
rinse
Image Conversion
qemu-img
vboxmanage
Sparse files
zerofree - http://intgat.tigress.co.uk/rmy/uml/index.ht
Having initially done the conversion of xen images to virtualbox using a relatively manual process. I was interest to see how much of this could be scripted. Particularly interesting was the use of grub to install a boot sector on a raw image file, so I would then not have to boot a CD-ROM image inside the virtual machine to install grub manually. I divided the scripts into 3 tasks.
- Create the disk image, complete with partition table
- Install the linux kernel, grub and fixup any files that need to be changed
- Convert the disk image to native Virtualbox format
Below the first script, using parted to create and populate the partition table. kpartx to mount the disk image, and res2fs to ensure the images fit the new disk partitions.
mkfedorasolr-1.sh
#!/bin/bash
rm fedora-solr.raw
dd if=/dev/zero of=fedora-solr.raw bs=1 count=1 seek=35G
parted fedora-solr.raw mklabel msdos
parted fedora-solr.raw mkpartfs primary ext2 0 12G #root
parted fedora-solr.raw mkpartfs primary ext2 12G 24G #data
parted fedora-solr.raw mkpartfs primary linux-swap 24G 26G #swap
parted fedora-solr.raw print all #just check what you have done
sudo kpartx -a fedora-solr.raw
ls -l /dev/mapper
sudo dd if=fedora-solr/root.img of=/dev/mapper/loop0p1
sudo dd if=fedora-solr/data.img of=/dev/mapper/loop0p2
sudo mkswap -f /dev/mapper/loop0p3
sudo e2fsck -f /dev/mapper/loop0p1
sudo resize2fs /dev/mapper/loop0p1
sudo e2fsck -f /dev/mapper/loop0p2
sudo resize2fs /dev/mapper/loop0p2
sudo kpartx -d fedora-solr.raw
Below the second script. Note the following techniques
- Using mount --bind to ensure devices are present in the chrooted environment
- mount --bind is also used to make the raw image visible from inside the chrooted environment. This is necessary for the installation of grub
- copy /etc/resolv.conf into the chrooted environment to make sure that apt-get and friends will be able to resolv dns
- The use of chrooted environment using linux32. This allows the preparation of a 32bit disk image from within a 64 bit environment
- Grub is used on the disk image file rather than the loopback device itself, since grub returns an error 22 if you try an use the loopback device
- Sed is used to edit key files such as /etc/fstab, /etc/mtab and /boot/grub/menu.lst to ensure the system boots and mounts correctly once under the virtual machine
mkfedorasolr-2.sh
#!/bin/bash
sudo kpartx -a fedora-solr.raw
sudo mkdir -p /mnt/loop2
sudo mount -t auto /dev/mapper/loop0p2 /mnt/loop2
sudo mount --bind /dev /mnt/loop2/dev
sudo mkdir -p /mnt/loop2/raw
sudo mount --bind /workarea/raw /mnt/loop2/raw
sudo cp /etc/resolv.conf /mnt/loop2/etc/
ls -l /mnt/loop2
sudo linux32 chroot /mnt/loop2 /bin/bash <<EOF
mount -t proc proc proc
apt-get -y update
apt-get -y install linux-image
mkdir -p /boot/grub/
cp --archive /usr/lib/grub/i386-pc/* /boot/grub/
grub --no-curses --device-map=/dev/null <<EOT
device (hd0) /raw/fedora-solr.raw
root (hd0,1)
setup (hd0)
EOT
update-grub -y
sed -i -e 's!/dev/xv!/dev/s!g' /etc/fstab
sed -i -e 's!/dev/xv!/dev/s!g' /etc/mtab
sed -i -e 's!/dev/xv!/dev/s!g' /boot/grub/menu.lst
sed -i -e 's!(hd0,0)!(hd0,1)!g' /boot/grub/menu.lst
umount proc
EOF
ls -l /mnt/loop2/boot/
ls -l /mnt/loop2/boot/grub/
sudo umount /mnt/loop2/raw
sudo umount /mnt/loop2/dev
sudo rmdir /mnt/loop2/raw
sudo umount /mnt/loop2
sudo kpartx -d fedora-solr.raw
A very simple script to invoke the conversion of the disk image into the native virtualbox format. This script will be enhanced to create and start the virtual machine too.
mkfedorasolr-3.sh
#!/bin/bash
rm ../virtualbox/fedora-solr.vdi
VBoxManage convertdd fedora-solr.raw ../virtualbox/fedora-solr.vdi
Hal Hald wont start on Ubuntu Jaunty
While trying to upgrade a virtual machine running on Sun VirtualBox from a console only installation to a full X11 installation I experience a problem with Hald not starting. Not only did this prevent the running of X11, even the installation could not complete successfully. It turned out that the problem was in some way linked to the file: /var/cache/hald/fdi-cache. If this is not present, hald won't start.
Doing some research with using a well known search engine. I found that some people had solved their problem using:
sudo cp /var/cache/hald/fdi-cache~ /var/cache/hald/fdi-cache
sudo chmod 755 /var/cache/hald/fdi-cache
So I did this, and found that hald did then start and I was able to complete the installation of ubuntu-desktop. However, after a reboot, when the X11 login screen (gdm) appeared, the mouse didn't move and no keyboard input was accepted. After fiddling around a bit more, I found that the following solution worked (executed from a ssh login)
sudo su -
/etc/init.d/hal stop
cd /var/cache/hald
rm *
/usr/lib/hal/hald-generate-fdi-cache #this takes quite a while.
reboot
After executing the above commands I found that the directory /var/cache/hald didn't contain the file fdi-cache~ and the generated file /var/cache/hald/fdi-cache was in fact twice the size of /var/cache/hald/fdi-cache~ I had used before.
Guessing about what the problem might be, hald when it first runs, finds the directory empty, and so runs /usr/lib/hal/hald-generate-fdi-cache automatically. It probably doesn't wait long enough for it to execute, and kills the script assuming its crashed and then dies itself. Perhaps a longer timeout in hald would work.
Conversion of Xen VM images to Sun VirtualBox Virtual Machines.
This year I have been using Xen on Ubuntu with a debian kernel. This works with Intrepid and Jaunty, however I am expecting problems with the upcoming released of Karmic Koala. Once more Canonical have not included a Xen kernel in the distution, choosing instead to support KVM. For this reason I have been looking at other Virtualization Solutions as my machines doesn't VT-X style techinology required by KVM. I am still extremely keen on Xen, and will likely switch back when I can work out how to make it work reliably on Karmic. In the mean time I have beeen looking and the Sun VirtualBox solution. This is available in the Ubuntu Repositories. When porting to virtual box there are a number of problems need to be resolved.I chose this over Vmware, firstly because its open source, and secondly it just seems lighter. The VMware server 2.0 web interface is very comprehensive but it desperately slow on my hardware. Another nice thing about VirtualBox is that you can script almost every aspect of the creation and management of virtual machines.
- The image files I am using in Xen are raw sparse loopback files, they just contain a filesystem (ext3), and no partition table for example. VirtualBox expects a disk image in its own format.
- The images files dont contain grub or kernel images. The kernel stored outside the xen image and loaded by xen (I use one kernel image in fact for all the vms)
Creating an image in the right format
Firstly create an empty image file
dd if=/dev/zero of=amandas3.raw bs=1 count=1 seek=384G
If you make this large, the final step of converting the image with the VirtualBox tool which take much longer, but it will probably be easier to resize partitions in your final running machine
Then create an populate partition table inside the empty image file. The partitions you create should be a little bigger than the images you create in order to be on the safe side.
parted amandas3.raw mklabel msdos
parted amandas3.raw mkpartfs primary ext2 0 10G
parted amandas3.raw mkpartfs primary ext2 10G 14G
#repead for the number of partitions you need
parted amandas3.raw print all #just check what you have done
Then map the partitions in the image into the device mapper
sudo kpartx -a amandas3.raw
kpartx is quite a useful too, if your image file contains LVM volumnes, it will even map these into the device mapper.
Copy in your image file into the partition. Repeat this for each image you have into its respective partition. Check using ls - /dev/mapper to see the names of the parititions.
sudo dd if=domains/amandas3/root.img of=/dev/mapper/loop1p1
Resize the file system to fit the destination partition. Repeat as needed for each partition.
sudo e2fsck -f /dev/mapper/loop1p1
sudo resize2fs /dev/mapper/loop1p1
Unmount the file from the loopback device and remove the mappings.
sudo kpartx -d amandas3.raw
Convert the raw image file into the Virtual Box internal format.
VBoxManage convertdd ./amandas3.raw ../virtualbox/amandas3.vdi
Installing Grub and the kernels
These instructions are more Ubuntu specific, and may need to be tailored if you use a different distribution. However try the following:
Create a VM using the image you created, and attach and ISO bootable image to the CDROM. Boot into the cd inside Virtualbox.
Open a terminal window, and create a chroot environment
sudo su -
mkdir /mnt/tmp
mount -t auto /dev/sda1 /mnt/tmp
mount --bind /dev /mnt/tmp/dev
cp /etc/resolv.conf /mnt/tmp/etc/
chroot /mnt/tmp /bin/bash
mount -t proc proc proc
Install the linux image. This should also download and install grub (though not yet make it bootable)
apt-get update
apt-get install linux-image
Using nano or some similar tool, check and fix the contents of /etc/mtab and /etc/fstab. Typicall you will want to change things from /dev/sda1 instead of /dev/xda2 and similar
Install Grub into the bootsector and create the bootmenu.
grub-install --recheck /dev/sda
update-grub
depmod -a
Shutdown the virtual machine amd then detach the iso image. restart the VM, it should now boot.
If you don't get a network conection, just theck the file /etc//udev/rules.d/70-persistent-net.rules and make sure it is clean.
