A frequently asked question for git is How can I checkout only part of a repository? or How can I clone just a subdirectory with git? Git doesn’t directly support this option, but since version 1.9 it comes pretty close. The answer is to use a combination of features, a git sparse checkout and shallow checkout.
While recently reviewing hosting options and looking at how to revive this blog, I was surprised at the services now on offer to provide hosting that don’t cost a penny. Going back in time, I have hired virtual machines, shared servers for dollars a month and more recently used Amazon S3 services very few dollars a month, with a bought-and-paid for SSL certificate. More recently I have discovered that all these services I have paid for in the past are now available for free. In this article I’ll run through the options I have chosen for this blog and what they offer.
How to resize the root partition on an Amazon EC2 instance which has its root partition backed by Elastic Block Storage (EBS).
At the time of writing the documentation seems a little incomplete in this respect, so here is what I did to make a larger root partition on an Ubuntu Karmic Server EC2 instance.
- Stop the machine (do not terminate it)
- Take a snapshot
- When the snapshot has completed, restart the machine
- Create a new volume in the same availability zone as the current one, but with the new size
- Attach the new volume to the old machine using the amazon aws console (the device I used is /dev/sdd)
- e2fsck -f /dev/sdd
- resize2fs -f /dev/sdd
- e2fsck -f /dev/sdd
- Detach the volume
- Create a new shapshot
- Create a new AMI from the new snapshot
./ec2-register -K ~/ec2/pk-mypk.pem -C ~/ec2/cert-mycert.pem -n MyUniqueName -s snap-mysnap --kernel aki-5f15f636 --ramdisk ari-0915f660
Virtualization software creates a lot of possibilities, in a wide variety of fields. There are a wide variety of solutions available which are free / open source. These include KVM, XEN, QEMU, User-mode Linux, VirtualBox and others. One can now easily create virtual appliances for specific purposes, a database server, a backup server, repository and or search.
Virtual Machines typically reside inside a virtual hard disk image, and knowing how to manipulate these images can allow one to create and update appliances outside the virtual machine environment, open even more possibilities to scripted creation, update and customization of virtual appliances. In this article I will provide a quick rundown of software I have found useful for preparing and manipulating virtual disk images; some of the tools mentioned below are common tools part of the the core Linux utilities you find in every distribution while others are a little more obscure or rarely used.
Having initially done the conversion of xen images to virtualbox using a relatively manual process. I was interest to see how much of this could be scripted. Particularly interesting was the use of grub to install a boot sector on a raw image file, so I would then not have to boot a CD-ROM image inside the virtual machine to install grub manually. I divided the scripts into 3 tasks.
- Create the disk image, complete with partition table
- Install the linux kernel, grub and fixup any files that need to be changed
- Convert the disk image to native Virtualbox format
Below the first script, using parted to create and populate the partition table. kpartx to mount the disk image, and res2fs to ensure the images fit the new disk partitions.
#!/bin/bash rm fedora-solr.raw dd if=/dev/zero of=fedora-solr.raw bs=1 count=1 seek=35G parted fedora-solr.raw mklabel msdos parted fedora-solr.raw mkpartfs primary ext2 0 12G #root parted fedora-solr.raw mkpartfs primary ext2 12G 24G #data parted fedora-solr.raw mkpartfs primary linux-swap 24G 26G #swap parted fedora-solr.raw print all #just check what you have done sudo kpartx -a fedora-solr.raw ls -l /dev/mapper sudo dd if=fedora-solr/root.img of=/dev/mapper/loop0p1 sudo dd if=fedora-solr/data.img of=/dev/mapper/loop0p2 sudo mkswap -f /dev/mapper/loop0p3 sudo e2fsck -f /dev/mapper/loop0p1 sudo resize2fs /dev/mapper/loop0p1 sudo e2fsck -f /dev/mapper/loop0p2 sudo resize2fs /dev/mapper/loop0p2 sudo kpartx -d fedora-solr.raw
Below the second script. Note the following techniques
- Using mount –bind to ensure devices are present in the chrooted environment
- mount –bind is also used to make the raw image visible from inside the chrooted environment. This is necessary for the installation of grub
- copy /etc/resolv.conf into the chrooted environment to make sure that apt-get and friends will be able to resolv dns
- The use of chrooted environment using linux32. This allows the preparation of a 32bit disk image from within a 64 bit environment
- Grub is used on the disk image file rather than the loopback device itself, since grub returns an error 22 if you try an use the loopback device
- Sed is used to edit key files such as /etc/fstab, /etc/mtab and /boot/grub/menu.lst to ensure the system boots and mounts correctly once under the virtual machine
#!/bin/bash sudo kpartx -a fedora-solr.raw sudo mkdir -p /mnt/loop2 sudo mount -t auto /dev/mapper/loop0p2 /mnt/loop2 sudo mount --bind /dev /mnt/loop2/dev sudo mkdir -p /mnt/loop2/raw sudo mount --bind /workarea/raw /mnt/loop2/raw sudo cp /etc/resolv.conf /mnt/loop2/etc/ ls -l /mnt/loop2 sudo linux32 chroot /mnt/loop2 /bin/bash <<EOF mount -t proc proc proc apt-get -y update apt-get -y install linux-image mkdir -p /boot/grub/ cp --archive /usr/lib/grub/i386-pc/* /boot/grub/ grub --no-curses --device-map=/dev/null <<EOT device (hd0) /raw/fedora-solr.raw root (hd0,1) setup (hd0) EOT update-grub -y sed -i -e 's!/dev/xv!/dev/s!g' /etc/fstab sed -i -e 's!/dev/xv!/dev/s!g' /etc/mtab sed -i -e 's!/dev/xv!/dev/s!g' /boot/grub/menu.lst sed -i -e 's!(hd0,0)!(hd0,1)!g' /boot/grub/menu.lst umount proc EOF ls -l /mnt/loop2/boot/ ls -l /mnt/loop2/boot/grub/ sudo umount /mnt/loop2/raw sudo umount /mnt/loop2/dev sudo rmdir /mnt/loop2/raw sudo umount /mnt/loop2 sudo kpartx -d fedora-solr.raw
A very simple script to invoke the conversion of the disk image into the native virtualbox format. This script will be enhanced to create and start the virtual machine too.
#!/bin/bash rm ../virtualbox/fedora-solr.vdi VBoxManage convertdd fedora-solr.raw ../virtualbox/fedora-solr.vdi