18. November 2016

Git Checkout Subdirectory with Sparse Shallow Checkout

A frequently asked question for git is How can I checkout only part of a repository? or How can I clone just a subdirectory with git? Git doesn’t directly support this option, but since version 1.9 it comes pretty close. The answer is to use a combination of features, a git sparse checkout and shallow checkout.


20. October 2016

Building command line tools (CLI) in Node.js JavaScript from scratch

Originally developed from the JavaScript engine you find in browsers such as google chrome, Nodejs is a JavaScript engine for running scripts in a variety of other environments. Given the large amount of libraries developed for JavaScript it provides a very productive environment for writing quite sophisticated command line tools. The command line tools can be installed globally on your computer, and if carefully written will work exactly the same on Windows as on Linux.

There are a large number of tutorials on the Internet showing you how to install NodeJS, and principles of JavaScript programming, as well as several well equipped editors with syntax highlighting and code completion, so I won’t dwell on those points here, rather this tutorial will concern itself with the unique needs of developing command line tools, such as parsing the command line, saving settings, installation and so on.


01. October 2016

Free Hosting done properly!

While recently reviewing hosting options and looking at how to revive this blog, I was surprised at the services now on offer to provide hosting that don’t cost a penny. Going back in time, I have hired virtual machines, shared servers for dollars a month and more recently used Amazon S3 services very few dollars a month, with a bought-and-paid for SSL certificate. More recently I have discovered that all these services I have paid for in the past are now available for free. In this article I’ll run through the options I have chosen for this blog and what they offer.


09. December 2009

Resize Root Partition on boot from root on EBS EC2 instance

How to resize the root partition on an Amazon EC2 instance which has its root partition backed by Elastic Block Storage (EBS).

At the time of writing the documentation seems a little incomplete in this respect, so here is what I did to make a larger root partition on an Ubuntu Karmic Server EC2 instance.

  1. Stop the machine (do not terminate it)
  2. Take a snapshot
  3. When the snapshot has completed, restart the machine
  4. Create a new volume in the same availability zone as the current one, but with the new size
  5. Attach the new volume to the old machine using the amazon aws console (the device I used is /dev/sdd)
  6. e2fsck -f /dev/sdd
  7. resize2fs -f /dev/sdd
  8. e2fsck -f /dev/sdd
  9. Detach the volume
  10. Create a new shapshot
  11. Create a new AMI from the new snapshot

./ec2-register -K ~/ec2/pk-mypk.pem -C ~/ec2/cert-mycert.pem -n MyUniqueName -s snap-mysnap --kernel aki-5f15f636 --ramdisk ari-0915f660

12. October 2009

Virtual Disk Image Tools

Virtualization software creates a lot of possibilities, in a wide variety of fields. There are a wide variety of solutions available which are free / open source. These include KVM, XEN, QEMU, User-mode Linux, VirtualBox and others. One can now easily create virtual appliances for specific purposes, a database server, a backup server, repository and or search.

Virtual Machines typically reside inside a virtual hard disk image, and knowing how to manipulate these images can allow one to create and update appliances outside the virtual machine environment, open even more possibilities to scripted creation, update and customization of virtual appliances. In this article I will provide a quick rundown of software I have found useful for preparing and manipulating virtual disk images; some of the tools mentioned below are common tools part of the the core Linux utilities you find in every distribution while others are a little more obscure or rarely used.


12. October 2009

Scripting Xen Image to VirtualBox VDI conversion

Having initially done the conversion of xen images to virtualbox using a relatively manual process. I was interest to see how much of this could be scripted. Particularly interesting was the use of grub to install a boot sector on a raw image file, so I would then not have to boot a CD-ROM image inside the virtual machine to install grub manually. I divided the scripts into 3 tasks.

  1. Create the disk image, complete with partition table
  2. Install the linux kernel, grub and fixup any files that need to be changed
  3. Convert the disk image to native Virtualbox format

Below the first script, using parted to create and populate the partition table. kpartx to mount the disk image, and res2fs to ensure the images fit the new disk partitions.


rm fedora-solr.raw  
dd if=/dev/zero of=fedora-solr.raw bs=1 count=1 seek=35G   
parted fedora-solr.raw mklabel msdos  
parted fedora-solr.raw mkpartfs primary ext2 0 12G #root  
parted fedora-solr.raw mkpartfs primary ext2 12G 24G #data  
parted fedora-solr.raw mkpartfs primary linux-swap 24G 26G #swap  
parted fedora-solr.raw print all #just check what you have done  
sudo kpartx -a fedora-solr.raw   
ls -l /dev/mapper  
sudo dd if=fedora-solr/root.img of=/dev/mapper/loop0p1  
sudo dd if=fedora-solr/data.img of=/dev/mapper/loop0p2  
sudo mkswap -f /dev/mapper/loop0p3  
sudo e2fsck -f /dev/mapper/loop0p1  
sudo resize2fs /dev/mapper/loop0p1  
sudo e2fsck -f /dev/mapper/loop0p2  
sudo resize2fs /dev/mapper/loop0p2  
sudo kpartx -d fedora-solr.raw 

Below the second script. Note the following techniques

  1. Using mount –bind to ensure devices are present in the chrooted environment
  2. mount –bind is also used to make the raw image visible from inside the chrooted environment. This is necessary for the installation of grub
  3. copy /etc/resolv.conf into the chrooted environment to make sure that apt-get and friends will be able to resolv dns
  4. The use of chrooted environment using linux32. This allows the preparation of a 32bit disk image from within a 64 bit environment
  5. Grub is used on the disk image file rather than the loopback device itself, since grub returns an error 22 if you try an use the loopback device
  6. Sed is used to edit key files such as /etc/fstab, /etc/mtab and /boot/grub/menu.lst to ensure the system boots and mounts correctly once under the virtual machine


sudo kpartx -a fedora-solr.raw   
sudo mkdir -p /mnt/loop2  
sudo mount -t auto /dev/mapper/loop0p2 /mnt/loop2  
sudo mount --bind /dev /mnt/loop2/dev  
sudo mkdir -p /mnt/loop2/raw  
sudo mount --bind /workarea/raw /mnt/loop2/raw  
sudo cp /etc/resolv.conf /mnt/loop2/etc/  
ls -l /mnt/loop2  
sudo linux32 chroot /mnt/loop2 /bin/bash <<EOF  
mount -t proc proc proc  
apt-get -y update  
apt-get -y install linux-image  
mkdir -p /boot/grub/  
cp --archive /usr/lib/grub/i386-pc/* /boot/grub/  
grub --no-curses --device-map=/dev/null <<EOT  
device (hd0) /raw/fedora-solr.raw  
root (hd0,1)  
setup (hd0)  
update-grub -y  
sed -i -e 's!/dev/xv!/dev/s!g' /etc/fstab  
sed -i -e 's!/dev/xv!/dev/s!g' /etc/mtab  
sed -i -e 's!/dev/xv!/dev/s!g' /boot/grub/menu.lst  
sed -i -e 's!(hd0,0)!(hd0,1)!g' /boot/grub/menu.lst  
umount proc  
ls -l /mnt/loop2/boot/  
ls -l /mnt/loop2/boot/grub/  
sudo umount /mnt/loop2/raw  
sudo umount /mnt/loop2/dev  
sudo rmdir /mnt/loop2/raw  
sudo umount /mnt/loop2  
sudo kpartx -d fedora-solr.raw

A very simple script to invoke the conversion of the disk image into the native virtualbox format. This script will be enhanced to create and start the virtual machine too.


rm ../virtualbox/fedora-solr.vdi  
VBoxManage convertdd fedora-solr.raw ../virtualbox/fedora-solr.vdi