Converting a virtual disk image: VDI or VMDK to an ISO you can distribute

Why would anyone in their right mind want to convert a VM into an ISO?

Good question, the answer for Conor Fox (who was the inspiration for this post - thanks Conor!) was to distribute his customized TurnKey PostgreSQL image so others could use it.

Distributing an ISO as opposed to a VM image allows it to be installed on any virtualization platform, as well as on bare metal, with the added bonus of running live.

I suppose that's a good enough reason, so lets get to it.


Convert VM disk to raw image and mount it

First we need to get qemu-img, a tool bundled with qemu (KVM's virtualization backend) to convert the VM disk to a raw image, and TKLPatch, the TurnKey customization mechanism to package the ISO.

If you are not using a TurnKey installation, see the TKLPatch installation notes.

apt-get install qemu
apt-get install tklpatch

I'll show how to convert a VMWare VMDK image into raw disk format. If you are using a different virtualization platform such as Virtualbox, see this post on converting a VDI to a raw image.

qemu-img convert -f vmdk turnkey-core.vmdk -O raw turnkey-core.raw

Next, mount the raw disk as a loopback device.

mkdir turnkey-core.mount
mount -o loop turnkey-core.raw turnkey-core.mount

GOTCHA 1: If your VM has partitions, it's a little tricker. You'll need to setup the loop device, partition mappings and finally mount the rootfs partition. You will need kpartx to setup the mappings.

loopdev=$(losetup -s -f turnkey-core.raw)

apt-get install kpartx
kpartx -a $loopdev

# p1 refers to the first partition (rootfs)
mkdir turnkey-core.mount
mount /dev/mapper/$(basename $loopdev)p1 turnkey-core.mount


Extract root filesystem and tweak for ISO configuration

Now, make a copy of the root filesystem and unmount the loopback.

mkdir turnkey-core.rootfs
rsync -a -t -r -S -I turnkey-core.mount/ turnkey-core.rootfs

umount -d turnkey-core.mount

# If your VM had partitions (GOTCHA 1):
kpartx -d $loopdev
losetup -d $loopdev

Because the VM is an installed system as opposed to the ISO, the file system table needs to be updated.

cat>turnkey-core.rootfs/etc/fstab<<EOF
aufs / aufs rw 0 0
tmpfs /tmp tmpfs nosuid,nodev 0 0
EOF

GOTCHA 2: If your VM uses a kernel optimized for virtualization (like the one included in the TurnKey VM builds), you need to replace it with a generic kernel, and also remove vmware-tools if installed.

tklpatch-chroot turnkey-core.rootfs

# inside the chroot
apt-get update
apt-get install linux-image-generic
dpkg --purge $(dpkg-query --showformat='${Package}\n' -W 'vmware-tools*')
dpkg --purge $(dpkg-query --showformat='${Package}\n' -W '*-virtual')

exit


Generate the ISO

Finally, prepare the cdroot and generate the ISO.

tklpatch-prepare-cdroot turnkey-core.rootfs/
tklpatch-geniso turnkey-core.cdroot/

Thats it!

Bonus: By default the ISO will boot automatically. If you want to include the TurnKey bootsplash and bootmenu, extract the cdroot from a TurnKey ISO and tell tklpatch-prepare-cdroot to use it as a template.

tklpatch-extractiso turnkey-core.iso
tklpatch-prepare-cdroot turnkey-core.rootfs/ turnkey-core.cdroot/
tklpatch-geniso turnkey-core.cdroot/

Ever needed to package a VM as a distributable ISO? Post a comment!

Comments

Alon Swartz's picture

The virtual kernel does not include squashfs support, so the resulting ISO won't be able to unpack the root filesystem, and the boot will fail.

Also, an ISO can be installed to bare-metal, so it doesn't make much sense to include the optimized virtual kernel even if it was possible (to be complete, it is possible with a couple of workarounds but not worth the effort, and possible side-effects).
Alon Swartz's picture

I suppose (without thinking about it too much) that the process should be very similar, minus the VM conversion.

In general it should go something like this:

  • Mount the secondary drive/partition, and copy out the root filesystem
  • Tweak as needed for ISO configuration
  • Generate the ISO

I would be interested to know if you try this, and what the outcome is. Keep us updated.
Jeremy Davis's picture

Once you have created the .raw file you then mount it (like you would an .iso) and copy out the contents (which should only be ~1.7GB - ie your original data size). Your resulting .iso should be around that size too. If not then I would assume there is something funny/wrong going on.

Jeremy Davis's picture

Perhaps you could try RemasterSys in the meantime? Although TBH I haven't used it with a server (but it's been quite useful for desktop use).

PS Did you document your iFolder progress? It'd be great if you could share as I'd love to create an iFolder TKLPatch and then perhaps an official TKL iFolder appliance might be released.

Jeremy Davis's picture

I will investigate further. When I get a chance I will look at perhaps uploading it to a PPA. I'll post if/when I get to that.

Jeremy Davis's picture

That's why it's larger than your original VDI. And actually it's a RAW file (which is a raw harddrive image). If you follow the tutorial it is only halfway through the process. To complete it you need to mount the image you have created and copy out the files, then make an ISO of the files only (that doesn't include the free space). This should bring your ISO back down to a similar (or perhaps smaller) filesize as your original VDI.

Jeremy Davis's picture

Just guessing but try:

tklpatch-prepare-cdroot Mikrotik

(No trailing slash). The clue for me was the error message (unless it's just a typo):

cp: omitting directory `Mikrotik//boot/'

Although must admit that the tutorial does include the trailing slash so not sure if that's it.

Jeremy Davis's picture

This tutorial works with TKL and I would imagine any Ubuntu or Ubuntu based distro. It may even also work on Debian - but perhaps not...

A quick scan of the second link you provided suggests that that particular version is a mishmash of 2 old versions of Debian but I couldn't quite work out what the go was with the new version. The fact that searching/scanning the website, forum and wiki resulted in no idea on what the OS is based on or what sort of licence it's under doesn't bode well (although perhaps I'm wrong). It seems quite closed and as such I'd suspect that the only way you will be able to do what you are hoping to is with the help and/or support of the makers/community. Have you posted on their forums?

But TBH unless there is a compelling reason, personally I'd be looking about for an opensource product that does the same thing thgat does support hacking etc. I haven't tested any but Wikipedia has a list of starters. No doubt there is probably more if you have a good google. My 2c anyway...

Andreas's picture

True regarding the opensource firewalls distros.

Except I've been through ALL of THEM. :) Seriously, I'm running on Zentyal at my office at the moment, but I've been through Smoothwall, pfSense, ClarkConnect, IpCop, IPFire, etc etc etc.

//disclaimer - I am not affiliated with Mikrotik in any way whatsoever. In fact, they would probably look down on what I am trying to do by creating a LiveCD.

Mikrotik is absolutly the best of the best. Its a bit funky to get started with, ( I was actually planning on running pfSense before I read a single post from a random comment on the pfSense forums that said "Try Mikrotik, once you learn it, you will never go back." I spent last weekend learning it, and without a doubt, it frakking rocks.

Just to get started, a web interface and windows interface for the firewall.. Why would you need the windows interface? Well, for starters, it allows you to connect to the firewall to reconfigure it regardless of your machines IP settings. It connects via MAC address, TCP/IP4 or TCP/IP6 as well as autoscans the local network so a freshly installed firewall is accessible without even touching the console.

The 24 hour time limit on the demo is a bummer but you can backup your config, reformat , and reinstall your config to continue using it or learning it. As well as their licensing is confusing and a bit expensive for SOHO usage.. but considering you can buy some of the RouterBoards including the license for less than the licensing alone, I'm not sure where I stand on that.

I highly reccomend it if you do any networking at all, or if you just want something fun to play with. 

Jeremy Davis's picture

AFAIK unless the original distro supports running live then this will not work for you to create a live distro.

I'm sure creating a live version of the router OS is possible, but not without some serious hacking. The TKL devs have hacked the default Ubuntu Server OS so it can run live so unlike default Ubuntu Server TKL can run live already. I imagine you would need to do similar hacking first to allow this distro to run live. How you would actually go about that is well beyond me and may also be outside the terms of the Mikrotik licence.

As you've possibly guessed from my previous posts I'm a strong advocate of the open source model. But I also try to be respectful of the relevant licencing. If you like Mikrotik RouterOS that much I suggest you consult with them on what your options are and what it will cost you to do what you want.

Jeremy Davis's picture

But if it's a big image, yes it will take a while. A good baseline timeframe to consider is how long would it take to copy the file? Then add a bit (depending on hardware). You can see if it's still doing something via a new terminal window. I don't remember the commands off the top of my head but if you google "check progress qemu-img convert commandline linux" or something similar. I haven't actually done it with qemu-img but I am assuming it would be similar to checking progress of dd.

Jeremy Davis's picture

vfat should be a recognised format and depending on what distro you are using, you may need to instal ntfs drivers before it will recognise NTFS FS.

Regardless of that though, because of the nature of how Windows works, I don't think you'll have any joy using this method anyway. Even if you do manage to convert your VM to an ISO I imagine that you'll find that trying to run it on anything that doesn't closely resemble the virtual hardware of your VM will result in a BSOD.

I'm guessing you are trying to create a Win7 ISO with preinstalled apps and/or customised config? I don't know enough about Win7 to give you any really good suggestions, but if you are looking to create a preconfigured image then the MS tool sysprep is probably your best bet and then create an image of your syspreped system. If you hunt about online I'm sure you'll find tons of info detailing the best course of action.

Jeremy Davis's picture

Another afterthought may be vLite. AFAIK it is the successor to nLite which was a quite cool way of creating a customised XP install CD (vLite is for Vista & 7 IIRC).

I have successfully been using a combination of sysprep and a FOG server to create and deploy Win7 images to multiple machines. It may be overkill (or simply not suitable) for your purposes but it works really well for mine! Possibly worth a look...?

Jeremy Davis's picture

And if not (but still something Linux) it should be possible, although to be honest I have no idea how you would go about that. I would suggest that you post on the forums/mailing list for whatever distro you're using and ask there. That way hopefully you will get some answers relevant to your scenario.

Jeremy Davis's picture

Although I haven't tried it on a server OS. Also I didn't realise that it was back up and running (IIRC it died some time ago, but obviously got revived somewhere along the line) so thanks for that info.

Jeremy Davis's picture

I suspect the issue is that TKL now uses LVM by default. Unfortunately I am not sure on a workaround.

Jeremy Davis's picture

If so then that would be the problem...

Otherwise that's really weird! According to Debian dpkg-divert is provided by the dpkg package. It is a fundamental part of Debian (and it's derivative's) package management system and should certainly be installed!

Jeremy Davis's picture

These instructions are really old so may need some updating. IMO they could certainly be improved!

Anyway, it's clear that for whatever reason your image isn't mounting properly. Does it have partitions? To see what is going on, try this:

kpartx -l turnkey-core.raw
If it is partitioned you should get something like this:
# kpartx -l turnkey-core.raw
loop0p1 : 0 9765 /dev/loop0 1
loop0p2 : 0 26624 /dev/loop0 12288
loop deleted : /dev/loop0
You can then use kpartx directly to mount the image as a loopback:
kpartx -a turnkey-core.raw
The partitions will be as per the output when the -l switch was used. I.e. /dev/mapper/loop0p1 will be the first partition and /dev/mapper/loop0p2 is the second. So you then use mount to mount /dev/mapper/loop0p1 to turnkey-core.mount. When you are done, umount turnkey-core.mount and delete the loopback device with kpartx:
kpartx -d turnkey-core.raw

Actually I just noticed that you are trying to mount /dev/mapper/loop0pl (lower case 'L') whereas it should be /dev/mapper/loop0p1 (number '1'). Perhaps that's your only issue?

Jeremy Davis's picture

This is pretty dated now and was Linux specific. I'm not even sure that it would still work with Linux, let alone an alternate OS. So I doubt that it would work with Mac OSX, but you could try and see how you go?!

Pages

Add new comment