One of the advantages to the idea of devops and having a mixed team of systems people and developers, is that it frees you from having only to use the tools written by other people. While using 3rd party tools is almost always preferable, if needed you can also fall back on having your developers whip something up.
On the hosting side of our business one of the big projects we have going on right now is creating tools to help make our server deployments more automated and reliable. As part of that we’ve recently found it helpful to have our developers be able to generate disk images (ie fake hard drives stored in files) for Linux so that they can test changes in a Virtual Machine locally or even be able to deploy them. Now, most any developer or systems administrator these days is going to be working with disk images, whether they are cloud hosting images, local VM images, or installer disk images. You would imagine that given how prevalent they are, there would be easy tooling around creating them.
As it turns out, there are some problems. The steps involved for creating a disk image are:
- Create a file of the correct size
- Partition it
- Create a filesystem on the partitions you need
- Write files to those partitions
- Make it bootable
Of those steps, only the first one can be done directly to an image file using the standard tools. Every other step requires the use of what on Unix are called loopback devices, essentially asking the OS to show your disk image as if it was a physical drive on the system, and then using the normal disk utilities. The problem with this is twofold, for one even just setting up the loopback requires a high level of system access, and then even after it’s setup the access level needed to work with it is the same needed to access the actual hard drives on the computer.
For some use cases this is fine, but for us asking every developer to give that level of access, where one mistake could wipe out all of their personal data, to our automatic tools was asking too much. In addition, neither of the two major Linux boot systems these days, grub and syslinux, provide Mac OS X tools for making disks bootable, and a good chunk of our team are Mac users. The only other solution seemed to be to have every developer run a VM just for doing image builds, but this turned out to be a big productivity killer as various components and files needed to be made accessible to the VM.
Ultimately we decided to look around and see if there were libraries that could be stitched together to produce a tool that could do everything we needed. Even there we found somewhat mixed news. We discovered a great library for working with ext4 called lwext4, but for everything else we ended up resorting to simply compiling in source files of a couple command line utilities like gptfdisk and syslinux.
The result is loopless, our new utility for working with disk images looplessly. In addition to handling each of the steps above, it’s also available for Mac, making it the first tool we’re aware of to allow creating bootable Linux disks on Mac OS X.
We’ve been using it as part of our work flow now for a couple of months and in it’s made the process of testing system configuration changes a breeze. In the future we plan on new versions with better error reporting (which is currently a fairly large blind spot), support for sparse vmdk images, and better support for physical disks.
We’ve published it on our public Gitlab in hopes it can save some other team some time.