lunix/ blog/ Self Serve Dev Environment

Navigation

Front Page

About

Meta

Sitemap

Recent Blog entries

Connecting a SIP Trunk to a Remote SIP Extension

Share any website in Google Reader

Installing Graylog2 on Ubuntu Lucid

A sane Varnish version for Ubuntu Hardy

External

My flickr

Arch Linux Wiki

My GitHub

My Twitter

Sydney Devops

Introduction

I am currently working in a web development shop. We develop and maintain a range of websites/webapps for customers.
At $WORK we have many varied customers each with their own unique Production Environments (PE from here on in).
Our $DEVs are also free to run any *nix based OS on their workstations.
One of the challeges we have had in the past is making a copy of a customer's PE easily available to our $DEVS.
This used to require lodging a task in Redmine and waiting for a Systems Admin to build you a VM on a central VM server.
This post will show how we are now using common FOSS tools to give enable to $DEVS to have VM's on their own workstations that mimick a customer's PE.

Whilst I make mention of some specific tools in this post they can be swapped out in most places for alternates however I have not tested the
alternatives yet. ie: puppet/chef, mercurial/git, centos/ubuntu.

Notes on the Challenge

  • needs to be simple not an obsticle - if it's slower than just getting a sysadmin to build it for you then its a fail
  • simplicity generally means easy to fix when something goes wrong in the wheel.
  • self-serve - no waiting for sysadmins
  • visability - everything in DVCS and Redmine (project managent software)
  • needed to be repeatable - $DEVs needed to easily be able to build, destroy and build again
  • relatively self documenting - read the kickstart or puppet manifests
  • I hate OS images - They're big, cumbersome and pain in my..err..storage See - Golden Image or Foil Ball?

After spending a fair amount of time on this and looking at many of the VM/cloud management solutions out there I have decided that while some are very nice and useful I do not believe they are suiteable for our situation. Most VM/cloud management tools are built around the "OS Image" and require each workstation to 'register' as a node.

Current solution

After doing the full circle of research we are now using a simple collection of existing tools.
It was all there staring me in the face all along. Libvirt, virt-install kickstart puppet, mercurial and a wiki entry. A $DEV just needs to make sure he/she has libvirt, virt-install, virt-viewer installed.
We are using KVM to provide the virtualisation layer but through the use of libvirt you should be able to use any libvirt compatible virtualisation provider.(virtual box etc)

Technologies used

  • a httpd server (nginx, apache etc) - to serve kickstart + yum repos/mirror
  • Own yum repos + centos mirror ( again ubuntu mirror etc )
  • puppetmasterd ( or other CF tool ie: chef etc ) with autosign turned on (we have a separate puppetmaser for the $DEVS)
  • some kickstart files - I use one per customer and bootstrap puppet from the %POST section
  • libvirtd + KVM/qemu - could be any supported virtualisation software supported by libvirt
  • python-virtinst + virt-viewer
  • dhcpd
  • forward and reverse dns - puppet will fail to work as expected without it ( I use powerdns-recursor for demos as it exports /etc/hosts )
  • redmine - we make use of Redmine's ACL's to visualize the repos for puppet and kickstart files per customer

Devs

The following is the steps needed for a $DEV to deploy a customer's PE.

  • Check network page and grab an available network mac to use (this is used for dhcp & dns so puppet works properly)
    and the name of the customers kickstart file.
  • update wiki page to say that network mac is in use.

  • deploy a VM on their workstation. - See Libvirt tips

    virt-install --connect qemu:///system --accelerate -n virt01 -m 54:52:00:37:2E:B9 -r 1024 --vcpus=1 --disk pool=lvm,bus=virtio,size=20 --vnc --os-type linux --os-variant=rhel5 --network=network:default -l http://192.168.1.250/os/CentOS/5.5/os/x86_64/ -x "ks=http://192.168.1.250/ks/project_customer1.ks"

This will take advantage of the fact that both CentOS and Ubuntu have the necessary PXE files stored in their mirrors for booting the installer.

  • wait approx 10 or so minutes and they have a clone of the customer's PE on their workstation ready to deploy to and hack on. see notes in conclusion below

New customers

The following is what's involved in preparing for a new customer:

  • A new customer has a VM/server provisioned in a DC by a hosting company.
  • I grab the current package list and make a kickstart file to replicate the install locally
  • Create a new project_customer3 in puppet and add details to bottom of the new kickstart file.
  • publish new kickstart file and update wiki entry

Conclusion

I have reduced the time it takes for a dev to get a copy of a customers PE down from days to minutes and its now a self serve solution.
There is still more to refine in this but it's already full of win as I now get to do more of 'stuff that matters'

It's early days for us using this new setup and I am yet to work out an easy, effective way of notifying a $DEV when puppet has finished the buildout. Suggestions welcome.

  • cucumber tests ?
  • using libnotify via Dbus ? ( suggestion made at a recent DevOps Sydney meetup )
  • ??