blog

Funny tech support email No.1

Thu, 29 Jul 2010 18:34:15 -0400

Funny Tech Support Email Number 1

In the late 1990's we purchased a few small ISP's and whilst auditing their old servers (terrible mess) I come across this beauty.
I thought I'd share this, plus one other, from another ISP, I found for this years System Administrator's Appreciation Day.
They both made me laugh back then and still make me laugh today. Oh how I miss the days of ISP land.

Funny tech support email No.2

Thu, 29 Jul 2010 18:34:15 -0400

Funny Tech Support Email Number 2

In the mid 2000's whilst working at an ISP the following email arrived.
It gave us all a good laugh at the time and today I share with you all for System's Administrator Appreciation Day.

Self Serve Dev Environment

Sat, 24 Jul 2010 03:12:11 -0400

Introduction

I am currently working in a web development shop. We develop and maintain a range of websites/webapps for customers.
At $WORK we have many varied customers each with their own unique Production Environments (PE from here on in).
Our $DEVs are also free to run any *nix based OS on their workstations.
One of the challeges we have had in the past is making a copy of a customer's PE easily available to our $DEVS.
This used to require lodging a task in Redmine and waiting for a Systems Admin to build you a VM on a central VM server.
This post will show how we are now using common FOSS tools to give enable to $DEVS to have VM's on their own workstations that mimick a customer's PE.

Whilst I make mention of some specific tools in this post they can be swapped out in most places for alternates however I have not tested the
alternatives yet. ie: puppet/chef, mercurial/git, centos/ubuntu.

Notes on the Challenge

needs to be simple not an obsticle - if it's slower than just getting a sysadmin to build it for you then its a fail
simplicity generally means easy to fix when something goes wrong in the wheel.
self-serve - no waiting for sysadmins
visability - everything in DVCS and Redmine (project managent software)
needed to be repeatable - $DEVs needed to easily be able to build, destroy and build again
relatively self documenting - read the kickstart or puppet manifests
I hate OS images - They're big, cumbersome and pain in my..err..storage See - Golden Image or Foil Ball?

After spending a fair amount of time on this and looking at many of the VM/cloud management solutions out there I have decided that while some are very nice and useful I do not believe they are suiteable for our situation. Most VM/cloud management tools are built around the "OS Image" and require each workstation to 'register' as a node.

Current solution

After doing the full circle of research we are now using a simple collection of existing tools.
It was all there staring me in the face all along. Libvirt, virt-install kickstart puppet, mercurial and a wiki entry. A $DEV just needs to make sure he/she has libvirt, virt-install, virt-viewer installed.
We are using KVM to provide the virtualisation layer but through the use of libvirt you should be able to use any libvirt compatible virtualisation provider.(virtual box etc)

Technologies used

a httpd server (nginx, apache etc) - to serve kickstart + yum repos/mirror
Own yum repos + centos mirror ( again ubuntu mirror etc )
puppetmasterd ( or other CF tool ie: chef etc ) with autosign turned on (we have a separate puppetmaser for the $DEVS)
some kickstart files - I use one per customer and bootstrap puppet from the %POST section
libvirtd + KVM/qemu - could be any supported virtualisation software supported by libvirt
python-virtinst + virt-viewer
dhcpd
forward and reverse dns - puppet will fail to work as expected without it ( I use powerdns-recursor for demos as it exports /etc/hosts )
redmine - we make use of Redmine's ACL's to visualize the repos for puppet and kickstart files per customer

Devs

The following is the steps needed for a $DEV to deploy a customer's PE.

Check network page and grab an available network mac to use (this is used for dhcp & dns so puppet works properly)
and the name of the customers kickstart file.
update wiki page to say that network mac is in use.
deploy a VM on their workstation. - See Libvirt tips

virt-install --connect qemu:///system --accelerate -n virt01 -m 54:52:00:37:2E:B9 -r 1024 --vcpus=1 --disk pool=lvm,bus=virtio,size=20 --vnc --os-type linux --os-variant=rhel5 --network=network:default -l http://192.168.1.250/os/CentOS/5.5/os/x86_64/ -x "ks=http://192.168.1.250/ks/project_customer1.ks"

This will take advantage of the fact that both CentOS and Ubuntu have the necessary PXE files stored in their mirrors for booting the installer.

wait approx 10 or so minutes and they have a clone of the customer's PE on their workstation ready to deploy to and hack on. see notes in conclusion below

New customers

The following is what's involved in preparing for a new customer:

A new customer has a VM/server provisioned in a DC by a hosting company.
I grab the current package list and make a kickstart file to replicate the install locally
Create a new project_customer3 in puppet and add details to bottom of the new kickstart file.
publish new kickstart file and update wiki entry

Conclusion

I have reduced the time it takes for a dev to get a copy of a customers PE down from days to minutes and its now a self serve solution.
There is still more to refine in this but it's already full of win as I now get to do more of 'stuff that matters'

It's early days for us using this new setup and I am yet to work out an easy, effective way of notifying a $DEV when puppet has finished the buildout. Suggestions welcome.

cucumber tests ?
using libnotify via Dbus ? ( suggestion made at a recent DevOps Sydney meetup )
??

migrating drupal6 to mercury

Tue, 13 Jul 2010 19:37:54 -0400

Migrating an existing Drupal6 site to Mercury

Introduction

Mercury is a very fast hosting solution for hosting Drupal sites.
The following quote from http://getpantheon.com/mercury/what-is-mercury describes it perfectly.

Mercury is a drop-in replacement for your Drupal website hosting service that  
delivers break-through performance. Mercury can serve two-hundred times more  
pages per second and generate pages three times faster than standard hosting  
services.  
How is this possible?  
By standing on the shoulders of giants.  
We've integrated, borrowed, tuned and tweaked the fastest open-source hosting  
technologies available so that they can at last work perfectly with Drupal at  
the click of a button.  
You can read all of the technical details here.  
The tools and techniques available in Mercury have been around for some time,  
but were expensive and tricky to integrate with Drupal in the past.  
Now, finally, they are available for everyone.

The following is a post on how I migrate my sites from a standard Drupal6 hosting server to a Mercury based hosting server.
We manage all our sites in GIT. You can read more about how we are doing it in another post - drupal-git-workflow-pt1
One thing to mention here is that when we build a new Mercury server there is only 3 modules placed into sites/all/modules
cacherouter memcache varnish. The rest are kept as part of a sites individual repository.
This allows a site to be able to easily migrated between a Mercury and non-Mercury server.

HOWTO

Clone the site's repository into the sites/ folder:

git clone gitosis@gitserver:example.com.au.git

Initialize submodules:

git submodule init

Update submodules:

git submodule update

Place existing site offline (on Drupal6 server) to stop any new changes to database happening (use drush)
Dump the database and load up on Mercury server
Copy sites/example.com.au/files to the new Mercury server.(rsync or scp -r)
Configure the settings.php file to point to the right database. (if necessary)
add the following to the bottom of settings.php

    ##########################
    #
    # Mercury Settings
    #
    # Alter With Caution :)
    #
    ##########################
    # Varnish reverse proxy on localhost
    $conf['reverse_proxy'] = TRUE;           
    $conf['reverse_proxy_addresses'] = array('127.0.0.1'); 
    # Memcached configuration
    $conf['cache_inc'] = './sites/all/modules/memcache/memcache.inc';
    $conf['memcache_servers'] = array(
      '127.0.0.1:11211' => 'default',
    );
    $conf['memcache_key_prefix'] = 'example.com.au';
    ### END Mercury settings

Create an Apache vhost and restart apache should already be done by puppet
Setup caching modules for site

    DSITE="example.com.au"
    drush -l $DSITE en cacherouter
    drush -l $DSITE vset --yes cache 3
    drush -l $DSITE vset --yes cache_lifetime 0
    drush -l $DSITE vset --yes page_cache_max_age 600
    drush -l $DSITE vset --yes block_cache 1
    drush -l $DSITE vset --yes preprocess_css 1
    drush -l $DSITE vset --yes preprocess_js 1
    drush -l $DSITE vset --yes page_compression 0

clear any Drupal cache entries

drush -l $DSITE cache-clear all

test on port 9880 first then port 80 if successfull.
install cron

    0 * * * * /usr/bin/wget -O - -q -t 1 http://example.com.au:9880/cron.php

Conclusion

Please note that this is how we do the migration onto our own servers built following the Mercury install documents minus the Solr-search. Some adjustments may be necessary by you to follow these on a complete Mercury platform.

drupal-git-workflow-pt1

Thu, 01 Jul 2010 02:34:25 -0400

Managing Drupal sites with git - Part 1

At $WORK we build and manage quite a few Drupal sites.
In an effort to streamline things we are trialling a new workflow when working on Drupal sites.
The goals we wanted to achieve was to have everything in GIT and to have each customers site portable.
By portable I mean that it can easily be moved between our different drupal servers and also between Drupal multisite and dedicated Drupal hosting.
We have GIT repos of all the Drupal modules we use and use GIT submodules to drag these modules in for a site.
Each night a tarball of the sites mysql and sites/example.com/files/ is sent to a central server that serves these out via HTTPS (with AUTH of course).
This makes it very easy for a developer to grab production data to develop with.
Below is an example of our current workflow. This is only a day old and not really been put to a lot of use but in testing it seems to flow ok. Time will tell.

Setup a Drupal install if you don't have one: git clone gitosis@gitserver:drupal6.git /var/www/drupal
Clone the site's repository into the sites/ folder: git clone gitosis@gitserver:example.com.au.git /var/www/drupal/sites/example.com
- Change into freshly cloned sites folder cd /var/www/drupal/sites/example.com
- Initialize submodules: git submodule init
- Update submodules: git submodule update
Download and install the latest database backup. Take care to remove the contained email addresses.
Download the latest files folder backup and extract into the site's folder.
Configure the settings.php file to point to the right database.
Create an Apache vhost and /etc/hosts entry to point traffic to your local installation
Make your modifications.
Commit to your Git repository.
Push to the main repository if you have write access, otherwise notify someone who does.

Notes

Everything is kept in the domains site folder(sites/example.com) and nothing goes in sites/all or sites/default
Sites must be self contained, i.e., they should not make reference to anything from another sites folder, including sites/all.
All modules must be added as Git submodules.

It's very early days using this new workflow so I am not sure how well it will go but so far it appears to be a big step in the right direction.
Once we have mastered this and converted all our sites over to GIT we will then look to finding a better way to handle sql changes progressing through the dev,test,staging,production lifecycle.

install-gitosis-on-centos5

Mon, 10 May 2010 19:51:46 -0400

Howto: Install git, gitosis & gitweb on CentOS 5

Introduction

GIT is a powerful DVCS system. Gitweb is a Web-UI to visualize the repos. Gitosis takes the pain out of managing multiple GIT repos and all the ACL's.
It uses a git repos to manage the git repos with all connections done via a shared ssh/shell account and authentication is done via ssh private/public keys.

Installation

In order to install git, gitweb & gitosis we need to add the EPEL yum repository:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm

Once that is done we install git, git-web and gitosis:

yum install git gitweb gitosis

If all went well you should now have all three things installed. Now to setup gitosis to manage our repos.
At the core of gitosis is a special git repos called gitosis-admin The contents of this will be explained soon.

To get started you will want to copy your ssh public key to a tmp place on the server tmp/user.pub and then issue the following command:

sudo -H -u gitosis gitosis-init < /tmp/user.pub

This will setup gitosis ready to serve git repos from /var/lib/gitosis/
On your local machine, you'll now be able to clone the gitosis admin repository with the following command:

git clone gitosis@example.com:gitosis-admin.git

The gitosis-admin repository contains a directory named keydir/ and a file named gitosis.conf.

The keydir/ contains the SSH public keys for your users in files named in the convention of [username].pub. Each user of your git repositories will have their own file in keydir/ the username is for internal gitosis use, and needn't correspond with any shell username.

The gitosis.conf file contains a set of access control rules that can be used to provide people access to a particular repository. An access control block looks like this:

[group devs-rw]
    writable = iphone-project wiki drupal7
    members = mick alex adam mary

[group devs-ro]
    readonly = iphone-project wiki drupal7
    members = john

This block gives the users (e.g. keys in the keydir/) "mick", "alex", "adam" and "mary" write(push) access to the iphone-project, wiki & drupal repositories. Note that repositories in the "writable" list needn't exist before a user pushes to them, as gitosis will create the new repositories if needed. Also the user "john" has readonly(clone) access to the same 3 repos. He is not allowed to push.

To create a new repository, just add it to the writable list of a gitosis group. All repositories will have "clone" or "remote" URLs in the following form:

gitosis@example.com:$reponame.git

You may have as many "groups" as you need to support your workflow.

You should now have a fully working gitosis server. All that is left to do is to enable gitweb so that you can visualize all of the repos in one place.
Lucky for us this is almost completely done with the yum install earlier. A simple apache restart and that's it.
The following URL should bring up a working gitweb instance. Of course it will be empty to start with.

http://example.com/git/gitweb.cgi

Note: A repos created with gitosis above will not be visible by default in gitweb. A simple file permission change will take care of this.

chmod 755 /var/lib/gitosis/repositories/$REPOSNAME

Conclusion

You should now have a fully working central git server managed by gitosis and visualised by gitweb.

Cheers Mick

archlinux

Fri, 05 Mar 2010 04:25:43 -0500

Archlinux

Archlinux is my favourite personal workstation distribution.

evilwm + trayer + conky ( + xbindkeys )

evilwm + trayer + conky + terminator ( + xbindkeys )

openssh and X11 forwarding

Thu, 29 Jan 2009 06:17:00 -0500

Occasionally you may come across times where you need to run an X application on a remote, headless server.
Today was one of those occasions for me.
Installing xorg etc to add X to the server is a reasonable amount of work and not really a choice.
openssh has an option to tunnel X11 connections and auto setup the $DISPLAY environment variable on the remote host.
This can be done either on the fly with a switch to the ssh client or a permenant setup by changing the config file for openssh.

[lunix@godzilla ~]$ ssh -Y admin@10.20.1.254
admin@10.20.1.254's password:
Last login: Fri Jan 30 00:22:23 2009 from dynamic-20.home.lunix.com.au
/usr/bin/xauth:  creating new authority file /home/admin/.Xauthority
[admin@gateway01 ~]$ echo $DISPLAY
localhost:10.0
[admin@gateway01 ~]$

This extract of ~/.ssh/config shows how to make the X11 forwarding more permenant.

Host remotehost
ForwardX11 yes

Once this has been done and the $DISPLAY env. variable is set you should be able to fire up an application that displays its output via X.

[lunix@godzilla ~]$ wireshark

This will then fire up wireshark sending its output to X via localhost:10.0 which will tunnel it across ssh to your local screen.
Remeber this may be a bit slower than running it locally but it works.

I ran into problems this morning the first time I logged in with ssh -Y with the $DISPLAY env variable not being set.
To diagnose this I retried the ssh command with the extra options of -vv. Here is a snippet of the output.

[lunix@godzilla ~]$ ssh -vv -Y admin@10.20.1.254
admin@10.20.1.254's password:
...
debug2: x11_get_proto: /usr/bin/xauth  list :0 2>/dev/null
debug1: Requesting X11 forwarding with authentication spoofing.
..
..
debug1: Remote: No xauth program; cannot forward with spoofing.
..
Last login: Fri Jan 30 06:48:28 2009 from dynamic-20.home.lunix.com.au
[admin@gateway01 ~]$ echo $DISPLAY

[admin@gateway01 ~]$

$DISPLAY wasn't set. In my case the debug1: line mentioning a missing xauth program was the problem.
A simple yum install xorg-x11-xauth fixed this. ( centos 5 )

Hope this helps.

Usefull strace primer

Thu, 08 Jan 2009 23:09:00 -0500

I'm posting this here for both future use by myself and in the hope it's usefull to anyone else.

Have you ever racked your brain to find that bug? You know, the bug that you can't find in the source code,
but which appears with deliberate consistency when your code is compiled and run.
Hacker, meet strace.
strace is a utility that allows you to trace system calls and signals for a given command and its arguments.

Full article is over at the Redhat Magazine

Simple portscanning with bash and netcat

Tue, 30 Dec 2008 20:05:00 -0500

Today I was presented with the need to find all the machines on a network that have port 22 open.
This can easily be achieved with nmap.

sudo nmap -sS -p22 192.168.1.0/24

However the only box I had was a 'linux router' with a minimalistic linux install and no access to install extra packages.
What I ended up using was a very simple shell script with nothing more than a for loop and netcat.

    #!/bin/bash
    for h in {2..254};
    do
            nc -z 192.168.111.$h 22;
            if [ $? -eq 0 ]; then
                    echo -e "192.168.111.$h\n";
            fi
    done

The -z flag to nc ( netcat ) tells it 'no io' then I just test for the exit code it gives.
Simple.