Connecting a SIP Trunk to a Remote SIP Extension

$WORK gives me a SIP extension on their Asterisk server for when I work from home.
I have an Asterisk + FreePBX box at home.
I wanted to be able to make/receive $WORK calls from home with my existing hard phones.
I didn't want to make any changes to $WORK's Asterisk server.

The SIP Extension at $WORK has the following settings:

          name: 1234
      callerid: Mick Pollard
   canreinvite: No
       context: default
      dtmfmode: rfc2833
          host: dynamic
      insecure: No
           nat: Yes
          port: 5060
       qualify: yes
        secret: 1234
          type: friend
      username: 1234

After some time researching it turns out this is not actually that hard.
The following is to be all done within FreePBX at home.

• Add a SIP trunk (use the details of your SIP extension on the office asterisk server)
• Add an outbound route
• add an inbound route

Add a SIP Trunk

The main difference here is you should leave "USER Context" & "USER Details" blank.

Add an outbound route:

The dial rules used here should be tuned to match the extension prefixes in use at your $WORK.
We have 4 digit extensions starting with either a 12 or a 22. I have also add a special prefix of 9|.
which allows me to route a call via $WORK. This is important so that clients get $WORK's callerID and not my home number !

Add an Inbound Route (optional)

I currently have an inbound route that allows any calls to go straight to a queue but you may want to change this.
You just need to create an inbound route that will match your WORK extension.

You should now be abe to make and receive work calls on your existing phones at home.

Share any website in Google Reader

I read a lot of feeds and Google reader makes this easier.
It also allows me to 'share' the ones I find useful/interesting as an RSS feed for others to read.
One thing I found is that I sometimes come across a great site or blog post from a source other than Google Reader
that I want to share. This small Chrome/chromium bookmarklet allows you to share any website you visit.
You just create a bookmark on the Chrome toolbar with the URL set to the following javascript snippet.

javascript:var%20b=document.body;var%20GR________bookmarklet_domain='http://www.google.com';if(b&&!document.xmlVersion){void(z=document.createElement('script'));void(z.src='http://www.google.com/reader/ui/link-bookmarklet.js');void(b.appendChild(z));}else{}

Installing Graylog2 via Ubuntu Packages

These packages and docs are currently beta.
The deb's are built on Ubuntu Lucid amd64 however should work on both i386 & amd64.

Please report bugs in this HOWTO or the packaging to me at aussielunix at gmail dot com.

graylog2-server

This installs graylog2-server and it's dependencies (mongodb-stable from 10gen) etc.
The graylog2-server will install all files to /opt/graylog2-server & a config file at /etc/graylog2..conf.
Be prepared as the java stuff drags in a lot of deps on a clean minimal Lucid install. (176 packages for me)

1) add public key for the 10gen mongo repository

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

2) add public key for the aussielunix (Mick Pollard) PPA

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv D77A4DCC

3) add the following four lines to /etc/apt/sources.list

# 10-gen's mongodb repos
deb http://downloads.mongodb.org/distros/ubuntu 10.4 10gen
# lunix's graylog2 debs
deb http://ppa.lunix.com.au/ubuntu/ lucid main

4) let apt see the new repositories

sudo apt-get update

5) install graylog2-server plus its deps - including java and mongodb

• This will take a while - go make coffee

sudo apt-get install mongodb-stable graylog2-server

6) secure mongo - add authentication

• add an admin user
• add a user to mongo for collection 'graylog2'

lunix@ubuntu-dev01:~/$ mongo
use admin
db.addUser('admin', 'admin-mongo-passwd')
db.auth('admin', 'admin-mongo-passwd')
use graylog2
db.addUser('grayloguser', 'grayloguser-mongo-passwd')

7) tell graylog2-server about the mongo auth

• edit /etc/graylog2.conf

mongodb_useauth = true
mongodb_user = grayloguser
mongodb_password = p4ssw0rd

8) turn mongo security on - it's off by default

• edit /etc/mongodb.conf

auth = true

9) restart mongo

sudo service mongodb restart

10) start graylog2-server

sudo service graylog2-server start

Conclusion

You should now have a working graylog2-server.
You can check the process tree for a mongodb instance and a java instance and that port UDP/514 is open.
You can now modify the syslog config on the graylog2-server host to send its data to 127.0.0.1:514
Move on to graylog2-web install/configure now.

graylog2-web

This installs graylog2-web and some of it's dependencies.
The graylog2-web package will install all of it's files to /opt/graylog2-web.
All of the gem dependencies have been vendored in.
The version of rubygems is too old in the Lucid repositories so I make use of a thirdparty PPA.
This PPA is from Mackenzie Morgan - a Ubuntu Developer - https://launchpad.net/~maco.m/+archive/ruby

1) add Mackenzie's PPA

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:maco.m/ruby

2) let apt see the new repositories

sudo apt-get update

3) install graylog2-web

sudo apt-get install graylog2-web

4) install budler

sudo gem install bundler

5) review/edit some rails configs:

config/mongoid.yml
confg/email.yml
config/general.yml

6) start graylog2-web as a daemon

script/rails server -eproduction -d -p3000

Conclusion

You should now have a working graylog2 server & web.

With Hardy getting a bit long in the tooth so are the versions of software.
Recently a client has tasked me to add varnish to their existing wordpress cluster.
The servers they are using are all Ubuntu Hardy and the version of Varnish in the repos is ancient (1.0.3-2)
It turns out it's not that hard to build a .deb of a more modern version of Varnish.
The following GIST shows the simple steps I used to build a Ubuntu package of Varnish 2.0.6.

Funny Tech Support Email Number 1

In the late 1990's we purchased a few small ISP's and whilst auditing their old servers (terrible mess) I come across this beauty.
I thought I'd share this, plus one other, from another ISP, I found for this years System Administrator's Appreciation Day.
They both made me laugh back then and still make me laugh today. Oh how I miss the days of ISP land.

Funny Tech Support Email Number 2

In the mid 2000's whilst working at an ISP the following email arrived.
It gave us all a good laugh at the time and today I share with you all for System's Administrator Appreciation Day.

Introduction

I am currently working in a web development shop. We develop and maintain a range of websites/webapps for customers.
At $WORK we have many varied customers each with their own unique Production Environments (PE from here on in).
Our $DEVs are also free to run any *nix based OS on their workstations.
One of the challeges we have had in the past is making a copy of a customer's PE easily available to our $DEVS.
This used to require lodging a task in Redmine and waiting for a Systems Admin to build you a VM on a central VM server.
This post will show how we are now using common FOSS tools to give enable to $DEVS to have VM's on their own workstations that mimick a customer's PE.

Whilst I make mention of some specific tools in this post they can be swapped out in most places for alternates however I have not tested the
alternatives yet. ie: puppet/chef, mercurial/git, centos/ubuntu.

Notes on the Challenge

• needs to be simple not an obsticle - if it's slower than just getting a sysadmin to build it for you then its a fail
• simplicity generally means easy to fix when something goes wrong in the wheel.
• self-serve - no waiting for sysadmins
• visability - everything in DVCS and Redmine (project managent software)
• needed to be repeatable - $DEVs needed to easily be able to build, destroy and build again
• relatively self documenting - read the kickstart or puppet manifests
• I hate OS images - They're big, cumbersome and pain in my..err..storage See - Golden Image or Foil Ball?

After spending a fair amount of time on this and looking at many of the VM/cloud management solutions out there I have decided that while some are very nice and useful I do not believe they are suiteable for our situation. Most VM/cloud management tools are built around the "OS Image" and require each workstation to 'register' as a node.

Current solution

After doing the full circle of research we are now using a simple collection of existing tools.
It was all there staring me in the face all along. Libvirt, virt-install kickstart puppet, mercurial and a wiki entry. A $DEV just needs to make sure he/she has libvirt, virt-install, virt-viewer installed.
We are using KVM to provide the virtualisation layer but through the use of libvirt you should be able to use any libvirt compatible virtualisation provider.(virtual box etc)

Technologies used

• a httpd server (nginx, apache etc) - to serve kickstart + yum repos/mirror
• Own yum repos + centos mirror ( again ubuntu mirror etc )
• puppetmasterd ( or other CF tool ie: chef etc ) with autosign turned on (we have a separate puppetmaser for the $DEVS)
• some kickstart files - I use one per customer and bootstrap puppet from the %POST section
• libvirtd + KVM/qemu - could be any supported virtualisation software supported by libvirt
• python-virtinst + virt-viewer
• dhcpd
• forward and reverse dns - puppet will fail to work as expected without it ( I use powerdns-recursor for demos as it exports /etc/hosts )
• redmine - we make use of Redmine's ACL's to visualize the repos for puppet and kickstart files per customer

Devs

The following is the steps needed for a $DEV to deploy a customer's PE.

• Check network page and grab an available network mac to use (this is used for dhcp & dns so puppet works properly)
  and the name of the customers kickstart file.
• update wiki page to say that network mac is in use.

• deploy a VM on their workstation. - See Libvirt tips

  virt-install --connect qemu:///system --accelerate -n virt01 -m 54:52:00:37:2E:B9 -r 1024 --vcpus=1 --disk pool=lvm,bus=virtio,size=20 --vnc --os-type linux --os-variant=rhel5 --network=network:default -l http://192.168.1.250/os/CentOS/5.5/os/x86_64/ -x "ks=http://192.168.1.250/ks/project_customer1.ks"

This will take advantage of the fact that both CentOS and Ubuntu have the necessary PXE files stored in their mirrors for booting the installer.

• wait approx 10 or so minutes and they have a clone of the customer's PE on their workstation ready to deploy to and hack on. see notes in conclusion below

New customers

The following is what's involved in preparing for a new customer:

• A new customer has a VM/server provisioned in a DC by a hosting company.
• I grab the current package list and make a kickstart file to replicate the install locally
• Create a new project_customer3 in puppet and add details to bottom of the new kickstart file.
• publish new kickstart file and update wiki entry

Conclusion

I have reduced the time it takes for a dev to get a copy of a customers PE down from days to minutes and its now a self serve solution.
There is still more to refine in this but it's already full of win as I now get to do more of 'stuff that matters'

It's early days for us using this new setup and I am yet to work out an easy, effective way of notifying a $DEV when puppet has finished the buildout. Suggestions welcome.

• cucumber tests ?
• using libnotify via Dbus ? ( suggestion made at a recent DevOps Sydney meetup )
• ??

Migrating an existing Drupal6 site to Mercury

Introduction

Mercury is a very fast hosting solution for hosting Drupal sites.
The following quote from http://getpantheon.com/mercury/what-is-mercury describes it perfectly.

Mercury is a drop-in replacement for your Drupal website hosting service that  
delivers break-through performance. Mercury can serve two-hundred times more  
pages per second and generate pages three times faster than standard hosting  
services.  
How is this possible?  
By standing on the shoulders of giants.  
We've integrated, borrowed, tuned and tweaked the fastest open-source hosting  
technologies available so that they can at last work perfectly with Drupal at  
the click of a button.  
You can read all of the technical details here.  
The tools and techniques available in Mercury have been around for some time,  
but were expensive and tricky to integrate with Drupal in the past.  
Now, finally, they are available for everyone.  

The following is a post on how I migrate my sites from a standard Drupal6 hosting server to a Mercury based hosting server.
We manage all our sites in GIT. You can read more about how we are doing it in another post - drupal-git-workflow-pt1
One thing to mention here is that when we build a new Mercury server there is only 3 modules placed into sites/all/modules
cacherouter memcache varnish. The rest are kept as part of a sites individual repository.
This allows a site to be able to easily migrated between a Mercury and non-Mercury server.

HOWTO

• Clone the site's repository into the sites/ folder:

git clone gitosis@gitserver:example.com.au.git

• Initialize submodules:

git submodule init

• Update submodules:

git submodule update

• Place existing site offline (on Drupal6 server) to stop any new changes to database happening (use drush)
• Dump the database and load up on Mercury server
• Copy sites/example.com.au/files to the new Mercury server.(rsync or scp -r)
• Configure the settings.php file to point to the right database. (if necessary)
• add the following to the bottom of settings.php

    ##########################
    #
    # Mercury Settings
    #
    # Alter With Caution :)
    #
    ##########################
    # Varnish reverse proxy on localhost
    $conf['reverse_proxy'] = TRUE;           
    $conf['reverse_proxy_addresses'] = array('127.0.0.1'); 
    # Memcached configuration
    $conf['cache_inc'] = './sites/all/modules/memcache/memcache.inc';
    $conf['memcache_servers'] = array(
      '127.0.0.1:11211' => 'default',
    );
    $conf['memcache_key_prefix'] = 'example.com.au';
    ### END Mercury settings

• Create an Apache vhost and restart apache should already be done by puppet

• Setup caching modules for site

    DSITE="example.com.au"
    drush -l $DSITE en cacherouter
    drush -l $DSITE vset --yes cache 3
    drush -l $DSITE vset --yes cache_lifetime 0
    drush -l $DSITE vset --yes page_cache_max_age 600
    drush -l $DSITE vset --yes block_cache 1
    drush -l $DSITE vset --yes preprocess_css 1
    drush -l $DSITE vset --yes preprocess_js 1
    drush -l $DSITE vset --yes page_compression 0

• clear any Drupal cache entries

drush -l $DSITE cache-clear all

• test on port 9880 first then port 80 if successfull.
• install cron

    0 * * * * /usr/bin/wget -O - -q -t 1 http://example.com.au:9880/cron.php

Conclusion

Please note that this is how we do the migration onto our own servers built following the Mercury install documents minus the Solr-search. Some adjustments may be necessary by you to follow these on a complete Mercury platform.

Managing Drupal sites with git - Part 1

At $WORK we build and manage quite a few Drupal sites.
In an effort to streamline things we are trialling a new workflow when working on Drupal sites.
The goals we wanted to achieve was to have everything in GIT and to have each customers site portable.
By portable I mean that it can easily be moved between our different drupal servers and also between Drupal multisite and dedicated Drupal hosting.
We have GIT repos of all the Drupal modules we use and use GIT submodules to drag these modules in for a site.
Each night a tarball of the sites mysql and sites/example.com/files/ is sent to a central server that serves these out via HTTPS (with AUTH of course).
This makes it very easy for a developer to grab production data to develop with.
Below is an example of our current workflow. This is only a day old and not really been put to a lot of use but in testing it seems to flow ok. Time will tell.

• Setup a Drupal install if you don't have one: git clone gitosis@gitserver:drupal6.git /var/www/drupal
• Clone the site's repository into the sites/ folder: git clone gitosis@gitserver:example.com.au.git /var/www/drupal/sites/example.com
  • Change into freshly cloned sites folder cd /var/www/drupal/sites/example.com
  • Initialize submodules: git submodule init
  • Update submodules: git submodule update
• Download and install the latest database backup. Take care to remove the contained email addresses.
• Download the latest files folder backup and extract into the site's folder.
• Configure the settings.php file to point to the right database.
• Create an Apache vhost and /etc/hosts entry to point traffic to your local installation
• Make your modifications.
• Commit to your Git repository.
• Push to the main repository if you have write access, otherwise notify someone who does.

Notes

• Everything is kept in the domains site folder(sites/example.com) and nothing goes in sites/all or sites/default
• Sites must be self contained, i.e., they should not make reference to anything from another sites folder, including sites/all.
• All modules must be added as Git submodules.

It's very early days using this new workflow so I am not sure how well it will go but so far it appears to be a big step in the right direction.
Once we have mastered this and converted all our sites over to GIT we will then look to finding a better way to handle sql changes progressing through the dev,test,staging,production lifecycle.

Howto: Install git, gitosis & gitweb on CentOS 5

Introduction

GIT is a powerful DVCS system. Gitweb is a Web-UI to visualize the repos. Gitosis takes the pain out of managing multiple GIT repos and all the ACL's.
It uses a git repos to manage the git repos with all connections done via a shared ssh/shell account and authentication is done via ssh private/public keys.

Installation

In order to install git, gitweb & gitosis we need to add the EPEL yum repository:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm

Once that is done we install git, git-web and gitosis:

yum install git gitweb gitosis

If all went well you should now have all three things installed. Now to setup gitosis to manage our repos.
At the core of gitosis is a special git repos called gitosis-admin The contents of this will be explained soon.

To get started you will want to copy your ssh public key to a tmp place on the server tmp/user.pub and then issue the following command:

sudo -H -u gitosis gitosis-init < /tmp/user.pub  

This will setup gitosis ready to serve git repos from /var/lib/gitosis/
On your local machine, you'll now be able to clone the gitosis admin repository with the following command:

git clone gitosis@example.com:gitosis-admin.git

The gitosis-admin repository contains a directory named keydir/ and a file named gitosis.conf.

The keydir/ contains the SSH public keys for your users in files named in the convention of [username].pub. Each user of your git repositories will have their own file in keydir/ the username is for internal gitosis use, and needn't correspond with any shell username.

The gitosis.conf file contains a set of access control rules that can be used to provide people access to a particular repository. An access control block looks like this:

[group devs-rw]
    writable = iphone-project wiki drupal7
    members = mick alex adam mary

[group devs-ro]
    readonly = iphone-project wiki drupal7
    members = john

This block gives the users (e.g. keys in the keydir/) "mick", "alex", "adam" and "mary" write(push) access to the iphone-project, wiki & drupal repositories. Note that repositories in the "writable" list needn't exist before a user pushes to them, as gitosis will create the new repositories if needed. Also the user "john" has readonly(clone) access to the same 3 repos. He is not allowed to push.

To create a new repository, just add it to the writable list of a gitosis group. All repositories will have "clone" or "remote" URLs in the following form:

gitosis@example.com:$reponame.git

You may have as many "groups" as you need to support your workflow.

You should now have a fully working gitosis server. All that is left to do is to enable gitweb so that you can visualize all of the repos in one place.
Lucky for us this is almost completely done with the yum install earlier. A simple apache restart and that's it.
The following URL should bring up a working gitweb instance. Of course it will be empty to start with.

http://example.com/git/gitweb.cgi

Note: A repos created with gitosis above will not be visible by default in gitweb. A simple file permission change will take care of this.

chmod 755 /var/lib/gitosis/repositories/$REPOSNAME

Conclusion

You should now have a fully working central git server managed by gitosis and visualised by gitweb.

Cheers Mick