Peter Blair

Easy testing with Perl

I always place a “t” directory right under my Perl lib dir, which is a great place to run all of my regression tests.

I’ve found the easiest way to get the harness up and running is to use a Makefile, and a Perl 1 liner:


$ cat Makefile
.PHONY:test all

all: test

test:
perl -MTest::Harness -e 'runtests sort @ARGV' *.t

So, when you call "make" or "make test", it will test against every .t file. No fuss.


Perl decorators

One of the great things that I enjoy about Python, are the decorators. I try to use them whenever possible in my Python projects, and sorely miss them at times in my Perl projects.

Last week I was thinking of profiling my Perl applications remotely and was pining for decorators. With Perl 5.10 lacking decorators, I created a Perl filter package to create “decorator like” syntax-sugar to support the emission of certain metrics like:

– Label assigned to function
– Time elapsed
– Return val

This triplet is then sent via UDP to some listener that can do with it as it may. Aggregate it, store it in a noSQL store, whatever.


vim: IP Extraction

At work I deal with lists of IP addresses all of the time, and I find it handy to be able to strip out all IP addresses from a region of a text file. It’s easy enough to pass a file through a filter to accomplish this, but I like to do it in buffer via vim so that I don’t have to resort to creating a scratch file.

The first function (mapped to CTRL-i) will extract all ips in the visual region, then the second function (mapped to CTRL-u) will remove repeat addresses.

Hope that you find this handy!


Using python decorators for authentication checks

Consider the following example: A class decodes a JSON string and inspects it for certain authentication tokens. These token can be anything representing an authenticated session, I’m using the simple boolean check for my example.

By using a decorator like the following:

You can use the decorator above any method that handles JSON requests for authentication.


Ruby and class methods

I’m just starting to learn to programme in Ruby, and wanted to play with some class methods to create a nice looking and expressive syntax.


drink = Coffee.with :milk, :cream, :sugar

When run:

$ ./coffee.rb
Creating coffee..
with Milk
with Cream
with Sugar
Cost of coffee: 2.6


Creating a team git repository

At work, we’re looking into using GIT to replace SVN for our team SCM.  I’m looking into “hosting” a repository on my workstation, and have outlined a few steps to make this as painless as possible.

Step 1: Create a local user “git”

This should be as easy as issuing “useradd git” as root, and setting all of the required permissions.  I’m going to assume that the home dir for git is “/home/git”, and all of my scripts will make the same assumption.

Step 2: Create script to restrict SSH access to just git

We’re going to handle permissions by sharing SSH keys.  But, in order to stop users from just SSHing into the machine, and messing with things, we’ll use this wonderful tiny script to handle this.  Install the script as described in the page, set the permissions, and we’re done.

Step 3: Import a test SSH key

From another computer, generate a public SSH key, and paste the public key into git’s .ssh directory, within the file “authorized_keys”.  Take note that we’ll restrict certain SSH actions:


command="/usr/bin/gitserve git rw",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAAA...

Step 4: Create a utility script for creating new bare projects

We’re only concerned with handling new bare project at the moment, and not importing existing project.  So, we can use this little shell script to do the needful:

Now, on the server, create a new project:

git@slowpoke:~$ bin/create_new_project.sh foo
Initialized empty Git repository in /home/git/tmp/foo/.git/
[master (root-commit) 988e2da] initial project
0 files changed, 0 insertions(+), 0 deletions(-)
create mode 100644 README
Cloning into bare repository /home/git/foo.git...
done.

And our project is now available for consumption.

Step 5: Clone from remote

On a remote machine (my laptop in this case) I’ll clone the new git project.  Note, I’ve already had my public ssh key for my laptop installed under my workstation’s “git” user’s .ssh/authorized_keys file, so no extra authentication should be required.


pblair@pblair-laptop:~$ rm -rf stuff/
pblair@pblair-laptop:~$ mkdir stuff && cd stuff
pblair@pblair-laptop:~/stuff$ git clone git@slowpoke.internal.WORK.com:foo.git
Initialized empty Git repository in /home/pblair/stuff/foo/.git/
user git authorized
remote: Counting objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.

pblair@pblair-laptop:~/stuff$ ls -F
foo/
pblair@pblair-laptop:~/stuff$ cd foo/
pblair@pblair-laptop:~/stuff/foo$ echo "indeed" >> README

pblair@pblair-laptop:~/stuff/foo$ git commit -m "indeed" -a
[master 2c0ca9f] indeed
1 files changed, 1 insertions(+), 0 deletions(-)
pblair@pblair-laptop:~/stuff/foo$ git push
user git authorized
Counting objects: 5, done.
Writing objects: 100% (3/3), 242 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To git@slowpoke.internal.WORK.com:foo.git
988e2da..2c0ca9f master -> master


Continuous Integration: Developer Commit and Staging Flow


The Flowchart outlines a simple path in which a developer committing to branch breaks the regressions tests, triggering the CI server to lock the repository from subsequent commits. The expectation is that the developer will be notified via build report that the branch is effectively broken, and they must ensure that their subsequent commit will pass all regression tests (note, the developer should be running regression tests against their branch before merging into the master, but not everyone will remember this). The SCM will be unlocked by some mechanism, be it a particular message indicating that the subsequent commit will fix it, re-opening the SCM to commits.

A successful regression test will trigger the SCM to be locked until the current build it successfully deployed. The staging server will create a new chroot image based on a meta file containing all software packages, creating said software packages from the most recent commit to the SCM.

Upon successful stage, the CI server will request from the Deployment server that this new chroot environment be deployed to whatever environment (QA, Testing, Production, etc) for any subsequent User Acceptance testing, or for the truly brave, live production.

Once deployed, the CI server will unlock the SCM, allowing developers to commit more merges into the master branch, thus triggering this process over again.


devops: continuous deployment infrastructure

Looking at how to handle a continuous deployment system in conjuncture with a continuous integration system. My thoughts are to use a CI system like Jenkins to automate the testing of all recently committed git repositories, and to kick off a check list of dependencies.

The general idea would be:

1. Developer commits to git/master
2. CI pulls from git/master
2.1 CI runs all regression tests against git/master
2.1.1 CI returns report to developer and halts if any fail
2.2 CI Looks up the package manifest of a given project, and pushes project to staging server
3. Staging server determines which OS project belongs under (Debian, etc) and generates debootstrap image if none exist
3.1 Staging server creates clone of debootstrap image, and apply all .deb packages from manifest to image
3.2 Staging server stores image to shared file space
4. CI calls deployment server to deploy the image to a given environment
5. Deployment server checks manifest, and determines which class of server image is to be deployed, and attempts to deploy the image to appropriate Dom0 for the given environment, as the appropriate DomU

Additional coverage testing would be included in the regression testing, and the first deployment would be to an acceptance testing environment. This setup is similar to http://morethanseven.net/2011/03/20/A-continuous-deployment-example-setup.html and other ideas by kitchensoap.com.


Get Adobe Flash playerPlugin by wpburn.com wordpress themes