2011 started with me taking a few months out from The Approachable Geek to form and built a new company.  Over the first 4 months of the year weddingtales.co.uk was born, shaped, coded and launched along with James Rutherford though The Difference Engine.  By the end of 2011 we’ve handled over 1,500 wedding guests and nearly 10,000 wedding photos.

Whilst building weddingtales.co.uk I did a few consulting jobs, mostly data thrashing including preparation for a huge Sage data import (155,000 product records needed cleaning before they got into the system)  and some quick and dirty Excel data jobs too.

In March I had one of my most challenging yet rewarding jobs of the year delivering a one day course for 12 to 17 years olds getting them up and running with Arduino circuit boards (and vibrobots).  A really tough day (for me) but one of the most satisfying.

The early summer saw some more Excel jobs, including reducing the hours a local estate agency spent matching payments to tenants by as much as 15 hours a week by building a smarter and more automated spreadsheet.  I also helped deliver the Blog It! masters course for Newcastle University, learning a lot about the common problems non-techies face with using domestic IT kit along the way.

Late summer though to the end of the year has had be focussing on two main projects, one writing complex data-capture scripts and social data storage mechanisms and the other being my most challenging project to day – running the Design It Built It conference.  More on that another time,

2012 is virtually upon me and as I type this I’m booked up until April which is a pleasant situation to be in.  DIBI is going to take up a lot of time, as is a new longer running project, but more news on that another time.

Redis is a key-value store, making is very fast and easy to push bits of data into a store without them having to fit into a per-defined schema.  It’s got great drivers for all of the languages I work in too as well as a command line interface.  Getting it up and running was reasonably simple thanks for Ubuntu’s apt-get command.  There are numerous great guides to Redis, what it does and how to get  started out there.

None of this will be anything new to hardcore geeks, but what surprised me was how I’ve ended up using Redis over the last few months.

A quick, dirty and brilliant cross program message queue

I suddenly found myself using Redis as a job queue thanks to it’s easy lists.  I can push driver values into a list with one process and pop them out from another.  You can use it as a First In Last Out stack by pushing and popping from the same end of the list, or as a First In First Out queue by pushing onto one end and popping from the other.  Best of all I could push jobs in with one process (a website front end for example) and pop them off with another (python scripts on the command line maybe).  Redis made this first step into cross process jobs trivial, in fact I did it without realising it.

A self-sufficient error log

Redis supported fixed size collections known as capped collections.  Sometimes it’s very useful to has a running log of what’s going on inside a program and logging is the way to do this.  Most frameworks have mechanisms to log out to a file on disk which is great but it leaves lots of things to think about at some point including managing the files, watching their size etc.  If you don’t need to keep this data for a long time, Redis provides a super quick to implement and very efficient running log.  Push log entires into it, read them off on an admin screen, forget about having to look after anything.  Also, given the ease of sharing a Redis instance between different processes you’ve suddenly got a a system wider logger with all the help that brings.

A handy store when data mangling

I’ve ended up doing a lot of data crunching over the last ten years.  A lot of this involves writing one of scripts to tear up text files and squirt out bits of it into a more usable form.  It’s hacking at it’s most pure I believe and having a mixed bag of tools to throw at problems is really helpful.  Being able to bolt together bits of perl, shell script, awk, sed, php, python and even Microsoft Excel is very powerful.  Often data crunching is a multi-step process with each step being glued together by squirting out data into text files and then reading it into the next step.  Because I can push data into Redis from a bunch of sources easily it has become an excellent binary-safe alternative to text files.  One script rips something up and pushes bits of it into Redis lists or keys, the next pulls out the bits it needs, and away we go.

I’ll probably write up my adventures in MongoDB at some point, but not just yet.

Parliament Hack was being run as part of Parliament Week which aims to help people improve their understanding of both the parliament and democracy.  By opening up a bit of data and letting 40 or so geeks run riot over it a stack of projects were built that help people interact or understand the inner workings of our government a little better.

So what did I build?  Firstly I got my head down and built Oh Lordy! a reasonably simple mash-up making it easy to search the House of Lord Register of Interest.  This data is already published but it’s grouped by peer, which is interesting but not that handy.  Wouldn’t it be more interest to be able to search the register to see who has interests in say… Tescos?  That’s what Oh Lordy does.  It’s a mix of mongodb, codeigniter reactor and the data was crunched using python and beautifulsoup.

On Sunday morning (after a few hours sleep under a desk) I started looking at the rather excellent unemployment data.  It’s really detailed and goes back years.  Unfortunately it was supplied in an Excel spreadsheet which isn’t brilliantly handy.  Using the same tool stack I built the UK Unemployment API to liberate the data.

I won a prize for Oh Lordy! too, which was nice, and whilst presenting  UK Unemployment API found out that the thing I had built already exists in the form of NOMIS’s API, and bloody good it is too.  Oh Lordy! also got tweeted about by Lord Knight too, which was pretty cool.

What makes Shopify really impressive is you can write (or buy) themes for your shop which are extreamly customisable.  The functional logical of the templates if written in a language called Liquid.  At first Liquid looks clunky but it’s actually very powerful when used in combination with Shopify’s available data.

Shopify also let you add into the theme any other assets you might need, css, images, JavaScript etc. and this lead me to a frustrating problem which it seems worth writing up.  Looking back it’s screamingly obvious, but at the time it was far from it.

I was trying add a lightbox to a product, a simple enough problem normally.  Because the site was already using jquery I chose a jquery based lightbox rather than introduce yet another set of JS libraries.  I added the JS files, the css and the image assets to the theme, bolted on the right selectors etc and it was up and running.  Clicking on the thumbnail caused a nice big image to be overlaid.

The problem was that the “Next” and “Close” buttons had broken images.  When you upload assets to Shopify it has no support for sub-directories in the assets directory so the paths of “../images/close_label.gif” needed modifying.  I tried “close_label.gif”, but that failed too, with the generated path being that of the page we were on, not the location of the JS file.  Very frustrating.

The problem was that I needed to get Liquid to render that image path so that it would take on the location as specified by Shopify.  JS assets however, unlike .liquid files, are not parsed, and so the {{ ‘close_label.gif’ | asset_url }} tag is never replaced.  I tried every dirty hack i could think of but nothing worked.

Finally, after swapping out jquery-lightbox for slimbox it worked !  Why?  Because Slimbox uses CSS properly to style to pop-over rather than having the JS write the images into the DOM.  Proper separation of display and content wins the day.  CSS taken on the relative paths to the images based on it’s location, rather than the location of the page as the JS method caused.

The moral of the story, separation of presentation and content is important.

This post completely fails to capture just how blown away I was not only with the young developers who attended the Newcastle event but with those whom I watched present on the live stream too.  It’s the most amazing and humbling thing to watch a bunch of people more than half your age knock spots off you not only technically but also in their ability to learn and adapt.

YRS is a week long event when volunteers (known as mentors) help a bunch of young developers (not kids, never call these guys kids) to use open data to build stuff.  Many of those taking part have never programmed before, some are already (at 15) old hands and show me up. At the end of the week they all truck down to London for a big show and tell (with prizes).

The Centre for Life here in Newcastle hosted us for the whole week and moved mountains to get the internet access we needed (slow connections are the bane of every developers life).  We were based out in the middle of a science museum too, rather than locked away, which was very cool.

Over the course of the week 7 guys under the age of 18 formed teams and with the help of 7 mentors, formed teams and build 4 stand alone projects, either in completed form, or as worked through and credible mockups.  We had a location aware iPhone app to help people make smart choices about their renewable energy decisions, a platform game where the display and the difficulty were influenced by weather and crime data, a fully working Google Earth implementation that mapped school results against voting records to look for correlations and a fully working Google Chrome QR code plugin (built by two guys who’d never programmed anything at all).

Sadly I couldn’t make it to London to watch the pitches but I watch the live stream for as much of the day as possible (the recordings are online).  I’m never been so flabbergasted in my life. Most of the teams presenting knocked the spots of several start up companies I could name, and seeing an 8 year old hand a government official his ass on a plate for the state of their public data was priceless.

One of the judges was Mike Butcher from TechCrunch EU.  His tweet through the day ay it all for me.

Today I’m a judge at the #yrs2011 event where school kids pitch apps based on gov/open data (link)

It does RT @timROGERS: @mikebutcherThis has got to beat judging that Apprentice crap… #yrs2011 (link)

Loving #yrs2011, see live streamhttp://t.co/pCfkiiB wish I’d been before. Every European country should do this (link)

I was a judge on The Apprentice. #yrs2011beats the crap out that cc. @lordsugar(sorry, but it does) (link)

YRS 2012 is open for registration now, and you can could me in.

Whenever I mention things like “teaching blogging” or introductions to social media all of my tech friend sneer. “It’s easy”, “you just do it” and “it can’t be taught” are phrases I’ve heard time and time again.  Annoyingly they are sort of right too, but what they don’t give credit to is that all of these things can be really intimidating for people who don’t live their entire lives online.  That’s where I come in, to provide a guidance, a bit of non-patronising hand holding, and some practical help if people are stuck.  I don’t pretend to have all the answers, and in many cases there is no one right way to do it (though there are often lots of examples of how not to do it).  I’m there to facilitate, not dictate.

Being no writer I was handling the “tech side” of blogging, from getting started on Blogger.com, wordpress.com (and a few others), though RSS, linking, embedding media, building an audience, blogging for business and finally a little about SEO and monetization. Being a writer Judith was working with students to improve their content, writing style and other such things.

Over the course of the 6 weeks I’ve learnt a lot, although most of it was how to do IT support for an entire class all on different kit and how badly the universities are set up to provide courses outside of office hours (no IT support after 5pm means all visiting teaching staff are on their own!) I’ve also got a lot better at helping to demystify some of the complexities of this internet thing too I hope.

Practical things aside, I’ve been amazed at the posts that my class have produced. I really enjoyed the first few posts from The Reluctant Buddhist, who sadly didn’t write a huge amount more. A post about self harming on the Antics Of A Brittle Asthmatic is a truly seductive and moving piece of writing. Little insights into the world of Estate Living have been really interesting (and I love “the view from the farm” snippet written in the persona of an animal in each one). I’m also hopeful that Musician Verses Student blossoms into an awesome online juxtaposition of lifestyles.

All of the students who stuck with the course right through produced work I was proud of, and I hope that they’ve found a sense of self confidence and assurity that means they will carry on blogging, either in their current guises or in new projects.

Thinking Digital Timelapse from Oli on Vimeo.

View from the roof of the Baltic from Oli on Vimeo.

Tynemouth in time from Oli on Vimeo.

From the roof of BOHO One, Middlesbrough from Oli on Vimeo.

A Gateshead tree in the snow from Oli on Vimeo.

I’ve been making use of an amazing open source project called Canon Hardware Development Kit I’ve been producing timelapse films for the last couple of months. CHDK is installed to a memorycard and replaces the normal software on a range of cheap Canon cameras, unlocking a range of features that are (at best) only available on the top of the range Canon kit, or completely new.

Installing CHDK is a touch fiddly, but the instructions are pretty good and if you’re using Windows or Mac there are installer such as cardtricks which simply the process.

CHDK does not run on all cameras, not even all Canon cameras, so check the website carefully before spending money. I have had success with a Canon A480 from Argos (which was on offer). You will also need an external powersupply which are fearsomely expensive from Canon, however an unofficial one from Ebay was only £15.

CHDKs biggest feature is that you can write scripts for the camera, and it will automatically process. In this case I got a timelapse script which takes a picture every minute and saves it. Using a nice large memorycard (4GB) you have plenty of space for thousands of photos.

I’ve had my camera installed in a range of great locations in the North East for the last month, mostly thanks to friends securing me access to places I’d never have got into without them. Installations are mostly in the MacGyver style, using plastic bags and ductape to make everything waterproof and less likely to blow away.

Photos thanks to David Coxon.

Once you get the photos back it’s just a matter of compiling them down into a film, for which there are lots of options. I’m using mencoder, which is a bit fiddly but has lots of options.

Finally, the films go up to my video hoster of choice, Vimeo.

Mostly they are using laptops with Windows 7 or Vista, with a scattering of iPads. Ignoring the iPads (which seem to result in far smoother operations), they’re all using OEM installs, and their machines are virtually unusable.

Lets start with their default browsers. Given that they nearly all equate the Internet Explorer icon with “the Internet” is gives a good idea of the penetration of Firefox and Chrome outside of the geek community. Because they don’t come installed already, nobody had known they could install another browser.

The newest version of IE seems less hateful than the proceeding versions, but the massively debilitating thing is that they come covered in crapware. From vendor specific toolbars, through Bing, and finally finishing with some invasive malware (which they nearly all have picked up), their browsers are a mess. URLs are getting typed into search boxes (one of the 3 or 4 address bar like objects on the screen), search terms are going into the address bar, and popups of the local rather than web variety are rife.

So, webbrowsing becomes hard, and worse, scary. Just about every action seems to trigger a vastly over-dire warning box to pop up along the lines of “You will contract a virus in the next 3 minutes unless you update Power Ultra Virus Scanner Pro from XXXXX” (where XXXXX is the vendor of the laptop/crapware/super market where they bought it. These guys are terrified at every step, because even actions such as posting a form are causing the triggering of these pressure sales pop ups.

I just bought a new laptop to replace my trusted workhorse. It’s an Acer 5742Z and I love it. or rather, I love it now it’s not running Windows. I booted up the OEM install of Windows 7 to see what it was like when I bought it. I lasted 2 hours.

Firstly even finishing the OEM install process was hard. It required an internet connection as far as I could tell. Imagine trying to get that sorted if you’ve just got broadband for the first time, a nightmare.

Once I’d got it fired up IE (to install Chrome) and I couldn’t even enter the address. Once I hit enter, Bing Toolbar popped up trying to help me. It forced me to choose between Bing handing search and not, and then hung. Eventually I used task manager to kill it, and tried again with more success, but after 2 hours I gave up and booted up with a Ubuntu CD in the drive.

It’s no wonder people are scared of IT and convinced they’re going to break things, after 3 years away I’d forgotten how unpleasant an experience it was.

I’ve just finished a short project with a Newcastle Upon Tyne based estate agent, helping to smooth over some of their accounting and records processes. Previously they had to manually reconcile their bank statements against each room in every property they had, and then reorganise this information by tenant to work out who’s not paid their rent for this month.

This now all happens in a single spreadsheet (or rather a single workbook with multiple spreadsheets), which are set up annually, and amended daily. Bank statements are imported from LLoydsLink, each payment is allocated to a tenant though a series of drop down menus (pick a property and it refines the options for tenants to just those living in that property). All the following calculations are derived from this data, mostly using the extremely powerful Pivot Table (and a few extra functions).

Pivot Tables make it easy to take a stack of tabular data and to group and filter specific columns based on the values of other columns. It’s uses are endless from summarising sales data to digging out nuggets of insight which are buried in the morass of information.

For the total cost of about three days of my time we’ve saved approximately 10 hours a week of admin time, as well as hugely reducing the chances of manual error.

What comes out of the room when you give geeks data and some time to play with it (also with food and coffee) is frankly amazing.

I was lucky enough to be awarded a travel bursar for Culture Hack Scotland last week, and so headed up to Edinburgh on Friday afternoon to bash out something (at that point unknown) using the amazing data sets that had been provided.

After a bunch of welcome talks and a few glasses of white wine, we got down to bashing ideas around and coding. I really wanted to work with the footfall data provided by City of Edinburgh Council. It has the number of people to pass 19 spots around the city for the last 365 days, in the form of Excel.

Lots of the data provided was historical (2010′s Festival data for example) because the 2011 data is not ready yet, however it’s all in the same format, so hopefully anything built on top of the 2010 data could be adapted to run in 2011.

When I started working with the footfall data it became apparent that whilst it was clean and neat (unlike a lot of publically available data) it was not very useful in a spreadsheet. What would have been lovely was a queryable API, where you can ask for the data for a given day, or venue, or day and venue, and get back a set of numbers. This was what I decided to built. It wasn’t going to be pretty or shiny like some of the works of art, but it would turn out to be very useful.

So, by Sunday morning, it was live. http://edinburghfootfall.theapproachablegeek.co.uk/ provides a set of end points which return nice clean JSON data. JSON seems to be the data type of choice for most web apps and API mashups these days, and it’s lovely to code with. Over the next few weeks I’ll update the API to add in some more filtering and hopefully keep it live for as long as it’s useful.

There were lots of amazing hacks produced over the course of the weekend, and I was blown away not just by the quantity but also the quality. 24 hours really is a long time at times.

Edit (6/6/2010) Some of the hack (including one based on the data I made available) made it into the Guardian Data Blog!