Making use of open data

devonspends

We were very pleased, a couple of weeks ago, to discover that some of our open data has been put to good use – Jim Patterson, a developer who’s just moved to Devon, put together the Devon Spends website as a project to summarise and display the ‘Spend over £500’ data that we release every month.

We caught up with Jim this week and asked him a few questions; here’s what we talked about.

Firstly, the big question; what brought you to Devon?

The beaches!  There’s a distinct lack of them in the East Midlands.  In truth, much of my extended family have moved down over the past couple of years; I thought I’d follow their lead.  But mainly the beaches.

Well, we can’t argue with that! How long have you been playing with open data like this?

I’ve always been a bit of a data geek – I have a background in marketing and BI.  The open data movement has grown alongside my geekiness – it’s something that’s held my interest for several years now, especially once I’d discovered data.gov.uk.  I think it’s a sadly overlooked area of transparency by the government, and has gotten to a point now where the breadth of data makes it possible to provide truly informative applications.

Did you specifically pick on finance data to work with?

It wasn’t a pro-active choice.  I was browsing for Devon-focussed data that I could potentially use to create an angularJS application, a skill I was looking to learn, when I came across the expenditure data on your website.

It struck me that this data was overly opaque – any month by month comparison, or aggregation required the end-user to download and compile individual csv files.  For the average user, this would be overly worksome; in effect the data wasn’t providing the visibility for which it had been created.

Once I’d identified this issue, I figured that improving the data presentation would be a useful learning project, could provide visibility of my business, and may even be something Devon people would be interested in seeing; their councils spending in an accessible format.

How did you get on with finding the data, downloading it and working with it?

Using data.gov.uk, finding the data was a breeze.  Downloading and working with the data was more problematic, primarily due to volume.  However, once I’d got the data into a database, it was much easier going.  In an ideal world, this data would be available via a json web-service from data.gov.uk.  In the end, I created this service myself, and will continue to manage and populate it moving forward.  I’m happy to release details of this endpoint if anyone wishes to reuse the data.

That’s a very helpful offer, thanks Jim. If you could send a message to the people working on open data for the council, what would it be?

Re-use the data within your own internal applications.  The data quality in the expenditure files is not great – company names include things like internal use codes and notes to the finance department regarding payment.  In addition, the company names themselves appear to be added informally; there’s nothing like a company registration number to key the data.  Any attempt to combine this data with other open data sources is a real pain.  I’ve added some companies house data to devonspends.co.uk, (primarily addresses and links to companieshouse.gov.uk profiles), but it’s a very hit-and-miss affair.  Even with a degree of ‘fuzzy’ matching, I was only able to link back to around a third of the companies.

Some work for us to do there, then. Are you planning to do anything with other Devon datasets in the future?

Certainly – it’s just a case of finding something that either would interest the ‘masses’, or provide some degree of insight for my company, PatternSoft Ltd.  The devonspends site is interesting, but not overly complex – I’d be very much interested in combining several sources of open data into some form of ‘bigdata’ project that may provide insights not yet realised.

We’ll look forward to seeing what you come up with, and helping if we can. What would you say to someone who’s thinking of doing a similar project?

Just do it!  There are many reasons for getting involved with the open data initiative:

  • providing large datasets for some learning project (e.g. ideal for school projects/company training)
  • creating applications of genuine use to ‘computer-illiterate’ users
  • mining for exploitable insights for commercial use
  • identifying potential areas of partnership with government

Just take a moment to browse through data.go.uk – there’s bound to be something that takes your fancy!

We strongly agree!

Thanks to Jim for taking the time to talk to us, and we’d love to hear about any similar projects out there that have found a use for the datasets we publish. If you’re looking for open data we have a dedicated page at https://new.devon.gov.uk/factsandfigures/open-data and you can also find our data uploaded on data.gov.uk. If you can’t find what you’re looking for, leave a comment here or send an email to lucy.knight@devon.gov.uk. 

Be the first to Comment

Your email address will not be published. Required fields are marked *

 

Comments are held for moderation. House rules