After getting initially very excited about YQL and streams from the weekend (each < 50k), it turns out that midweek streams at 3-4meg are too big for YQL and I get exceed-limit warnings.

I still have some ideas of for things you could go on and use YQL for though.

Reminder : if you are planning on fetching your local vacancies daily, you might be best not to do it on Sunday or Monday mornings as there are very few jobs in there.

As I only made a soft-release of the url on last Friday evening, and nobody seems to be using the daily service yet I thought I'd make a last minute change after I read 10 common mistakes made by API providers (especially #4, #comments: I agree #10 should be the top one too, but this is not a REST service, but at least I know why)

The node (or endpoint?) for the JobsGoLocal API is now using the api. subdomain so:

http://api.jobsgolocal.org.uk

The samples and examples have been updated to show this change.

I will leave the old node in place for a week or so to see if anyone had bookmarked it.

www.jobsgolocal.org.uk remains the public site, with introductory documentation, map of offices etc and the blog at jobsgolocal.posterous.com contains API documentation, other supporting documentation and announcements such as this.

If you are planning using the API, then I'd encourage you to subscribe to this blogs' RSS feed so I keep you up to date with changes such as:

• status changes
• upstream problem, or known issues
• additions to the API

The JobsGoLocal API is very simple.

You tell it which offices you would like to see the latest vacancies for, and how you would like the results formatted.

THE NODE

http://api.JobsGoLocal.org.uk/

equivalent to:

http://api.JobsGoLocal.org.uk/index.htm

THE CALL

It accepts a number of GET arguments, only the first one, offices is mandatory.

offices=

Up to 10 pipe ( | ) delineated three letter "Jobcentre Plus" office codes. The codes can be collected up from the map.

format=

[optional] One of either json [default], php, xml or yaml.

callback=

[optional] The Javascript function to be called when it arrives back at the client. When set the payload will be the same as json, but contained within a callback so it will become jsonp.

url=

[optional] Let me know which site you using url=yoursite.com or use url=testing if you are playing around and will be hitting the site more than once per day and I will cut you some slack.

See documentation for each format option to see example calls, code samples, error codes and headers information.

json, jsonp, xml, php, yaml.

Sample GET call for 2 offices, Guildford [GUF] and Woking [WOA] - returns json by default.

http://api.JobsGoLocal.org.uk/?offices=GUF|WOA

Sample GET call for 1 office, Guildford [GUF] - returning xml;

http://api.JobsGoLocal.org.uk/?offices=GUF&format=xml

FREQUENCY

The data is only updated once a day, at midnight (GMT) from the zois ftp service, so make your call to the API in wee the small hours. Leave it till after 1am and then you don't need worry about daylight saving time changes on any of the servers involved.

You are best leaving out calls on Sunday and Monday morning as there are very few jobs added on the preceding days.

CACHING

As the data is only updated once a day you are strongly advised, like really, I insist, to cache the results on your own servers and do any post-production assembly, filtering or formatting there.  I will give you a bit of leeway about testing, just append &url=testing to your call.

HOW TO PROGRAMME A CALL TO THE API

This is going to depend very much on your programming language and/or the o/s of your server.

For those with *nix servers you can just add the single line to your crontab file:

15 2 * * 1-5 WGET -q -O /var/cache/ourjobvacancies.json http://api.JobsGoLocal.org.uk/?offices=WOA|GUF

The url shown is the shorthand equivalent of:

http://api.JobsGoLocal.co.uk/index.htm?offices=WOA|GUF&format=json

json is served by default and index.htm is the default doc.

The WGET example tells the server : On Mondays to Fridays [1-5] at 02:15 go and get the output from that url and put it in a folder called /var/cache and save the file as ourjobvacancies.json

If anyone knows the Win32 equivalent AT and DOS commands to go out then please post it below.

CURL

Called similarly from a crontab entry:

15 2 * * 1-5 curl -s -o /var/cache/ourjobvacancies.json http://api.JobsGoLocal.org.uk/?offices=WOA|GUF

*TODO add an example CURL call from a script

GET STARTED

Go to the map page on the main site and pick the Jobcentre Plus Offices in your locality to assemble your API call.

If you have any questions or comments then post them below, thanks!

The data is extracted from Martin Sullivan of Zois Limited's scrape of all the latest vacancies at the jobseekers.direct.gov.uk website.

Read more about his motivations for making this available from his page The Unofficial National Jobcentre Plus Mirror.

You are free to get the entire csv file of all job vacancies in the UK from his ftp site.

Martin announced the availability of this data on the Apps section of the data.gov.uk website on 12th July 2010.

So, let me spell it out again for you, this raw data stream is totally unofficial - contains no warranties for fitness of purpose whatsoever and you should bear in mind that it can be withdrawn by the data owners at any time.

The data is a csv file containing 14 columns, they are:

title,reference,location,hours,wage,work_pattern,employer,employer_ref,

pension,duration,closing_date,description,apply,added,office_code

For a fuller explanation of the contents of the original csv file then grab Martin's readme.txt.

Even a quick rummage through the raw data will expose that you will likely want to do some post-processing in order to make it palatable for the public.

• some fields contain a mixture of upper and lower case text, this seems to depend upon the whims of local operators
• some fields are left empty
• wage for example can contain hourly, weekly, monthy and annual rates as well as comments such as "Meets the national minimum wage"
• dates are database-friendly dates YYYY-MM-DD which will likely cause public confusion
• office_code is in the format GUF for Guildford, you will need to dereference this in order that the public know which office is advertising the vacancy
• JCP offices is another csv file you can download from Martins ftp site although you can look them up in a Google Fusion Table or pick them from a map.

It is possible to do some simple post-processing and here is a page that shows what can be achieved. This example also demonstrates how the JCP offices are dereferenced and how they can mashed up with Google static maps along with thier contact details.

www.godalming-tc.gov.uk/localjobs.htm

If you want me to prepare a similar page-stream which you can just slot into your website daily, then get in touch on twitter @paulgeraghty or email me directly using the name foofoonet and my provider is gmail.com (so you should be able to put the @ in the right place, yeah?)

Cache-Control: no-cache, must-revalidate
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Content-Type:text/xml; charset=utf=8

With JobsGoLocal PHP Webservice Option php sent the return value is a string containing a PHP serialized array of jobs from the offices you stipulated. This is probably only any use to you if you have access to PHP on one of your servers.

Example call:

/?offices=GUF|WOA&format=php

You should only call this service once a day and cache the file locally.

Format:

The return value is a serialized PHP array as a string.

a:7:{i:0;a:15:{s:5:"title";s:27:"ENERGY SALES REPRESENTATIVE";s:9:"reference";s:9:"ELY/26247"; etc

You will need to run the PHP unserialize() function over the string to extract the array of jobs.

The jobs will contain associated arrays of elements which can then be looped through.

Sample code:

<?php

$jobs = file_get_contents( 'yourcache.txt' );
$jobs_array = unserialize( $jobs ) ;
if( array_key_exists( 'zero' , $jobs_array ) ) {
  echo $jobs_array['zero'] ;
}else{
  foreach( $jobs_array  as $job )
  echo $job['title'] . ' at ' . $job['office_code'] .'</p>';
}

?>

Errors:

If you make a fundamental mistake in the call, say when testing, you will get a simple string telling you what was wrong.

"You failed to submit any jobcentreplus office codes"

If there are no jobs at all matching the call you submitted either because the office code(s) were not found in the parent .csv file or because there were simply no jobs (which can happen on Sunday and Monday mornings) you will get an array whose first element is named "zero", and you should check for its existence in order to be able to at least degrade to a reasonable message, and carry on.

Example of the serialized PHP array as a string :

a:3:{s:4:"zero";s:41:"Sorry, no job vacancies were found today.";s:7:"offices";s:72:"No jobs were found matching the jobcentreplus offices you specified: ABC";s:4:"date";s:11:"16 Aug 2010";}

Aside from messages about incorrect service calls, the error message associated with this format will be an array with one element containing the serialized PHP array as a string:

a:3{s:4:"fail"};

Headers:

The headers accompanying the response for this format will be:

Cache-Control: no-cache, must-revalidate;
Expires: Mon, 26 Jul 1997 05:00:00 GMT;
Content-Type:text/plain; charset=utf-8;

[{ title: 'ENERGY SALES REPRESENTATIVE', reference: ELY/26247, location: 'Ely and surrounding areas CB7', hours: '37 HOURS OVER 5 DAYS' etc

Cache-Control: no-cache, must-revalidate
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Content-Type:text/yaml; charset=utf=8