Tagged: api

Extracting Your Data from the AudioBoo API

Earlier this week, I wrote about the Future of AudioBoo. I'm sure the service is going to be just fine - but thought it would be an interesting exercise to liberate my data from there just in case.

As I begin the move to decentralised services where possible, I think it's important that I take responsibility for my own data.

The API docs for AudioBoo are very clear, so here's a quick guide on how to download all you Boos and (most of) their data.

Get All Your Boos

The AudioBoo API - unlike some - doesn't require any authentication. It also doesn't restrict you to only downloading your own data. So, if you want to swim in Stephen Fry's vocal delights, go right ahead!

The API call is very simple:

http://api.audioboo.fm/audio_clips?
   username=edent&
   page[items]=150&
   page[number]=1

You can grab up to 150 items at a time. To move the the next page of items, change the "page[number]=" to 2 - and so on.

Examining The Data

Each page of data starts with something similar to:

  1. {"window":60,
  2. "version":200,
  3. "timestamp":1365710614,
  4. "body":{
  5.  "totals":{
  6.   "count":64,
  7.   "offset":0
  8.  },
  9.  "audio_clips":[....]

This tells us how many Boos there are (in my case, 64) and - if we're no longer on page 1 - which number Boo to start with.

The Boo's Data

Boos are presented in reverse chronological order - the most recent first.

Here is what an individual Boo's JSON looks like - it's fairly simple and self explanatory.

  1. {
  2. "id":363171,
  3. "title":"QR Codes at OpenTech",
  4. "user":{
  5.  "id":2861,
  6.  "username":"edent",
  7.  "counts":{
  8.   "audio_clips":64,
  9.   "followers":20,
  10.   "followings":8
  11.  },
  12.  "urls":{
  13.   "profile":"http://audioboo.fm/edent",
  14.   "image":"http://www.gravatar.com/avatar/a4c65091cd258c86e6187eaaed4ef939?default=http%3A%2F%2Fd15mj6e6qmt1na.cloudfront.net%2Fassets%2Favatar_green-a51f577eaf0b191acb694b5ec3c733dc.gif"
  15.  }
  16. },
  17. "duration":1173.53,
  18. "mp3_filesize":9388160,
  19. "uploaded_at":"2011-05-21T21:23:09Z",
  20. "recorded_at":"2011-05-21T21:23:09Z",
  21. "location":{
  22.  "description":"City of London, Camden Town, United Kingdom",
  23.  "longitude":-0.131275,
  24.  "latitude":51.5226,
  25.  "accuracy":29.6755
  26. },
  27. "counts":{
  28.  "comments":1,
  29.  "plays":133
  30. },
  31. "urls":{
  32.  "detail":"http://audioboo.fm/boos/363171-qr-codes-at-opentech",
  33.  "high_mp3":"http://audioboo.fm/boos/363171-qr-codes-at-opentech.mp3",
  34.  "image":"http://audioboo.fm/files/images/0120/4984/Twitter_Search.png"
  35. },
  36. "tags":[
  37.  {
  38.   "display_tag":"edent",
  39.   "normalised_tag":"edent",
  40.   "url":"http://audioboo.fm/tag/edent"
  41.  },
  42.  {
  43.   "display_tag":"open data",
  44.   "normalised_tag":"opendata",
  45.   "url":"http://audioboo.fm/tag/opendata"
  46.  }
  47.  ]
  48. }

The two fields which we're probably most interested in are "high_mp3" which gives you the URL to a high quality MP3 recording, and "image" which contains the picture you associated with the Boo. As you can see - the data is fairly comprehensive. Perhaps the only thing it is missing are the comments. You get told the number of comments left, but not their content.

Parsing The Data

Once you've called the API and got a response, it's trivial to extract your data for use.

  1. < ?php
  2. $json = json_decode($string,true);
  3. $audio = $json["body"]["audio_clips"];
  4.  
  5. foreach ($audio as $key)
  6. {
  7.  $mp3 = $key["urls"]["high_mp3"];
  8.  $image = $key["urls"]["image"];
  9.  // Do Something...
  10. }

Personally, I wrote the URLs to a file, then used wget to download all the mp3s and images. You could also write the data to CSV or something similar and then import it into WordPress (or your blogging platform of choice).

Limitations

Perhaps the only minor downside is that the files are MP3s. I thought that AudioBoo stored them as FLAC. That said, MP3s are more than adequate for voice recordings. They also have fully comprehensive ID3 tags.
The lack of comments is a little sad - but my Boos never quite garnered enough interest to attract much attention.

There are no download limiations that I ran into. I quite happily scoffed down 5GB of data as fast as they could send it to me.

So, there you have it! A simple way to download all your AudioBoo data. Here's a quick PHP file which should download your first 150 Boos - good luck!

  1. < ?php
  2. $AudioBooAPIresponse = file_get_contents("http://api.audioboo.fm/audio_clips?username=YOUR-USERNAME&page[items]=150&page[number]=1");
  3. $json = json_decode($AudioBooAPIresponse,true);
  4. $AudioClips = $json["body"]["audio_clips"];
  5.  
  6. foreach ($AudioClips as $key)
  7. {
  8.  $id           = $key["id"];
  9.  $title        = $key["title"];
  10.  $description  = $key["description"];
  11.  $recorded_at  = $key["recorded_at"];
  12.  $location     = $key["location"]["description"];
  13.  $longitude    = $key["location"]["longitude"];
  14.  $latitude     = $key["location"]["latitude"];
  15.  $accuracy     = $key["location"]["accuracy"];
  16.  $original     = $key["urls"]["detail"];
  17.  $mp3          = $key["urls"]["high_mp3"];
  18.  $image        = $key["urls"]["image"];
  19.  $duration     = $key["duration"];
  20.  $mp3_filesize = $key["mp3_filesize"];
  21.  
  22.  // Save The Files
  23.  $mp3_file = file_get_contents($mp3);
  24.  file_put_contents($id . ".mp3", $mp3_file);
  25.  
  26.  $image_file = file_get_contents($image);
  27.  file_put_contents($id . ".jpg", $image_file);
  28. }

What Twitter Can Learn From App.net's Developer Incentive Program

Twitter makes a lot of money out of me. At least, I assume so. The code I helped write, and the sites I run, are used by millions of Tiwtter's users. I've sent a tonne of traffic their way, and what has Twitter given me?

Not even a "thank you."

Seriously, no one from Twitter has ever said "Thanks for all the customers. Thanks for helping develop our presence in certain markets. Thanks writing tools that keep our users playing on our service. Thanks!"

Compare and contrast to App.net. The owners of that service have just written me a cheque for $74.
app.net earnings
I know, I know! I'm not making Zuckerberg money off that. If I'm lucky enough to get that every month, I'll have enough to buy myself a really fancy bottle of wine. But, hey, it's a start.

App.net have hit on a cunning plan to keep developers engaged - as part of their Developer Incentive Program every month they take $20,000 and divide it between developers based on user ratings.

So, my two apps - Dabr for Android and Dabr for Mobile Web - have netted me 0.3% of the developer pot. Sweet!

More importantly, they said thank you!

Of course, App.net has a small catch. There's a fee for joining.
App.net Fees

The fees aren't onerous - and are lower than they were - but I accept that the current pricing excludes many people.

As a developer, I like the fact that the App.net API is consistent and works really well - unlike the unloved hodge-podge that is the Twitter API. The issues list is taken seriously, and they seem to act on the feedback they actively solicit.

As a user, I like the community. I like the lack of advertising. I like the features (muting, streaming, ability to see who has starred a message, properly threaded conversations, etc) which are all conspicuous in their absence from Twitter.

The only thing it's missing is you. So join App.net today!

I've written before about how Twitter has abandoned those who helped make it a success. Ewan Spence has written brilliantly on how developers are now being shut out of the Twitter.

Just imagine if Twitter paid third party developers. Twitter are shitting themselves that non-official apps will steal revenue from them. Well, duh, the developers have to put food on the table. What if, instead of trying to shut down the people bringing value to the network, they reflected how much they were worth by paying them.

Or even, you know, tried saying "thanks" once in a while.

API Design is UI for Developers

I've been thinking a lot about APIs and their design recently.

I stumbled on this fantastic quote from Greg Parker:

A programming language is a user interface for developers. Language authors should learn from HCI principles.
@gparker
Greg Parker

When I first started learning C++ (back in the bad old days) I was convinced that any 1st year student could design a better programming language. One which behaved in a sane fashion without a lot of legacy cruft. In many ways, PHP is that programming language. It's simple, logical, and works without having to know lots of esoteric computer science.

I see API design in the same way. Most of the APIs I come in contact with were obviously designed by the original developer for the original developer - with no thought of those who would come after her. Often, they're designed for one specific internal usage, reluctantly opened to the public, and never updated to account for how people actually use them. Oh, and you can forget about decent documentation!

One hackathon I went to a few weeks ago had a Developer Relations employee stand up and say:

Who wants to use our API? It uses SOAP - sorry. If you want documentation - come see me because it's not on the website. Oh, and it's read only. Let me know as soon as possible because it takes 6 hours to approve your API key.

This is madness. Developers are human too! They need some HCI love between them and their APIs.

So, here are my hastily scribbled thoughts on what an API needs at a minimum to entice the busy developer.

I don't think any of these are Earth-shattering, but it's amazing how many APIs fail to meet even these basic requirements.

Easy Access

  • Don't make me register, set up an account, or fill in a load of forms - I just want to see what I can do with you before I make a commitment.
  • Do give me an Apigee console - or similar - so I can see for myself what's happening.
  • Don't give me absurd key signing requirement - I should be able to call the API from a web browser just by typing stuff in.
  • Do make it as easy as possible for me to get started - get your CEO (or similar) and see how long it takes her to set up an account & launch her first call. If it's more than 5 minute, go back to the drawing board.

Example Code

  • Don't tell me what your API can do - show me a few cool calls to start me off.
  • Just because you love Ruby, doesn't mean everyone does. Show examples in a variety of languages.
  • Provide examples of code which won't work and explain why.
  • Comment your code. It may be obvious to you what "choes=17|L" means, but it may not be to me.

Documentation

  • This is possibly the most important area.
  • I need to know what I can do, how I can do it, why I should do it a certain way, and what response to expect.
  • For. Every. Single. Function.
  • If you can't document your API, people can't use it.

Full Enumeration of Responses

  • Tell me the full range of response I can expect.
  • A good example is Twitter's "retweet_count", the documentation used to imply that the response would always be an integer. However, it would occasionally respond with a string of "100+". Naturally, this meant developers would write code expecting ints which would fail whenever a string was encountered. If you can't tell me what responses to expect - how can I code something which handles those responses correctly?
  • What error codes are you likely to throw?

Variety of Response Language

  • Yes, you love XML. Guess what? I don't!
  • The customer is always right. If the customer (developer) wants JSON, XML, PHPobject, or just plain text - you should give it to them.
  • It's the API designer's job to make life easy for developers - so reply in whatever formats the developer wants.

Human Readable Response

  • Developers are humans! Yes! It's true! And they can't all pretty-print JSON in their heads. Give them something they can read without resorting to external tools.
  • The Wikipedia API is a brilliant example of this. They have a human readable response for their API calls.
  • Unless you have a very good reason not to - all your responses should be pretty-printed. It helps with debugging and makes life just that little bit easier for a struggling developer.

Human Readable Requests

  • Developers are humans! Yes! It's true! And they can't all remember every little acronym in their heads. Give them something they can read without resorting to documentation.
  • What's easier to remember "gnxID" or "getNextId"?
  • Keep your parameters consistent.

Unchanging

  • Consistency is a virtue. If you have two similar APIs (say, search & read) they should take the same parameters and produce identically formatted responses.
  • If you have to change the way your API responds, or the way it accepts request, that's fine - but use versioning so that developers don't have to update their code if they don't want to.
  • Never deprecate anything! Once an embedded device has had its firmware burned, it's unlikely to ever be updated. If people or services rely on you, it's simply unacceptable to kill something off. Remember, not every developer or end user can update their software.

Simplicity

  • Perhaps the most important of all HCI commandments - Keep It Simple.
  • Don't engage in a needless dance where developers have to take several steps to do a single action.
  • Try to explain what you're doing in a single sentence. If you can't - it's probably too complicated.

Libraries

  • I'm sure you're very proud that your community has created some libraries - but they're not good enough.
  • Want people to use your API? Provide libraries in a wide variety of languages.
  • Update and support those libraries.

Detailed Errors

  • It's not enough to say an error has occurred. Say why it has occurred.
  • Error codes must always be accompanied by human readable error messages.
  • Do you really need to throw an error? Can your API take a "best" guess at what the user was trying to do? Be generous in what you accept.

Feedback

  • Your API sucks. Accept that.
  • Provide a mechanism where people can feed back what they think is broken, poorly implemented, missing, or just plain confusing.
  • Discuss the feedback openly. See what the rest of your community thinks.
  • Act on feedback. If a feature request hasn't been acted on after 6 months, you've probably failed.

So What?

There is nothing new in this post to the seasoned developer. But as Matt Gemmell reminded me recently - some people just don't know the basics.

If you're interested in making your API useful for developers, you have to treat it like any other product. You have to consider HCI factors, you have to do product testing, design, and planning.

Your API is a product. Treat your developers as you would your most profitable users.

Further Reading

There are many books on this subject - the two I recommend are:
Basics of the Unix Philosophy (free).
The Design of Everyday Things (paper or ebook).

When is a URL not a URL?

Summary

Twitter's way of linking URLs is broken. It's annoying to users, and a pain in the arse to developers. This quick post talks about the problem and offers a solution.

I've raised a bug with Twitter and I hope you'll star it as important to you.
Continue reading

Displaying Twitter Photos via Entities

Twitter has announced that it will soon open up a native photo sharing service.

Rather than using an external service like Embed.ly to retrieve thumbnails, all the data is embedded within Twitter Entities.

So, if you request a status using "include_entities=true", you will be able to grab the image and display the thumbnail using the following code.

  1. function twitter_get_media($status) {
  2.    if($status->entities->media) {
  3.  
  4.       $url = $status->entities->media[0]->media_url_https;
  5.  
  6.       $width = $status->entities->media[0]->sizes->thumb->w;
  7.       $height = $status->entities->media[0]->sizes->thumb->h;
  8.  
  9.       $media_html = "<a href=\"" . $url . "\" target='_blank'>";
  10.       $media_html .=  "<img src=\"" . $url . ":thumb\" width=\"" . $width .
  11.          "\" height=\"" . $height . "\" />";
  12.       $media_html .= "</a><br />";
  13.      
  14.       return $media_html;
  15.    }        
  16. }

So, a tweet like this:


Will render like this (in Dabr):
Twitter Dabr Images

Notes

This is very rough and ready proof of concept code. Beware of the following:

  • This will only take the first image from the tweet.
  • Only images are supported - I'm not sure how their proposed video sharing will work.
  • There's no error checking.
  • In the above code, the https URL is used - if you want a non-SSL link, you'll need to remove the "_https"

Enjoy!