Freesteel Blog » Replace your tired API with a shiny codewiki

Replace your tired API with a shiny codewiki

Thursday, March 19th, 2009 at 11:52 am Written by:

There is a theory that webpages that serve lots of data (like Farmsubsidy.org or the Guardian’s new and totally befuddling Open platform thingy) ought to provide an effective API to access the data (in addition to their own surf-able front-end) for power-users to design different front-ends of their own, and “mash up” the data in ways that the designers of the original website were not clever enough to think of.

An API is basically just a webpage containing the data without any pretty decoration in a form that standard programming functions can easily parse into an internal representation. A popular one is JavaScript Object Notation (JSON).

To facilitate this theory, it is proposed that your standard data serving website ought to be separated into two independent pieces — the first piece is just the API server on the database, and the second piece just queries this API and decorates the data into nice normal looking webpages. Both components can be on the same server, but having them separated out means that the API is proven to be sufficiently well powerful enough to generate the standard webpages.

(Normally if you just threw in some random API functions onto your website without yourself intending to use them for real, there’s no reason to expect it’ll be enough to do anything with.)

The MediaWiki API (for accessing data out of Wikipedia) is not powerful enough to recreate the site. You can obtain just about anything from it (for example the Caves of Yorkshire category), but not the actual text from any page.

This infuriated me for hours as I kept looking everywhere for this obvious function, and they don’t say that it’s deliberately not there. Wikipedia also blocks you from webcrawling their pages because it would “overload their servers”. Instead, you have to do some kind of deeply irritating Wikipedia:Database_download, which I was trying to do one time, before I got too annoyed with the process. It was all part of my effort to make the PublicWhip database of MPs be a direct reflection of a fast and special implementation of DBpedia. It needs to get updated on demand, so that when an MP resigns and the fact gets noted on his Wikipedia page, it appears in the database. Maybe one day. Or someone else can do it. I keep spreading this idea around in the hope that it happens. I mean, if so many people were writing webcrawlers of Wikipedia that they got annoyed enough to block them, why can’t I find one person among their bumbers interested enough to do something this blindingly useful.

Speaking of which, TheyWorkForYou has an API as well, serving 19 functions in 6 different formats.

I’m not sure many people have actually used it productively.

There is no API for PublicWhip— although the webpages themselves are so big and tabular they are practically like API output in raw form. People complain about the PublicWhip interface because it’s so complicated and intimidating. The next time I get lectured about how it would magically all make sense and be accessible to everyone if only it was designed by someone a million times better than me, I’m replacing all the pages with pictures of a kittens. Or maybe we ought to make clear the objective of a website when it’s being criticized. Everyone can understand and appreciate fluffy kittens, can’t they? But then they won’t learn anything about Parliamentary votes, will they? It will never be pretty. Admittedly the website could be a whole lot better, but it will never be as good as pictures of kittens. And I would expect a web designer who is trying to make it better to know as much as he or she can about Parliamentary votes in order that they don’t unintentionally delete all that data and replace it with kittens. Part of the idea of making APIs is it’s a cop-out: “if you can present the data of this website better, here are some tools for it.” You are free to decorate it with kittens.

The policy with PublicWhip is that when anyone asks about an API, they get given login and passwords onto the (still sick) seagrass server (on which it is hosted) so they can write exactly what they want.

Hasn’t worked yet, though. But it does suggest something.

A web API is an Interface.

As with all interfaces, it’s not possible to make it generally suitable for everything. In general in most intensive programming, you design the interface while you are making the application. Like, “I need another function to do this thing”, so I have to go back and put it into the interface. It’ll never be complete; it only has to be powerful enough to get the job done that you’re doing at the time.

Therefore the API will be limited by the jobs you thought it was going to do when it was being designed and implemented.

Hence, if you made something powerful enough to serve data into a google mashup, then it’s probably going to be limited to just that, and you shouldn’t be disappointed when the best that any of your users can build with it is another google mashup.

We yawn at it. But its predetermined, like those scrapheap challenge episodes where happen to find two bits of scrap that somehow have all the right bits to bolt together.

People are not going to make something like a barchart of the Parliamentary data aggregated statistically, or a scatter-plot graph of MPs’ travel expenses against (a) distance from London or (b) cost derived from the isochrome travel map (That would need an iso-cost travel map. Who’s going to make one?). The data is simply not there available through the TheyWorkForYou/API. The API is limited, and so all the results are limited.

Well, I suppose you could work around it by polling the API repeatedly in order to recreate the data in your own database and then query it. But do you know how much quicker+easier (= much more likely someone will do it) it would be to simply access the database from within the webserver?

The proposal is that tools should be made available to Roll Your Own API.

Specifically, if the website is built within a framework, such as Django, then the basic components are the views (small functions that assemble lists of data from the database) and templates (concise systems for transforming lists and items of data into a nice looking webpage).

What you need is to put one views function and one template file into a wiki system, so that people can use all the power of the database queries and function writing, and all the power of the template to write it out in whatever format they want (JSON, XML, CSV, RDF, whatever, a new one every year) for their downstream application they are building.

That way, we won’t wind up writing a whole bunch of API functions that can only run google maps (if that) with a proliferation of output formats that tick the boxes, yet don’t actually seem to lead to any wider use.

Clearly, for fastest work, the same programmer has got have developmental access to both sides of the interface– not as it is now where some guy who doesn’t know the eventual uses the data writes some boring API functions in one year, and three years later someone else tries to use it in spite of all the gaps.

My proposal could, of course, be done as an API where the URL takes two parameters:

http://www.webpageurl.com/djangoapi?views=”someverylongstringofpythoncode”&template=”anotherverylongstringoftemplatecode”

but that would be cheating.

In the future, all such innovative websites that are interested in feeding data into other modules are going to use one of these many Javascript-based source code editors which will allow you to deposit fragments of code on the server such that the right stuff is served up on a webpage.

The best codewiki I’ve found so far is this demo of the orc programming language. Lovely work.

With this technology, these fruitless APIs are going to be dead in the water (they were never alive in the first place).

Just another fine theory that didn’t really work because no one thought it out for long enough.

Of course, I’m almost certainly entirely wrong as usual. For proof, go see all the activity going on at the Apps for America competition.

1 Comment

  • 1. Topics about Travel &raqu&hellip replies at 19th March 2009, 2:20 pm :

    […] All Info About placed an observative post today on Replace your tired API with a shiny codewikiHere’s a quick excerpt…from the isochrome travel map (That would need an iso-cost travel map. … In the future, all such innovative websites that are interested in […]

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <blockquote cite=""> <code> <em> <strong>