beard, avalanche, coy

You don't really want it any more from me

People use Excel a lot. Abuse is possibly a better way of putting it. In fact they abuse the whole of the office suite - you've all received the three word reply wrapped up in a Word document or the PowerPoint document that just has a bunch of images in it.

The difference is that some of the Excel abuses are actually pretty clever. There's those quizzes that get emailed around - guess the lyric, solve the puzzle. But even beyond that you get the Finance and Ad booking departments that run off a single shared Excel spreadsheet of such breathtaking utility that you kind of have to step back and gasp.

It's so horrible! Yet so beautiful! It does everything for them - it tracks bookings and various sales stats and shows past performance and projects future trends. Sure, the programmer in you is recoiling in absolute horror - this isn't programming, this is a crime against compilers. Forget spaghetti code or lasagne code. This is macaroni baked with Velveeta. Yet like Mac'n'Cheez[tm] it works. It's delicious. It's a remarkably sophisticated app that does exactly what they want and does it fast.

And they wrote it. The sales people wrote it. These people who seem to look down on programmers and who won't update the HTML for the copy on the web page because "I was never very good at maths at school".

I'm being flippant of course. And despite some occasional minor differences, I've always got on with the biz dev teams in the companies I've worked for which is why it's sometimes baffled me that they won't even touch, say, HTML.

The problem, I think, is familiarity. We know that the amount of damage they could do with HTML is minimal. Except, well, if they leave out a single character then maybe the whole page is blank. It's not a big deal to us - we know it's not hard and that anything is easy to fix but that's beside the point. We're familiar with HTML, with the Web stack. They're not. In much the same way we're not familiar with (plucks bogus example completely out of thin air) filling in form 64-8 for authorisation of a VAT agent.

Yet they're familiar with Excel. And not just as a spreadsheet - as an environment. They know what makes it tick, what its capabilities and quirks are and how to make it dance in a way that you and I may never.

I'm kind of reminded of this quote about welders

"One of the unexpected things about watching the steel guys work is how the solidity of metal means nothing to them. Most people think of metal as something hard and inflexible, but welders don't. Which should be obvious in hindsight, I guess. But, for example, they have these saw-horses that are made of tube steel. And I can see how that came about: they needed some saw-horses; they had some steel. It took them 30 seconds to make them. And, an example with the stairs: the legs of the stairs' landing platform have big threaded bolts for feet, to fine-tune the height of the legs for levelling. And there are these steel tube sleeves that go around the legs, that drop down and cover the bolts. So when they were moving this platform in, they had to flip it over, and they didn't want the sleeves to fall off while they did this. Now to me, that job calls for duct tape. To them: they welded the sleeves in place, then de-welded them when they needed them to move again."

When you step back and look at it - Excel is a programming language. But not in the way that you and I think about it - with source code and a compiler and a execute/debug cycle. It's a cell based programming language where data and code are intermixed but never mistaken for each other. It's self modifying but easily observable. Changes are implemented trivially and immediately. And in parallel. Whilst most programmers have been struggling to get to grips with functional programming and Monads our brethren at the other end of the office have been doing it for years. And their IDE is better than any of ours. Hell it even has visual data and code dependency tracking:

Think I'm joking? Take a look at this article about building a 3D Engine in Excel. A 3D Engine with multiple rendering backends. And, if we try and shoe horn conventional programming metrics in it does it in 30 lines (and 24 columns) of 'code'. Including comments.

To be honest I'm being a little light hearted here but actually my point is kind of serious. There's a new wave of programming paradigms - Map/Reduce, asynchronous execution, grid computing, sharded and column orientated databases and others. These aren't new ideas, especially in the academic world , but they're gaining more widespread acceptance. A cell, or to look at it slightly differently, a node based approach makes a lot of sense for a bunch of them.

At my previous job we were used to dealing with huge quantities of data a day. Our rendering farm sat on the list of the worlds top supercomputers. We dealt with parallelism all the time - from Renderman to pixel and vertex shaders. We did our compositing using a program called Shake which is entirely node based.

Shake's kind of interesting (apart from Apple's slightly comical attempts to dissuade people from using it on Linux) - it's a very different way of doing image creation than what most people are used to. I watched with amusement the blogosphere cooing over the price drop for OS X and then giggled when they fired it up for the first time and didn't know what the hell to do with it. But my co-workers who really knew how to drive it used it all the time - need to put a mountain behind the crusaders. Use Shake. Need to create a facsimile of a bustling medieval London Bridge using nothing but a background plate of somewhere in Prague, some smoke elements and some video of a pigeon from outside. Use Shake. Need to create an icon for that new Shake node you just wrote ... use Shake.

It's the same familiarity as the Excel users and the welders.

We ended up writing a Node based programming language called Ripple that automatically went and deployed itself over the farm. It self balanced, passed variables and sorted out the DAG. You just strung nodes together and ran the script and tied into Alfred if you wanted a visual feedback and/or to reprioritise or delete tasks. If you wanted new functionality then you just wrote a new node type - we had nodes that did everything from skinning and compositing to generating dailies and emailing people. It was, to be frank, pretty cool.

I've been lead to believe that several of the major banks have node based languages which do things like complex price matching or constantly take input from the market, ripple changes down through the DAG and give pretty much immediate access to the level of exposure that the bank. Want more capacity? Just chuck more servers in - it should just all work.

There's no real conclusion to this other than - parallel, grid based computing looks like it should be hard but it's coming, it has significant advantages once you can get your head round it and as long as the tools are good it might actually turn out to be a better way to program.
don't encourage

Who, when, where, whatever

I was trying to think of something I could do to play around with FireEagle and came up with something which both tickles my development fancy and also is so incredibly insular San Francisco navelgazing Wanking2.0 that I kind of feel compelled to do it.

So, and excuse the hand waving here, the way it would work is this:

  1. You'd purchase a cheapo RFID reader from somewhere - the ones from ThinkGeek, Phidget and Parallax all look good.

  2. Hook it up to a computer and run TheSoftware which, as yet, exists only in my brain. You will tell TheSoftware what the physical location of your card reader is.

  3. Swipe your brand new card which will prompt you to register yourself with a remote, centralised service.

  4. This service will prompt you to give it FireEagle access.

  5. From this point on whenever you swipe your card over the reader TheSoftware will inform the centralised service which will, in turn, tell FireEagle.

In and of itself this is not very useful but if you had a reader at work then you could swipe in there in the morning and then swipe in at home at the end of the day (or if you work somewhere suitably large then put multiple readers around the place). And then if your friends started getting RFID readers and installing them in their homes then when you went round there you'd be able to easily let FireEagle know where you were. Hell if you could persuade your favourite bars and clubs to do it then you could do it there. Hook it up to your Social Graph and then you can easily work out where all your friends are.

Then of course the data can be subpoenaed by the Government to prove that you're a terr'ist or something.

We didn't start the Fire

Yahoo! recently released FireEagle and jolly nice it was too - I've hooked it up to my Dopplr account and I have an idea of what to do with it of which more later.

However things were a bit confusing - there was a Net::FireEagle on CPAN by Aaron Straup Cope yet also a Net::FireEagle::Client linked to on the FireEagle page itself and they weren't really that much alike.

Because all of SF is a seething cabal I asked around and found that the CPAN version was an early version based on the old version of the API. And the ::Client version was somewhat lacking in things like, well, documentation. Or comments. Also it wasn't on CPAN which makes it somewhat of a second class citizen in the Perl world.

So after a bit, I ended up taking over the both of them. I renamed ::Client to just Net::FireEagle, adding CPAN scaffolding, refactored the hell out of it, wrote a load of docs and some (very basic) tests and a nifty little command line script which also serves as an example of how to do the Auth Dance[tm] (which reminds me - the OAuth Auth Dance is much nicer than the Google, Flickr and especially Facebook one).

And lo the updated version now resides on CPAN. It even has a user.
diesel, learning, evil, sweeti

My life in Non-Tweets

Since I don't have, and refuse to ever get, a Twitter account I have decided to summarise this weekend in the style of LoudTwitter because, yes goddammit I AM that geeky.

  • Friday night French Laundry
  • Saturday afternoon A Luau complete with a pit baked pig
  • Saturday night Awesome ice cream with two completely random new friends on the way to ...
  • Saturday night (again) ... a house warming party where it turns out that the two random new friends who were just giving me a lift knew someone at the party.
  • Saturday night (even later, more Sunday morning) Watch second half of Malaysian Grand Prix at Overtime on 7th and Harrison
  • Sunday morning Tool hire, DIY, ladders, drilling
  • Sunday afternoon the Big Wheel race down Vermont. Hilarity and Panda Bears ensued.
  • Sunday evening post BYOBW pizza and beers and Jesus Camp at ydnar's.
Automatically shipped by My Hands

Sixers (and Jesus) on Vermont

Living in San Francisco really doesn't suck.

Originally posted on

beard, avalanche, coy

By the way, I tried to say

WRT my last post I think I managed to spectacularly avoid saying the most important thing in my head which was this ...

I don't think that's anything fundamentally wrong with any of the PubSub systems in existence at the moment. However most of them seem to have escaped from or are inspired by the kind of messaging you need at banks and other financial type institutions. This is great and many of the design goals are the same but they're designed to be complicated and complete from the get go. And this works for them.

However I want something more like Memcached or Rails or similar - you install it out of the box and it Just Works[tm] and for 80% of people that'll probably suffice modulo some trivial tweaking.

Then there will be another 10-15% of people who can take that base and after some simple to moderate modifications make it do what they want.

There may even be a further 1% who can make it go even further but, at this point, it's diminishing returns and really if things were changed to make things easier for them it would compromise how simple things are for the 80%. And to be frank, the 1% would probably be better off with something designed from the start to do what they want.

Not everyone wants Oracle - some people are just happy with SQLite and MySQL. Hell some people are more than happy with BerkeleyDB.

And that's a good thing.
skinhead, tea

Standing in Line

For years the default website stack has been something similar to the classic LAMP stack - originally "Linux, Apache, MySQL, Perl" but now really meaning "Free Unix, Free Web server, Free Database and Free Web Language" or just "OS, Web server, Database, Language" to be even more general and friendly to the Windows people out there.

Relatively recently we've started to see that we should add another layer - a Cache. To be honest, from what I can see, Memcached has pretty much got this sewn up by virtue of being awesome although there are other technologies like APC. So I'd like to coin a new phrase - the CLAMP stack for Caching LAMP. I can't find a reference to it so I'm going claim it as mine. MINE. MUHAHAHAHAHAAH. Maybe in the future it will make me famous. Maybe. *cough*

Even more recently I think there's been a need for some sort of new layer.
Collapse )
idea, intrigue

What Do I Do Now?

I'm a little discombobulated at the moment for various reasons but this idea popped in to my head and I have no idea how stupid it is. What better to test than to fling it the internet like so much poop and see if it sticks.

Imagine if sites had a

<meta name="searchurl" value="" />[1]

tag in their headers. This would allow agents to autodiscover and utilise a site's search engine if one was available simply by substituting %q for a url encoded query. There could even be a type="..." attribute that gave the mime type of the results - Atom would be good. Although that could just as easily be done with Accept headers and the other standard mechanisms for negotiating types.

Search engines could even use it to get better results from stuff like shopping and review sites.

Of course there's a possibility (nay, a probability even) that it'd be co-opted by spammers and also you have to ask yourself - why would sites provide this as a service and who would want to use it anyway so it's probably one for the "WTF were you thinking" file but hey ho.

I need more tea.

[1] Although it occurs to me that

<link rel="alternative" name="search" href="..." />

might work even better.
fuck code

Anytime, anyplace, anywhere

There are tonnes of JSON modules on CPAN. Why do it one way right when you can do it a hundred ways wrong? JSON::Any mitigates some of these problems by abstracting away the interface so that you can use JSON, JSON::XS, JSON::Syck, JSON::DWIM ...

Annoyingly JSON::XS completely changed its API between versions 1 and 2. JSON::Any dropped support for JSON::XS 1.x and now only supports 2.x.

Until now. This patch feels somewhat dirty but, meh, what the hell, it works.

Stuck in the Middle with You

Yesterday got me thinking - I think it was the combination of the impromptu burlesque show at the flower shop, the gig, the pint, the synchronicity and the conversation but for whatever reason it got me thinking about Shelf again.

Shelf is people orientated - it makes heavy use of the address book and finds connections between what you're doing now and people you know. Which is fine. But it could be cooler.

Instead of just have a person as an initial seed for the clues how about other things? Starting simply - how about urls?

There's already information out there about urls - for a start there's whether it's owned by someone you know. Or its stats from Alexa. Maybe its PageRank value. Then there's when you last visited it and how often you've visited it and what's changed since then. And whether you it or Duggit or whatever. And whether it was mentioned in any of your RSS or Twitter feeds or emails. You could add notes to annotate it.

The next natural step is your friends - what have they said? Have they added notes? When did they last visit it (ignoring the glaring privacy concerns for the moment)? Where did they go next? Hell, throw it open to everyone. What has the rest of the world got to say about this? Suddenly every page has comments whether they like it or not. And notes and errata. It's a Web! It's a Wiki! It's a Dessert Wax and a Floor Topping!

And then there's places. You're looking at a museum or a gallery and it tells you what pubs and restaurants are nearby. And if any of your friends will be close by. Show you photos from the location. Throw in a map. Maybe some historical information or local trivia. Great for when you're sitting at your desk but even better when you're actually out on the street and you look down at your iBlackickreo95 and it's using Cell location or GPS to work out where you are.

Listening to music? Album covers, lyrics, other albums, recommendations. Films? Stuff from IMDB - the actors, what else they've been in, awards, trivia, more recommendations from my friends.

Nurse! Come quick! I think the restraints are coming loose.