?

Log in

No account? Create an account

Tue, Apr. 8th, 2008, 11:50 am
You don't really want it any more from me

People use Excel a lot. Abuse is possibly a better way of putting it. In fact they abuse the whole of the office suite - you've all received the three word reply wrapped up in a Word document or the PowerPoint document that just has a bunch of images in it.

The difference is that some of the Excel abuses are actually pretty clever. There's those quizzes that get emailed around - guess the lyric, solve the puzzle. But even beyond that you get the Finance and Ad booking departments that run off a single shared Excel spreadsheet of such breathtaking utility that you kind of have to step back and gasp.

It's so horrible! Yet so beautiful! It does everything for them - it tracks bookings and various sales stats and shows past performance and projects future trends. Sure, the programmer in you is recoiling in absolute horror - this isn't programming, this is a crime against compilers. Forget spaghetti code or lasagne code. This is macaroni baked with Velveeta. Yet like Mac'n'Cheez[tm] it works. It's delicious. It's a remarkably sophisticated app that does exactly what they want and does it fast.

And they wrote it. The sales people wrote it. These people who seem to look down on programmers and who won't update the HTML for the copy on the web page because "I was never very good at maths at school".

I'm being flippant of course. And despite some occasional minor differences, I've always got on with the biz dev teams in the companies I've worked for which is why it's sometimes baffled me that they won't even touch, say, HTML.

The problem, I think, is familiarity. We know that the amount of damage they could do with HTML is minimal. Except, well, if they leave out a single character then maybe the whole page is blank. It's not a big deal to us - we know it's not hard and that anything is easy to fix but that's beside the point. We're familiar with HTML, with the Web stack. They're not. In much the same way we're not familiar with (plucks bogus example completely out of thin air) filling in form 64-8 for authorisation of a VAT agent.

Yet they're familiar with Excel. And not just as a spreadsheet - as an environment. They know what makes it tick, what its capabilities and quirks are and how to make it dance in a way that you and I may never.

I'm kind of reminded of this quote about welders

"One of the unexpected things about watching the steel guys work is how the solidity of metal means nothing to them. Most people think of metal as something hard and inflexible, but welders don't. Which should be obvious in hindsight, I guess. But, for example, they have these saw-horses that are made of tube steel. And I can see how that came about: they needed some saw-horses; they had some steel. It took them 30 seconds to make them. And, an example with the stairs: the legs of the stairs' landing platform have big threaded bolts for feet, to fine-tune the height of the legs for levelling. And there are these steel tube sleeves that go around the legs, that drop down and cover the bolts. So when they were moving this platform in, they had to flip it over, and they didn't want the sleeves to fall off while they did this. Now to me, that job calls for duct tape. To them: they welded the sleeves in place, then de-welded them when they needed them to move again."


When you step back and look at it - Excel is a programming language. But not in the way that you and I think about it - with source code and a compiler and a execute/debug cycle. It's a cell based programming language where data and code are intermixed but never mistaken for each other. It's self modifying but easily observable. Changes are implemented trivially and immediately. And in parallel. Whilst most programmers have been struggling to get to grips with functional programming and Monads our brethren at the other end of the office have been doing it for years. And their IDE is better than any of ours. Hell it even has visual data and code dependency tracking:



Think I'm joking? Take a look at this article about building a 3D Engine in Excel. A 3D Engine with multiple rendering backends. And, if we try and shoe horn conventional programming metrics in it does it in 30 lines (and 24 columns) of 'code'. Including comments.



To be honest I'm being a little light hearted here but actually my point is kind of serious. There's a new wave of programming paradigms - Map/Reduce, asynchronous execution, grid computing, sharded and column orientated databases and others. These aren't new ideas, especially in the academic world , but they're gaining more widespread acceptance. A cell, or to look at it slightly differently, a node based approach makes a lot of sense for a bunch of them.

At my previous job we were used to dealing with huge quantities of data a day. Our rendering farm sat on the list of the worlds top supercomputers. We dealt with parallelism all the time - from Renderman to pixel and vertex shaders. We did our compositing using a program called Shake which is entirely node based.







Shake's kind of interesting (apart from Apple's slightly comical attempts to dissuade people from using it on Linux) - it's a very different way of doing image creation than what most people are used to. I watched with amusement the blogosphere cooing over the price drop for OS X and then giggled when they fired it up for the first time and didn't know what the hell to do with it. But my co-workers who really knew how to drive it used it all the time - need to put a mountain behind the crusaders. Use Shake. Need to create a facsimile of a bustling medieval London Bridge using nothing but a background plate of somewhere in Prague, some smoke elements and some video of a pigeon from outside. Use Shake. Need to create an icon for that new Shake node you just wrote ... use Shake.

It's the same familiarity as the Excel users and the welders.

We ended up writing a Node based programming language called Ripple that automatically went and deployed itself over the farm. It self balanced, passed variables and sorted out the DAG. You just strung nodes together and ran the script and tied into Alfred if you wanted a visual feedback and/or to reprioritise or delete tasks. If you wanted new functionality then you just wrote a new node type - we had nodes that did everything from skinning and compositing to generating dailies and emailing people. It was, to be frank, pretty cool.

I've been lead to believe that several of the major banks have node based languages which do things like complex price matching or constantly take input from the market, ripple changes down through the DAG and give pretty much immediate access to the level of exposure that the bank. Want more capacity? Just chuck more servers in - it should just all work.

There's no real conclusion to this other than - parallel, grid based computing looks like it should be hard but it's coming, it has significant advantages once you can get your head round it and as long as the tools are good it might actually turn out to be a better way to program.

Wed, Apr. 9th, 2008 04:21 am (UTC)
crucially

"I've been lead to believe that several of the major banks have node based languages which do things like complex price matching or constantly take input from the market, ripple changes down through the DAG and give pretty much immediate access to the level of exposure that the bank. "

That would be using excel in front of ODBC and Tibco sources :D


Wed, Apr. 9th, 2008 06:08 am (UTC)
billyabbott

I find it scary when I sit down in front of a group of people who I have seen wring incredible things from excel and the c-like data manipulation language that my company provides as part of its service, only to have them tell me that I'll have to go slowly in my training of a simple scripting language as they aren't programmers. A little bit into the class all but the most ardent 'I Can't Do It's realise that actually, things aren't that hard.

They answer my excel questions quickly though.

Wed, Apr. 9th, 2008 09:17 pm (UTC)
pfig

if you pick up spreadsheet::{parse|write}excel, you don't even see perl, there's just last night's expenses and the cell in the red font.

Wed, Apr. 9th, 2008 09:14 pm (UTC)
pfig

exceedingly cool post, thanks!