Simon Wistow (deflatermouse) wrote,
Simon Wistow

Now that it's raining more than ever

I've been thinking about Pub/Sub messaging systems for a while now.

ActiveMQ and I are no longer friends. I dislike going into details but things had been simmering for a while, the odd snide remark, a bit of passive aggressiveness tinged with disappointment until one day something small turned into a big argument and, well, let's just say that things were said. "Buggy, no good piece of crap" springs to mind now that I come to think of it.

Yesterday, some other 6Aers and I had lunch with Blaine Cook and some of the Twitter crew to talk to them about Starling, their new high performance messaging system based around Memcache and to swap war stories. Hark at me! Hanging around with the hip SOMA kids! It's like the time the cool kids at school stopped wedgieing me and let me hang around with them as long as I took the blame for torching that car. Ah, good times.

Anyway, it's an interesting approach - obviously not suitable as a drop in replacement for TibCo's Rendezvous or whatever but definitely useful in certain situations. One limitation is that there are only Queues i.e only one consumer can grab a message. However I think it should be possible to hack things to allow Topics i.e multiple

One idea was

  1. Designate a certain key a Topic somehow, possibly by prefixing the key with "topic://" or similar.

  2. Every client that subscribes to this Topic is noted by the system.

  3. Messages are posted just as normal.

  4. Then a special client listens to the Queue and then duplicates N copies of the message (where N is the number of subscribed clients) and inserts the copies in a queue for each client.

  5. Each client then polls their individual queue as normal.

This is conceptually very easy but has the disadvantage of needing N copies of the message which could be a waste of memory.

However it would probably also be possible to only have 1 copy of the message but keep a ref count. Hell, Memcache even has atomic dec() and inc() operators - Que Convenient!.
Tags: activemq, batshit hacks, memcache, pubsub, twitter

  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.