Making the real-time web real-time

RSS, Atom, SUP, Web Slices - in the context of receiving site update notifications they are all based on regularly querying the site to find out what has changed. To quote Kirk in one of his rants about SUP:

But it's still polling! It still ultimately doesn't scale. You can make polling suck less, but it will always suck.

Great though SUP in its current form is, it's really an optimisation hack rather than a solution, designed to reduce the traffic to a given site. But still if I have 10,000 consumers hitting my site's SUP address(es) every second then that's still going to give me a headache.

The solution part 1: SUP-Push

The first part of my solution to this problem is to turn the problem around. Whilst there's nothing in particular about the Internet Protocol that is asymmetrical, the whole World Wide Web has grown up around a client requesting information from a server which then sends it a response. The reasons for this are partly the evolution of WWW from FTP and then Gopher, and partly because it puts everything under the control of the publishing site rather than relying on any intermediaries.

As we move toward the real-time web, consumers don't want to have to wait to see a Twitter message or even a new news event appear in their client app, they want them instantly. The only away to achieve this is to push update notifications rather than polling for them.

This requires two changes at the publisher site:

• Provide (and advertise) a way for clients to make a connection that they keep open and down which updates are sent.
• Send updates down this connection as soon as they happen (or in batches at a frequency determined by the publisher).

The good news is that providing a connection is easy - let's just use a URL to a TCP/IP socket (or say HTTP) for that - and that with not very much work at all we can use the SUP update document format to deliver the changes.

The bad news is that keeping lots of sockets open to clients is not scalable, and so to:

The solution part 2: Distribution network

Obviously every client keeping a socket open to every server is wrong, so what we need is some kind of distribution network so that clients can connect to just a few (or maybe just one) distribution node and receive updates for any of the feeds to which they are subscribed. Similarly, the web sites will not want to have to publish to every distribution node just in case someone is subscribed, so they will publish to a few "seed" nodes (deliberately using someTorrent terminology here) which then propagate into the wider network.

What we end up with therefore is some kind of self-organising mesh that works out the most efficient toplogy based on some function of load and internet distance (by which I mean latency) to the consumer.

It would be nice if there was already a ubiquitous asynchronous message-oriented middleware out there that we could use to implement the distribution network, but although AMQP may get there eventually it's not there yet. Similary it would have been nice if IPv6 had helped us out here but although they've overhauled multicast and introduced "anycast", these still don't really help outside an organisation.

So instead, if we assume that the end-client (you or I) has a few well-known nodes that they can connect to (hosted by their ISP say), the sequence probably would look something like this:

• Client establishes connection to local distribution node.
• Client sends request for subscription URL to its local distribution node.
• If the node already has that subscription then it just starts sending updates down to the client. End.
• Otherwise, the node requests the publishing site for a connection.
• The site can either give the connection and start sending updates, or alternatively send a redirection to one of its seed nodes.
• If redirected to a seed node, the local node and seed node then use some cunning algorithm based on existing routes to give the local node the most efficient source for its updates.

Over time as loads on nodes change and as nodes appear and disappear, the network will update its topology to optimise delivery similar to spanning trees in IP routing. There may well also be some interesting overlaps with the way torrents work and in particular Broadcatching, but I've not that through yet.

But who should pay for the distribution nodes?

This is probably the key thing that would drive the success, and we have three main options:

• The end-user pays, as they receive the benefit of improved performance. This would translate into ISPs hosting nodes but charging extra to clients to be able to use them.
• The ISP pays as it reduces network consumption through their network. As more people move to the protocol, network traffic due to polling will decrease so the ISPs obviously get benefit from hosting nodes at key points just on the internet backbone. Latencies between ISPs would be minimal, and the distribution network topology would align to major routes across the internet.
• The source web sites pay, as they have lower traffic on their servers to deliver the same amount of content to users. Again this would mean the company pays some provider to use their nodes as seeds. As well as ISPs, companies like Akamai would be well placed to deliver this kind of service.

In reality I'd expect it would turn out to be a combination of the three - to start with it would be free, hosted by people developing the idea, then as ISPs catch on people at both ends will have to pay something to get better performance than the freebies, and finally it becomes taken for granted so incorprated by ISPs into the general cost of using the internet.

So what next?

Unfortunately I have a day job (or maybe fortunately in this market), so for this to go any further it would need to be picked up by some cool developers that are also in a position to push the agenda. As the authors of SUP, maybe the FriendFeed developers should take it to the next stage...

FriendFeed developers, give us some love!

I like FriendFeed a lot. It has displaced FaceBook and my PS3 as the thing I do when I'm not doing anything. And there are new things happening on it all the time, so the developers are obviously hard at work. And they have a FriendFeed feedback room where people talk about the things they want, and they have bug submission forms, etc., etc.

But I've no idea whether the developers really pay attention to what people say in the room or the forms. Given the fix-it Fridays I guess they're looking at the bugs, but who drives the new features?

Well, it should be us, the users.

And this may be happening, but there's no way of telling.

I wrote previously about UserVoice, and I'm bringing it up again now because I think this is exactly what FriendFeed needs. I'll even give you the piccy again:

Let us tell you as a community what's important to us, and let us see what you're doing about it.

And of course if you managed to do some cool integration between the feedback room and the UserVoice page that'd be even sweeter - comments map to comments, likes map to votes, don't need to register separately and so on.

Go on, show us some love!

Give me more social interop!

I'm sure most people by now are like me and have a proliferation of accounts across social media sites: Twitter, Digg, Blogger, FaceBook, WordPress, FriendFeed, MySpace - the list is endless.

Fairly recently I signed up on FriendFeed, which does an excellent job of bringing all these together into one place and allowing you to do interesting things with the combined social stream. The developers are really active improving integration with other sites (especially Twitter), respond to feedback from the community, plus they sometimes come out with things like SUP which have much broader implications.

But it's not perfect. And it's mainly not their fault.

My main beef at the moment is around what happens if someone posts something that people then comment on. Say for example that Robert Scoble posts something an interesting article on his blog, which will then automatically appear on FriendFeed. FriendFeed does an OK job of spotting when someone else shares it on something like Digg or Google Reader it so it appears in their feed as well - it marks it as a related article of the original post. But where do you comment on the entry - in FriendFeed or in the blog? If you do it in FriendFeed then anyone not also on FriendFeed and subscribed either to you or to the author will not see it. If you do it in the blog then the comment doesn't make it to FriendFeed. The author has to keep track of multiple separate discussion threads across the original blog site and FriendFeed.

Open it up everybody!

So everyone needs to stop playing the "my site is the one true site" thing and accept that there will always be n sites catering to different audiences, but with some severe overlap. Now we're all thinking straight, open up the APIs for commenting/retweeting. If I comment in FriendFeed it should be added to the blog entry (maybe this should be an option for the author when they add their feed to FriendFeed). If someone comments in the blog entry then it should appear in FriendFeed (again an option for the author) and if FriendFeed knows my ID on the blog/Twitter/whatever, then it should match this up so it appears in my activity stream in just the same way as if I'd commented directly in FriendFeed.

How hard can that be? I don't even think there are commercial arguments against it, as surely this would just drive extra traffic to the sites because there would be more interesting discussions.

So FriendFeed developers please come up with a cool API for doing this, then everyone else please implement it so we can bring it all together.

Update

Just spotted that when you comment in FriendFeed to someone else's Tweet then you have the option of also doing it as an @reply on Twitter. Good stuff - now what about the rest?

Update 2

It looks like using Disqus to manage your comments gives you some integration with FF, but I'm not sure yet if posting a reply in FF will come back as a comment...