Back in the day, 2007 and 2008 there was live running federation between Twitter and Jaiku. I remember i was sitting in the room when Ralph and Blaine got it working, at 4am at a foo camp.
Unfortunately Jaiku was bought by google and they spent a long time with it offline only to come out as Google Buzz which didn't take off.
Many of the early twitter engineers like Blaine Cook and Alex Payne pushed hard for twitter to be more open and federated, but lost that argument and left or were pushed out. It's a business not a technology decision which makes twitter centralized.
551 words spilled on a simple question, with a simple, obvious answer: E-mail.
* Immediacy: if a post has been made by someone I follow, I can see it in my timeline right away (or close enough that I don’t notice the difference). "E-mail"
* Chronology: posts always appear in order by time posted. "E-mail". Chronology as the author imagines it doesn't even occur for Twitter, likely their machines' timestamps are fuzzy +/- perhaps up to a minute or more.
* Monotonicity: timelines grow only from the top; older posts are never retroactively inserted. So have your client mark a message by the time it's received, provide a second sort order. Ugh. "E-mail"
There are genuine technical challenges to implementing a robust many-many federated messaging system, this post mentions none of them.
It's both possible and necessary. Think about it - even a centralized service like twitter or facebook will have many distributed & disparate services that need to work together to provide a feed. Hence, there probably already is some kind of federation going on, you just don't know it.
Source: I work @ LinkedIn on these types of problems.
Author here: a few people have raised this point. Perhaps I should have made a distinction between "federated within the same data center" and "federated across the internet." They are two entirely different problems.
If it was impossible for a federated network, it would also be impossible for Twitter. They can either guarantee that messages only show up at the top, or that messages always show up in order (meaning older posts get inserted into the middle of the timeline, as you eventually receive them).
You could easily allow a user to see messages ordered by "send time" or "received time". You could even add an "unread" flag to messages, so that unread messages show up first, then migrate to their proper locations in order.
Right, but since Twitter manages all of Twitter's servers (and the dedicated low-latency network via which they're presumably all connected) they can satisfy both of these constraints without sacrificing the third (immediacy). I'm speculating here, but I think that's why it takes a few seconds to post a tweet: it's propagating across the internal federation so that all the nodes have it before it "goes live."
You can build a distributed system that has hashtags (ex: by using a DHT to map tags to lists of nodes that act as clearinghouse for messages with those tags) but that system is not the web.
The web is not "federated": site A does not communicate with site B. Contrast with SMTP or XMPP which require federated servers to communicate with one another to establish dialog between two users. All of them are decentralized though.
As the post says, with sufficiently slow servers it's impossible to maintain chronology plus monotonicity, and as it says, since chronology is really only important within a conversation, the best solution is to ensure each reply has an in-reply-to field (which tweets already have), then ensure that the original tweet is fetched before displaying the reply, and accept only approximately chronological ordering for "threads". But... that doesn't mean the UI has to change to look threaded, or that you should see an entire conversation if you're following the person who started it. That's a crazy leap. With such an algorithm it can still look exactly like Twitter and most people won't notice the difference.
As an alternative, by the way, since these posts are so short, you could simply include the entire thread with each reply! A few kilobytes won't kill any servers.
Also, one of the blog comments raises a good point: IRC has the same problem and it works fine.
This is addressed at the bottom. I don't know if this is true but the problem seems to be, although the posts show up chronologically in each thread, the threads might be out of order with respect to each other.
Most email IMAP servers provide both Date Sent and Date Received properties for each message, allowing the client to sort by either. Even in active mailing list threads, I've found that it's not that common for messages to get too out of order.
Assuming federated social services keep track of Date Received, it'd probably be okay to just sort by that.
your node can just wait until it’s received the latest content from every other node before displaying anything.
"Every other node" just means the nodes where the people you follow are hosted, not every node in the network. For most people, that would probably be a very small number.
My hunch is that it's technically possible to go not just federated but radically distributed. However, there's more money to be made in centralization, creating defensible toll booths. And right now the sorts of developers and protocol/system architects who could make a radically distributed alternative are now fully overemployed doing other lucrative or beneficial things.
So it's a little harder technically, and lot less lucrative, to make a radically distributed alternative. And whatever the abuses of the centralized proprietors, they're still not changing those tradeoffs -- most users and even platform participants can still make-do under centralization.
A big tech bust freeing more talent to work on ideological (rather than pecuniary) projects, or enough heavy-handedness from the centralized proprietors, could change that balance.
Personally, I'm willing to lose monotonicity. I mean, it's only Twitter. If I don't see every single message because I didn't scroll back far enough, I don't really care.
I doubt the single global ordering is as important as the author suggests. If nothing is in practice delayed for more than a short period, then either ordering-by-arrival or ordering-by-declared-time-with-'seen'-indicator both mostly work. It's whatever you get used to.
And I've seen all sorts of micro-failures in Twitter's provision of global ordering: gaps in my display timeline in one client but not others; tweets seeming to appear below the 'topmost' position when revisiting the stream; difficulty reloading recent history that was already local at one point.
A distributed/federated system might be a little fuzzier on some ordering behaviors... but make up for that in other dimensions.
[+] [-] rabble|13 years ago|reply
Unfortunately Jaiku was bought by google and they spent a long time with it offline only to come out as Google Buzz which didn't take off.
Many of the early twitter engineers like Blaine Cook and Alex Payne pushed hard for twitter to be more open and federated, but lost that argument and left or were pushed out. It's a business not a technology decision which makes twitter centralized.
[+] [-] forgotusername|13 years ago|reply
* Immediacy: if a post has been made by someone I follow, I can see it in my timeline right away (or close enough that I don’t notice the difference). "E-mail"
* Chronology: posts always appear in order by time posted. "E-mail". Chronology as the author imagines it doesn't even occur for Twitter, likely their machines' timestamps are fuzzy +/- perhaps up to a minute or more.
* Monotonicity: timelines grow only from the top; older posts are never retroactively inserted. So have your client mark a message by the time it's received, provide a second sort order. Ugh. "E-mail"
There are genuine technical challenges to implementing a robust many-many federated messaging system, this post mentions none of them.
[+] [-] alttab|13 years ago|reply
[+] [-] joshhart|13 years ago|reply
Source: I work @ LinkedIn on these types of problems.
[+] [-] dwineman|13 years ago|reply
[+] [-] sophacles|13 years ago|reply
[+] [-] sp332|13 years ago|reply
[+] [-] mbreese|13 years ago|reply
You could easily allow a user to see messages ordered by "send time" or "received time". You could even add an "unread" flag to messages, so that unread messages show up first, then migrate to their proper locations in order.
Ya know... kinda like email.
[+] [-] dwineman|13 years ago|reply
[+] [-] Lammy|13 years ago|reply
[+] [-] rabidsnail|13 years ago|reply
Edit: More information on how to actually use Vector Clocks (which are sort of like multi-way Lamport Clocks) in a real system: http://basho.com/blog/technical/2010/04/05/why-vector-clocks...
[+] [-] michaelpinto|13 years ago|reply
[+] [-] rabidsnail|13 years ago|reply
You can build a distributed system that has hashtags (ex: by using a DHT to map tags to lists of nodes that act as clearinghouse for messages with those tags) but that system is not the web.
[+] [-] lloeki|13 years ago|reply
[+] [-] comex|13 years ago|reply
As an alternative, by the way, since these posts are so short, you could simply include the entire thread with each reply! A few kilobytes won't kill any servers.
Also, one of the blog comments raises a good point: IRC has the same problem and it works fine.
[+] [-] jimktrains2|13 years ago|reply
[+] [-] rabidsnail|13 years ago|reply
[+] [-] taybin|13 years ago|reply
[+] [-] sp332|13 years ago|reply
[+] [-] kennywinker|13 years ago|reply
[+] [-] pixelcort|13 years ago|reply
Assuming federated social services keep track of Date Received, it'd probably be okay to just sort by that.
[+] [-] icebraining|13 years ago|reply
"Every other node" just means the nodes where the people you follow are hosted, not every node in the network. For most people, that would probably be a very small number.
[+] [-] gojomo|13 years ago|reply
So it's a little harder technically, and lot less lucrative, to make a radically distributed alternative. And whatever the abuses of the centralized proprietors, they're still not changing those tradeoffs -- most users and even platform participants can still make-do under centralization.
A big tech bust freeing more talent to work on ideological (rather than pecuniary) projects, or enough heavy-handedness from the centralized proprietors, could change that balance.
[+] [-] rabidsnail|13 years ago|reply
[+] [-] Tichy|13 years ago|reply
[+] [-] moe|13 years ago|reply
Regardless I don't find the notion of a federated twitter particularly interesting anyway. The future is P2P.
[+] [-] kniht|13 years ago|reply
[+] [-] kscaldef|13 years ago|reply
[+] [-] gojomo|13 years ago|reply
And I've seen all sorts of micro-failures in Twitter's provision of global ordering: gaps in my display timeline in one client but not others; tweets seeming to appear below the 'topmost' position when revisiting the stream; difficulty reloading recent history that was already local at one point.
A distributed/federated system might be a little fuzzier on some ordering behaviors... but make up for that in other dimensions.
[+] [-] mparlane|13 years ago|reply
[+] [-] unknown|13 years ago|reply
[deleted]