top | item 5769295

Google defends dropping chat federation with inaccurate comments

175 points| AndrewDucker | 13 years ago |plus.google.com | reply

102 comments

order
[+] kkowalczyk|13 years ago|reply
Unfortunately, the guy responds with his own misinformations.

Google said: "For example, mobile has several requirements around bandwidth and battery that are simply not part of the standard"

He says: "This glances over the the fact that the X in XMPP stands for eXtensible, which still results in proposals for new protocol extensions every month."

His misinformation is this: XMPP is XML based. The bandwidth that Google most likely refers to is a result of verbosity of XML. You can't eliminate that by extending XMPP, something that a guy who serves on XMPP Council should know.

The only way to fix that specific problem is to change the protocol. Which is what Google did.

This is an example a bad advocacy: badmouthing companies that decided to no longer use something you work on.

I also find troubling the pounding Google gets for every little mis-step.

When it comes to using and promoting open technologies Google is seemingly from another planet compared to other tech giants (Apple, Microsoft, Facebook, Amazon, you name it).

And yet the furor is not about Microsoft using completely secret protocol (Skype), not about Apple promising to publish their secret protocol (Facetime) and then not doing it, not about Apple preventing fair browser competition in iOS etc. but about Google no longer using XMPP.

A lack of perspective.

[+] jerf|13 years ago|reply
"You can't eliminate that by extending XMPP, something that a guy who serves on XMPP Council should know."

Ehhh... well... if you're willing to put some work into the matter that may not be as true as you'd think. Yes, to speak the federated protocol to another server you're going to be speaking the XML protocol, but between a server and a client that come to accommodations with each other, you don't really have to speak XML.

You'd want to speak something isomorphic to the XML, but the XMPP XML is not that verbose on a node-by-node basis or anything (minus a couple things, see below), it really is the XML, not the semantic content. It would be fairly easy for a medium-to-high skill developer to come up with an isomorphism for use with mobile clients. (It is sufficiently tricky that there would be a few traps that could catch the inexperienced or unwary (watch those namespaces!), but it isn't that difficult.) The end result is a more bandwidth efficient protocol that is a drop-in replacement at the XMPP library layer in the client for XMPP. (Of course, if your client is programmed to fling around XML all over the place internally this won't help, but, well, that's the price of failing to abstract properly.)

Where you'd still have trouble is things like roster size, but yes, there are XEPs for that, and before long you're looking at a reasonably svelte protocol (not perfectly, but reasonably svelte) that is still largely XMPP and easy to write communication with the outside world.

What's really difficult is semantic IM translation between entirely different systems, not protocol translation. ejabberd, the server whose internals I am most familiar with, essentially works by internally speaking Erlang terms with itself, and translating to the XML protocol only at the edges. It wouldn't be very hard to drop in different serialization at the edges and get a different protocol; as proof of concept of the fact this is possible, one could easily simply serialize those Erlang terms with Erlang's term serialization, and indeed, when operating ejabberd in a clustered mode, that's actually happening across the wire as the cluster nodes communicate with each other.

However much effort that may be, it seems easier than starting from scratch.

[+] crsna|13 years ago|reply
The furor is not about dropping XMPP; it is about dropping the philosophy of open communication they championed: "Client Choice, Service Choice and Platform Choice" (https://developers.google.com/talk/open_communications). If Google chose to drop XMPP in favor of some other protocol that allowed others to adopt and interoperate, the discussion would have been about the effort required to migrate. But this move by Google is a huge blow to those who hoped for a world of ip based open communication solution.

Google is in a position to define the course of internet and has used that power in several ways in the past. This move from an open communication network to a closed network, to me, is as defining as WebGL and WebRTC are, but in a very unhealthy way. Google's move therefore is clearly 'evil' (as per their own definition of evil) as it is a clear choice they made to safeguard their vested interests, stating that other corporations are unduly benefiting from their openness as a reason.(http://www.youtube.com/watch?v=AfK8h73bb-o [2:40])

They stated that major networks did not interoperate with them. But in 2007 AIM started interoperating with Google Talk. http://gmailblog.blogspot.in/2007/12/gmail-chat-aim-crazy-de.... Doesn't that count? What did Google do to promote federation, other than asking people to contact them to federate? It appears that when Google was small, they tried to be open to benefit from other big players. Now that they are the biggies, they don't see any value in being open.

I believe more than any other company you mentioned, Google benefited significantly by projecting themselves as the messiahs of internet, purportedly promoting openness and standardization. They captured the imagination of people like us by promising to be only doers of good, placing the objectives of internet and humanity ahead of the corporation's benefits. They did so, as long as it worked in their favor. Those initial adopters who stood with them for these objectives are either absorbed into Google to work as if Google's interests are internet's interests or they are too few and left with fewer alternatives to significantly influence Google and its corporate objectives. With many of the influential personalities deeply affiliated with Google, we have fewer voices calling a spade as a spade.

[+] magic_haze|13 years ago|reply
Really? There are plenty of valid reasons why XMPP is not suitable for mobile usage (most notably, the proprietary push notification systems that all platforms insist you have to use), but XML isn't one of them. If you use TLS with XMPP (as gtalk does [1]), you get the compression protocol for free, but even if you can't, there's XEP-0138 [2], which specifies the use of zlib and has been in development for over 12 years now, and finalized since 2009. XML is fugly, but it works. At the scale Google operates, it is laughable that the use of XML in chat protocols could possibly impact their bandwidth accounting.

[1] https://developers.google.com/talk/open_communications#devel...

[2] http://xmpp.org/extensions/xep-0138.html

[+] ghshephard|13 years ago|reply
Re: "His misinformation is this: XMPP is XML based. The bandwidth that Google most likely refers to is a result of verbosity of XML. You can't eliminate that by extending XMPP, something that a guy who serves on XMPP Council should know."

He directly addresses the Bandwidth concern by noting that interoperability on the server side is irrelevant to mobile anyways:

"Of course none of these concerns for mobile are applicable for server federation. As far as I know, the Google Talk client on Android doesn't even use XMPP as the client-to-server protocol."

[+] shmerl|13 years ago|reply
The only way to fix that specific problem is to change the protocol. Which is what Google did.

Creating new and better protocols is OK. Making them closed and non interoperable - is not. XMPP is not ideal, but it's open for participation in its development and for its usage. Whatever Google decided to create is not open, and that's their main failure.

[+] icebraining|13 years ago|reply
Nobody expects better from Microsoft, but they still expect better from Google, hence the sense of betrayal and subsequent furor.
[+] anoncow|13 years ago|reply
>And yet the furor is not about Microsoft...

The furore is partly also about Google moving away from an open protocol to a supposedly secret protocol. If Google were to publish the new protocol that Hangouts uses all/most of the animosity would vanish. And people would come up with a xmpp to hangouts bridge too.

Google has supported and even championed open source initiatives and it has served them well. And imo it will serve them well in the long term if they continue being open, instead of trying to monopolize a segment of the market.

[+] sbuk|13 years ago|reply
Not a lack of perspective at all, more of an attempt at deflection because Google are being dicks. Skype's protocols predate Microsofts takeover, so attempting to 'blame' them is churlish, and there already has been a furore over Apple and FaceTime. The browser thing has absolutely sweet FA to do with this.
[+] eunice|13 years ago|reply
Google have spent years building a brand out of how """open""" they are; a lot of their marketing/fud is based around it.

Basically they co-opted the language of open source to rube nerds, while nobody is under any illusions re: Microsoft & Apple.

[+] rasterizer|13 years ago|reply
This.

Chromium alone puts them in the plus column standard-wise. Usually apps are judged on merit, yet in Google's case there is the misconception that they have to defend every engineering decision they make.

I don't see how this is a hot topic seeing as every other IM is an island, and quite frankly if it means directing more resources towards projects like Google Fiber, personally I think it's more than worth it to sunset GReader or move away from a limiting standard.

[+] dvt|13 years ago|reply
Really? Can you so harshly judge Google on abandoning a (lets face it) terrible protocol?

Maybe I could get over the fact that it's so complex -- which puts an unnecessary strain on both servers and clients -- and maybe I could get over the fact that (on average) servers consume 40% more memory than their IRC counterparts. Maybe I could even get over the fact that the spec could change at any moment (the eXtensible nature of it isn't necessarily a good thing, you know).

But what I couldn't possibly in a million years get over is that... it's XML (shudder).

-- To the downvoters:

Here's some previous discussion about why XMPP sucks: https://news.ycombinator.com/item?id=2069810

I'm currently working on a project that deals with real time communication (think IRC for the new web) and after fidgeting around with XMPP for a couple of weeks, I gave up on it completely. It's awful. I doubt many people have implemented an XMPP client (or even worse.. a server). Try it some time.

[+] aimatt|13 years ago|reply
I understand the frustration of working with a fat protocol. What is a viable, open alternative though?
[+] davorak|13 years ago|reply
> All that talk about Client Choice, Service Choice and Platform Choice has been replaced with "if the other big players don't play, why should we?".

It seems like the impact of the other big players not playing is down played here. Did it not allow facebook users to start conversations with google talk users but not vice versa?

Playing the cooperation card first is what google did, but in the iterated prisoners dilemma you often need to play the non-cooperative card if that is what your peers are doing.

That really seemed to be the core of the issue. Now 5-25 years from now maybe google will have an entrenched culture of playing the non-cooperative card and will not reach for the cooperative card first but I do not judge this to be the case yet.

[+] ralphm|13 years ago|reply
I believe I get to downplay that because the federation they joined helped grow their own network, and allowed them to do Google Talk for other domains. I know many companies that have moved their XMPP stuff to Google for that.

Now those outside contacts are cut off, without any warning. Worse, they see their inside contacts online, can send them messages that never show up in Hangouts!

Those big players are hardly affected, though. Also, Google could simply block those entities that don't fully federate (only).

[+] threeseed|13 years ago|reply
There are a few examples though where Google has opted not to support or develop standards. For example WebM, SPDY and XMPP and not to mention their continued support for Motorola's despicable FRAND abuses.
[+] hayksaakian|13 years ago|reply
It was ultimately a business decision - the same kind that leads to product shutdowns and all the other "evil" things that new Google does.
[+] steveklabnik|13 years ago|reply
Oh, a _business_ decision. I guess we can just let them continue to embrace, extend, and extinguish the commons we've built over the last decades, then. No complaints necessary.

(I'm not a GOOG shareholder, so their business needs don't matter to me. I _am_ now going to be cut off from my contacts, though. Part of markets is the feedback that comes with mis-steps, and saying "it's just business" does a dis-service to those of us who feel that Google has been over-stepping their bounds.)

[+] ralphm|13 years ago|reply
That is perfectly valid, to me. However, there is no reason to spread misinformation to cover such a decision and to bluntly cut off existing communications between people outside and inside of the Google Hangout domains. Without any warning to the former, even.
[+] kyrra|13 years ago|reply
A lot of the shutdowns have been due to low (for google) user bases, or just no engineering staff to support it. A lot of their shutdowns I see as a way for google to become more focused as a company, which I feel like was needed. They had too many half-baked services.

As for XMPP, it was their business decision. I'm betting engineering had a good reason for it, it just may not be shared with us.

[+] jjuliano|13 years ago|reply
The problem with XMPP is that it requires several open ports (5222, 5223, 5269, 5298, 8010) on the client, on the technical and practical standpoint the reason why XMPP is unpopular to big players because many businesses won't be bothered to open their ports, now if you are a solutions provider creating software for XMPP solutions for those companies, the most viable solution is to use BOSH to direct XMPP traffic to port 80, but there's another big problem, the only available BOSH library is a GPL BOSH library that you would license in order for you to not to ship your software along with it's source-code. Now if you can do the same thing over HTTP 1.1 stream, why bother with XMPP?
[+] BarkMore|13 years ago|reply
* The problem with XMPP is that it requires several open ports (5222, 5223, 5269, 5298, 8010) on the client.*

XMPP does not require any open ports on a client.

[+] mh-|13 years ago|reply
I have no idea where you got the idea about binding all (any) of those listening ports on the client, but that is incorrect.

anecdotally:

  % lsof -i4 -T s
  COMMAND   PID USER   FD   TYPE             DEVICE 
  imagent   158   mh    6u  IPv4 0x2969ca78b2e9b6bb      0t0  TCP mac:52812->pb-in-f125.1e100.net:5223 (ESTABLISHED)
authoritatively:

http://wiki.xmpp.org/web/SRV_Records#Ports

[+] Zash|13 years ago|reply
Why not? The same applies to running an HTTP or email server, which many businesses do.
[+] rakeshpai|13 years ago|reply
I think I might have 2c to clarify the OP's stand, having worked on a mobile XMPP client before. This is with regards to his point about bandwidth and battery usage on mobile.

XMPP is a connection-oriented stateful protocol. The connection setup process is very expensive: Connect, authenticate, advertise the list of supported client features, get the roster (contacts), and IIRC presence exchange, etc. Therefore, once you've established a connection, you really don't want it to break. Even just reconnecting is very expensive.

However, in the mobile scenario, regular connection breakage is the norm. What most mobile based clients will then do is proxy the XMPP connection on the server-side and connect to the client over HTTP instead. This is well defined in the BoSH spec (XEP:124). BoSH was originally designed for web-based clients, but happens to meet the need of the mobile use-case.

However, when it comes down to implementation, there are several optimizations that can be applied:

1. The roster exchange at the connection time is very expensive, and usually only sends data to the client that the client probably has already stored locally from a pervious session. Connection overhead can be reduced drastically if there was some way of versioning the roster and only transferring deltas. This was the OP was referring to. (Roster versioning, XEP:273).

2. Presence packets are one of the most chatty aspects of the protocol (x went offline, x is now online, etc). I've heard someone say that it constitutes about 80% of the packets on a typical connection, but this is anecdotal. In cases where the user isn't actively interacting with the client, having presence packets being sent all the time seems wasteful. One way to tackle this is to buffer up and de-duplicate the presence packets on the server, and send them in batches only when necessary. (For example, if x goes online, then offline, then online again, and then goes 'busy', the client only needs to know that x is busy.) I think this is what the OP was referring to when he mentioned the google:queue for delayed presence delivery. This alone can be a huge win for battery and bandwidth, since transferring data and processing it can be reduced drastically.

Aside: Presence is so chatty, that some mobile clients simply don't handle presence at all. Take WhatsApp for example. It doesn't show whether every person in your list of contacts is online. The assumption is that every user is online by the nature of the device, and this assumption works in the common cases. In the edge cases is where you'd need to buffer up messages. This reduces chattiness drastically, making the protocol seem much more lightweight. (WhatsApp is based on XMPP, but have spun it off into their own thing.)

Just thought I'd add these points to the discussion here, since there seemed to be some lack of understanding of the protocol. IMHO, the XML, as verbose as it is, isn't really the problem. The protocol itself is verbose and whether the transport uses XML or whatever else is of comparatively little consequence. This verbosity of the protocol (and not so much the format of data transfer) has its effect on battery life and bandwidth usage.

[+] mh-|13 years ago|reply
>Aside: Presence is so chatty, that some mobile clients simply don't handle presence at all. Take WhatsApp for example.

One could also take Hangouts for example. :)

anyways, very informative post and good anecdotes, thank you for sharing.

my understanding of the XMPP spec is that (assuming the server follows the server-client spec, anyways) there's no way to unsubscribe from your contacts' presence notifications and keep them on your roster [and therefore able to be contacted without reauthorization, usually.]

can you confirm/enlighten me? thanks!

[+] fakeer|13 years ago|reply
1. Microsoft/Facebook (and maybe more corporates) didn't play by the rules. They don't believe in "give and take", they just do "take, take,..". Google didn't like it so walked out. Instead of showing the sportsman's spirit and instead d of carried on despite of all the follies they simply left having pissed off. Good decision.

2. Universities, organisations? The number is too small to be bothered about, esp when compared with the likes of the two above^. They would rather innovate and try to lock-in users into their own platforms rather than hoping someday Cuck might turn the benevolent saint.

3. Whatever you say but unless you are short of manpower and talent it's actually easier difficult to innovate in-house sth proprietary and not when you've to conform to a "standard" and that too when the rewards are [1].

4. Even I don't like the move but it was a business decision and this one made a lot more sense than killing off GR.

The surprising fact here is, we hardly see anyone criticizing the providers that chose not to share back. We, keep on saying "no problems with XMPP" and "even though MS and Fb were not giving back, Google should have just let them free-load after all it's a do no eveil corp". Huh.

[+] onedev|13 years ago|reply
I'm interested in how you think MSFT and FB just "took, took, took..."
[+] kmasters|13 years ago|reply
Isn't there a better question to be asked here? Federation is supposed to enable, well, federation. Google is not preventing XMPP federation in any way. Google is not "shutting down" XMPP or killing it.

Why is Google the company important to the XMPP community? I scarcely see that it should matter.

Open protocols are not exclusive clubs or infinite universes, anyone can opt in or out at any time, the rest of us will live.

I think its great when big companies such as Google or whoever participate in open standards, but I also appreciate that they are in the business of providing the best product for their users (in their eyes) and are free to do what they want.

When they do participate, we are getting a gift from them. Its not their obligation, right or even necessarily to their benefit. When they find other priorities fine, maybe someone more committed will move the project forward. And in other cases their NON-participation can be a gift as well.

So whats the big deal?

[+] Macha|13 years ago|reply
Google Talk was the largest federated XMPP server, and it provided an easy way for people using XMPP to talk to non-technical users. "I'll add you on XMPP" = blank stares, "I'll add you on Google Talk, just use your Gmail account" works, even if you yourself are not using Google Talk.