Wow, “Oh look, we’re going to re-invent the wheel for what is at least the third time.”
When I worked at Google I sat down one day across from one of the gRPC engineering leads who was talking about the things they were doing for the then current generation of gRPC. I asked if I could ask some questions about it and they agreed and then dissected their design in half a dozen ways that would fail in both non-important but irritating ways, and in critical ways at scale. They were amazed at I had thought about this topic so deeply as it was all “state of the art” and I was, nominally, “old.” I pointed out that I had been the ONC RPC architect at Sun in the ‘80s during the first RPC wars and while the implementations had change, fundamentally messaging as a form of procedure call has some fundamentally bad properties. These challenges manifest in all aspects of RPC, from marshaling data, to rendezvousing, to delivery reliability and guarantees. Andy Birrell at DEC SRC and Leslie Lamport had written dozens of papers looking at these challenges in small systems and large. There was literally decades of solid research that the engineer in front of me at the cafeteria that day was re-discovering from first principles.
RPC protocols from Sun, DEC, Microsoft, OSI, SOAP, the IETF, and the Open Group
have run at this problem again and come up with different solutions that each have their own set of warts. Good for some things, not great for others. But at this point there are enough options at this point.
What is missing from Buf’s material is what I might call the “Chesterson’s fence” material that dives into why all of these previous versions were insufficient and how their new version of gRPC will solve all those problems without adding new wrinkles.
I think it is great that they are trying to improve the state of the art, I would feel better about it if they also demonstrated they understood what had come before.
Note that Buf is building tooling around Protobuf/gRPC, not replacing them.
Maybe I'm just another clueless millennial developer who doesn't understand the history of the 80's or whatever, but I've never been able to understand this claim that RPC is broken. There's a lot of assertions that everyone knows RPC was broken because smart people in the 80's said so but... no one has ever been able to give me a concrete reason why.
RPC, at least as I've always known it, really just boils down to request/response protocols. You send a request, you get a response. While this is admittedly not the only possible networking pattern, it is the dominant one across almost all distributed systems I've worked with. HTTP itself is request/response -- it's basically the same thing.
All gRPC and Protobuf are doing that is different from HTTP is they're using a binary protocol based on explicitly-defined schemas. The protobuf compiler will take your schemas and generate convenient framework code for you, so you don't have to waste your time on boilerplate HTTP and parsing. And the binary encoding is faster and more compact than text-based encoding like JSON or XML. But this is all convenience and optimization, not fundamentally different on a conceptual level.
Neither HTTP nor RPC protocols have ever pretended to solve higher-level distributed systems concerns of fault tolerance, network partitions, reliable delivery, etc. Those are things you build on top. You need a basic mechanism to send messages before you can do any of that.
What, exactly, is the magical non-RPC approach of the 80's that we're all missing? Can you explain the alternative?
EDIT: Also, like, the entirety of Google is built out of services that RPC to each other, but the 80's called and said that's wrong? How am I supposed to take this seriously?
May 2020- $1M Pre-Seed
Sept 2020- $3.7M Seed
April 2021- $20.7M Series A
Dec 2021- $68M Series B
How can so many rounds be condensed so quickly for a business like this? Is the number of Homebrew downloads (37k as advertised on the home page) a metric that can lead to an 18 month ramp to Series B now?
I think there's a trend at play here I'd love to hear more about.
I am on the exact opposite end of the spectrum. I have been promoting json ever since Douglas Crockford discovered it. Even my twitter handle is @json.
If someone wants to use json.com to create a company to promote json - DM me.
When shit like this happens, I just always think "I don't understand finance at all and never will."
I read the company's primary blog blog post, https://buf.build/blog/api-design-is-stuck-in-the-past, about "schema driven development" and agree with a lot of it. Which is why I'm a huge fan of GraphQL and related completely free open source libraries, where I define my API endpoints with a strongly typed yet easily evolvable schema, and auto-generate my Typescript types from my GraphQL definitions.
I was just about to ask ‘what are the differences between this and GraphQL?’ Maybe tag on an AWS AppSync & DynamoDB, and you have pretty much all of this?
And before anyone goes all ‘but what about Dropbox?’ when scrutinizing this idea... Dropbox was never really made for technical people, this is squarely at people who know what JSON means, so technical.
Imagine this offering: "Why go through the hassle of generating clients for your service for each language when we can build ergonomic clients automatically" and "Why manually look for breaking changes in your API when we can detect them manually" or "We can give you $AMAZING_FEATURE for free by using a clearer language to describe your api" where your feature could be:
1. Reduced bandwidth ingress
2. Automatic tracing of PII through your system
3. Developer-controlled ops stuff (annotating an RPC as cachable, etc)
4. Automated tracing instrumentation
5. Message streaming (gRPC streams are amazing)
I can think of a whole host of features that can be built off of protos (I've even built ORMs off of protobuffs for simple things [0]). The value prop is there IMO. HTTP + json APIs are a local minima. The biggest concerns "I want to be able to view the data that is being sent back and forth" is a tooling consideration (curl ... isn't showing you the voltages from the physical layer, it is decoded). Buff is building that tooling.
The idea of "I'm raising money to get people to stop using REST/JSON" seems kinda weird to me. I get that they have a product and all but the general lead in here seems weird to me.
Yet another non-business grabbing $ from clueless investors. Surely there must be a word for the “we are grabbing $ from clueless investors” business model? WeWork is the poster child for that one.
The $93M number in the headline is somewhat misleading, as it's cumulative across all the rounds of funding. Can the title be changed to state that the specific round raised now is a $68M Series B?
Article quote:
> We just closed a $68M Series B co-led By Lux and Tiger Global, with participation from Greenoaks Capital Partners, Lightspeed, Addition, and Haystack.
I’m not up-to-date with protobuf, but last time I reviewed it like 4 years ago, it still wasn’t consistent at describing schemas as well as avro, esp in describing schema changes.
Good luck to buf.build to sell a revamped wsdl and getting everyone to adopt it.
$93M..oh no, soon we will be flooded with ads and Steve Jobs like keynotes on why Rest sucks, I thought we were done with those after GraphQL hype slowed down, I guess not..please give it a...rest.
> Hmm why do they need to raise so much cash every 9 months?
Well, they say the best time to take investment is when you don't need it. If you wait until you need it then the terms will be worse. If investors are offering you money when you don't need it, it may be the best time to accept it.
The sequence of raises here look like a fairly normal sequence for a growing startup, except that they happened much closer together than would be typical. The terms aren't shown but assuming they are in line with a typical sequence then this is a great outcome for buf as it gives them lots of room to build their vision without needing to stress over money for a while.
> Also does anyone here use them and have any thoughts about their product?
FWIW, long ago I was the maintainer of Protobuf at Google, including putting together the first open source release. I like what buf is doing -- enough that, full disclosure, I made an angel investment in their seed round.
There's a huge amount of room for better tooling around Protobuf. Binary and strongly-typed protocols require strong tooling to be usable, but with tooling they can be much better than dynamic and text-based approaches. Like, the fact that the protocol is binary shouldn't make it any harder for a human to read it, because your tools should decode it for you on-demand as easily as you could `cat` a text file. Protobuf historically has had sort of the bare minimum tooling and required a lot of ad hoc copying proto files around between projects to get anywhere, which was a pain. A registry seems like the right first step to making things easier, but I'm really excited about what can be done after that... once you have strong type information, you can have tools to dynamically explore APIs, trace communications, etc.
I don't even know why they need so much money for what they do.
Based on their website, they solve the following problems:
- a central schema registery. Even if that's something you actually want, it's not a problem that requires $93M to solve, or a commercial company to operate
- communicating schema changes primarily via human-oriented sources like handwritten documentation on emails. I mean sure, if you are a masochist (or a sufficiently inefficient org), you might do just that. The rest of us here in the 21st century can check the schema into a Git repo instead.
- schema drift on the client end. Tough, this is what happens when you write software. Adding a third party won't help here.
- dependency management. For APIs? I cannot imagine a single case where that would help and your API isn't already a monstrosity.
We already have schema based validation in tons of different shapes.
A binary format may save some bandwidth and be slightly harder to reverse engineer - at the cost of being easily introspectable out of the box during development.
I don't think there is enough value to sell something.
I hope it gains traction and cargo cuting companies with bored engineers start using them, so hopefully the next company I work with won't have some terribly complicated and unusable graphql but just protobufs.
For those who still want / need binary protocols and schemas, look at FlatBuffers or Cap'n Proto instead. At least they are capable of representing domain structures properly.
I can highly vouch for Bufs protobuf linter and breaking change detector [1]. It’s open source.
IMO. Generating gRPC code in multiple languages is pretty tedious setup and maintain. Buf has the potential to replace / free up a lot of time for a small team of people maintaining this sort of thing in house.
[+] [-] ChuckMcM|4 years ago|reply
When I worked at Google I sat down one day across from one of the gRPC engineering leads who was talking about the things they were doing for the then current generation of gRPC. I asked if I could ask some questions about it and they agreed and then dissected their design in half a dozen ways that would fail in both non-important but irritating ways, and in critical ways at scale. They were amazed at I had thought about this topic so deeply as it was all “state of the art” and I was, nominally, “old.” I pointed out that I had been the ONC RPC architect at Sun in the ‘80s during the first RPC wars and while the implementations had change, fundamentally messaging as a form of procedure call has some fundamentally bad properties. These challenges manifest in all aspects of RPC, from marshaling data, to rendezvousing, to delivery reliability and guarantees. Andy Birrell at DEC SRC and Leslie Lamport had written dozens of papers looking at these challenges in small systems and large. There was literally decades of solid research that the engineer in front of me at the cafeteria that day was re-discovering from first principles.
RPC protocols from Sun, DEC, Microsoft, OSI, SOAP, the IETF, and the Open Group have run at this problem again and come up with different solutions that each have their own set of warts. Good for some things, not great for others. But at this point there are enough options at this point.
What is missing from Buf’s material is what I might call the “Chesterson’s fence” material that dives into why all of these previous versions were insufficient and how their new version of gRPC will solve all those problems without adding new wrinkles.
I think it is great that they are trying to improve the state of the art, I would feel better about it if they also demonstrated they understood what had come before.
[+] [-] kentonv|4 years ago|reply
Maybe I'm just another clueless millennial developer who doesn't understand the history of the 80's or whatever, but I've never been able to understand this claim that RPC is broken. There's a lot of assertions that everyone knows RPC was broken because smart people in the 80's said so but... no one has ever been able to give me a concrete reason why.
RPC, at least as I've always known it, really just boils down to request/response protocols. You send a request, you get a response. While this is admittedly not the only possible networking pattern, it is the dominant one across almost all distributed systems I've worked with. HTTP itself is request/response -- it's basically the same thing.
All gRPC and Protobuf are doing that is different from HTTP is they're using a binary protocol based on explicitly-defined schemas. The protobuf compiler will take your schemas and generate convenient framework code for you, so you don't have to waste your time on boilerplate HTTP and parsing. And the binary encoding is faster and more compact than text-based encoding like JSON or XML. But this is all convenience and optimization, not fundamentally different on a conceptual level.
Neither HTTP nor RPC protocols have ever pretended to solve higher-level distributed systems concerns of fault tolerance, network partitions, reliable delivery, etc. Those are things you build on top. You need a basic mechanism to send messages before you can do any of that.
What, exactly, is the magical non-RPC approach of the 80's that we're all missing? Can you explain the alternative?
EDIT: Also, like, the entirety of Google is built out of services that RPC to each other, but the 80's called and said that's wrong? How am I supposed to take this seriously?
[+] [-] digitailor|4 years ago|reply
I think there's a trend at play here I'd love to hear more about.
[+] [-] catsarebetter|4 years ago|reply
[+] [-] felipellrocha|4 years ago|reply
[+] [-] opendomain|4 years ago|reply
If someone wants to use json.com to create a company to promote json - DM me.
[+] [-] moralestapia|4 years ago|reply
Shoot me an email, mine's on my HN profile.
[+] [-] hn_throwaway_99|4 years ago|reply
I read the company's primary blog blog post, https://buf.build/blog/api-design-is-stuck-in-the-past, about "schema driven development" and agree with a lot of it. Which is why I'm a huge fan of GraphQL and related completely free open source libraries, where I define my API endpoints with a strongly typed yet easily evolvable schema, and auto-generate my Typescript types from my GraphQL definitions.
$93 million dollars is just nuts to me.
[+] [-] anonymouse008|4 years ago|reply
And before anyone goes all ‘but what about Dropbox?’ when scrutinizing this idea... Dropbox was never really made for technical people, this is squarely at people who know what JSON means, so technical.
[+] [-] xyzzy_plugh|4 years ago|reply
I don't understand this comparison. Apollo raised $130M this past summer -- doesn't seem that different to TFA. Is that also nuts to you?
The Protocol Buffers and gRPC ecosystem are also completely free open source libraries. Replace GraphQL with Protobuf and your post is still correct.
[+] [-] webinvest|4 years ago|reply
[+] [-] selfhoster11|4 years ago|reply
[+] [-] gravypod|4 years ago|reply
1. Reduced bandwidth ingress 2. Automatic tracing of PII through your system 3. Developer-controlled ops stuff (annotating an RPC as cachable, etc) 4. Automated tracing instrumentation 5. Message streaming (gRPC streams are amazing)
I can think of a whole host of features that can be built off of protos (I've even built ORMs off of protobuffs for simple things [0]). The value prop is there IMO. HTTP + json APIs are a local minima. The biggest concerns "I want to be able to view the data that is being sent back and forth" is a tooling consideration (curl ... isn't showing you the voltages from the physical layer, it is decoded). Buff is building that tooling.
[0] - https://github.com/CaperAi/pronto
[+] [-] anm89|4 years ago|reply
[+] [-] Guest42|4 years ago|reply
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] pixelgeek|4 years ago|reply
[+] [-] Guest42|4 years ago|reply
[+] [-] duxup|4 years ago|reply
[+] [-] mbrodersen|4 years ago|reply
[+] [-] kentonv|4 years ago|reply
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] bern4444|4 years ago|reply
[deleted]
[+] [-] PhoenixReborn|4 years ago|reply
Article quote:
> We just closed a $68M Series B co-led By Lux and Tiger Global, with participation from Greenoaks Capital Partners, Lightspeed, Addition, and Haystack.
[+] [-] bfung|4 years ago|reply
Good luck to buf.build to sell a revamped wsdl and getting everyone to adopt it.
[+] [-] aogaili|4 years ago|reply
[+] [-] catsarebetter|4 years ago|reply
Also does anyone here use them and have any thoughts about their product?
[+] [-] kentonv|4 years ago|reply
Well, they say the best time to take investment is when you don't need it. If you wait until you need it then the terms will be worse. If investors are offering you money when you don't need it, it may be the best time to accept it.
The sequence of raises here look like a fairly normal sequence for a growing startup, except that they happened much closer together than would be typical. The terms aren't shown but assuming they are in line with a typical sequence then this is a great outcome for buf as it gives them lots of room to build their vision without needing to stress over money for a while.
> Also does anyone here use them and have any thoughts about their product?
FWIW, long ago I was the maintainer of Protobuf at Google, including putting together the first open source release. I like what buf is doing -- enough that, full disclosure, I made an angel investment in their seed round.
There's a huge amount of room for better tooling around Protobuf. Binary and strongly-typed protocols require strong tooling to be usable, but with tooling they can be much better than dynamic and text-based approaches. Like, the fact that the protocol is binary shouldn't make it any harder for a human to read it, because your tools should decode it for you on-demand as easily as you could `cat` a text file. Protobuf historically has had sort of the bare minimum tooling and required a lot of ad hoc copying proto files around between projects to get anywhere, which was a pain. A registry seems like the right first step to making things easier, but I'm really excited about what can be done after that... once you have strong type information, you can have tools to dynamically explore APIs, trace communications, etc.
[+] [-] elzbardico|4 years ago|reply
[+] [-] selfhoster11|4 years ago|reply
Based on their website, they solve the following problems:
- a central schema registery. Even if that's something you actually want, it's not a problem that requires $93M to solve, or a commercial company to operate
- communicating schema changes primarily via human-oriented sources like handwritten documentation on emails. I mean sure, if you are a masochist (or a sufficiently inefficient org), you might do just that. The rest of us here in the 21st century can check the schema into a Git repo instead.
- schema drift on the client end. Tough, this is what happens when you write software. Adding a third party won't help here.
- dependency management. For APIs? I cannot imagine a single case where that would help and your API isn't already a monstrosity.
[+] [-] jokethrowaway|4 years ago|reply
A binary format may save some bandwidth and be slightly harder to reverse engineer - at the cost of being easily introspectable out of the box during development.
I don't think there is enough value to sell something.
I hope it gains traction and cargo cuting companies with bored engineers start using them, so hopefully the next company I work with won't have some terribly complicated and unusable graphql but just protobufs.
[+] [-] ghostwriter|4 years ago|reply
For those who still want / need binary protocols and schemas, look at FlatBuffers or Cap'n Proto instead. At least they are capable of representing domain structures properly.
[+] [-] kentonv|4 years ago|reply
My previous commentary: https://news.ycombinator.com/item?id=18190005
[+] [-] onionisafruit|4 years ago|reply
[+] [-] anm89|4 years ago|reply
[+] [-] rescbr|4 years ago|reply
[+] [-] 4kelly|4 years ago|reply
IMO. Generating gRPC code in multiple languages is pretty tedious setup and maintain. Buf has the potential to replace / free up a lot of time for a small team of people maintaining this sort of thing in house.
- person who helps manage a protobuf monorepo.
[1] https://docs.buf.build/tour/detect-breaking-changes
[+] [-] Hnrobert42|4 years ago|reply
[+] [-] onionisafruit|4 years ago|reply
[+] [-] unknown|4 years ago|reply
[deleted]
[+] [-] OJFord|4 years ago|reply
[+] [-] eps|4 years ago|reply
[+] [-] selfhoster11|4 years ago|reply
[+] [-] duxup|4 years ago|reply