To be honest I kind of find myself drifting away from gRPC/protobuf in my recent projects. I love the idea of an IDL for describing APIs and a great compiler/codegen (protoc) but there's just soo many idiosyncrasies baked into gRPC at this point that it often doesn't feel worth it IMO.
Been increasingly using LSP style JSON-RPC 2.0, sure it's got it's quirks and is far from the most wire/marshaling efficient approach but JSON codecs are ubiquitous and JSON-RPC is trivial to implement. In-fact I recently even wrote a stack allocated, server implementation for microcontrollers in Rust https://github.com/OpenPSG/embedded-jsonrpc.
Varlink (https://varlink.org/) is another interesting approach, there's reasons why they didn't implement the full JSON-RPC spec but their IDL is pretty interesting.
My favorite serde format is Msgpack since it can be dropped in for an almost one-to-one replacement of JSON. There's also CBOR which is based on MsgPack but has diverged a bit and added and a data definition language too (CDDL).
Take JSON-RPC and replace JSON with MsgPack for better handling of integer and float types. MsgPack/CBOR are easy to parse in place directly into stack objects too. It's super fast even on embedded. I've been shipping it for years in embedded projects using a Nim implementation for ESP32s (1) and later made a non-allocating version (2). It's also generally easy to convert MsgPack/CBOR to JSON for debugging, etc.
There's also an IoT focused RPC based on CBOR that's an IETF standard and a time series format (3). The RPC is used a fair bit in some projects.
What I really like about protobuf is the DDL. Really clear schema evolution rules. Ironclad types. Protobuf moves its complexity into things like default zero values, which are irritating but readily apparent. With json, it's superficially fine, but later on you discover that you need to be worrying about implementation-specific stuff like big ints getting mangled, or special parsing logic you need to set default values for string enums so that adding new values doesn't break backwards compatibility. Json-schema exists but really isn't built for these sorts of constraints, and if you try to use json-schema like protobuf, it can get pretty hairy.
Honestly, if protobuf just serialized to a strictly-specified subset of json, I'd be happy with that. I'm not in it for the fast ser/de, and something human-readable could be good. But when multiple services maintained by different teams are passing messages around, a robust schema language is a MASSIVE help. I haven't used Avro, but I assume it's similarly useful.
> I love the idea of an IDL for describing APIs and a great compiler/codegen (protoc)
Me too. My context is that I end up using RPC-ish patterns when doing slightly out-of-the-ordinary web stuff, like websockets, iframe communications, and web workers.
In each of those situations you start with a bidirectional communication channel, but you have to build your own request-response layer if you need that. JSON-RPC is a good place to start, because the spec is basically just "agree to use `id` to match up requests and responses" and very little else of note.
I've been looking around for a "minimum viable IDL" to add to that, and I think my conclusion so far is "just write out a TypeScript file". This works when all my software is web/TypeScript anyway.
Same; at my previous job for the serialisation format for our embedded devices over 2G/4G/LoRaWAN/satellite I ended up landing on MessagePack, but that was partially because the "schema"/typed deserialisation was all in the same language for both the firmware and the server (Nim, in this case) and directly shared source-to-source. That won't work for a lot of cases of course, but it was quite nice for ours!
State of the art for both gzipped json and protobufs is a few GB/s. Details matter (big strings, arrays, and binary data will push protos to 2x-10x faster in typical cases), but it's not the kind of landslide victory you'd get from a proper binary protocol. There isn't much need to feel like you're missing out.
That's sort of where I've landed too. Protobufs would seem to fit the problem area well, but in practice the space between "big-system non-performance-sensitive data transfer metaformat"[1] and "super-performance-sensitive custom binary parser"[2] is... actually really small.
There are just very few spots that actually "need" protobuf at a level of urgency that would justify walking away from self-describing text formats (which is a big, big disadvantage for binary formats!).
[1] Something very well served by JSON
[2] Network routing, stateful packet inspection, on-the-fly transcoding. Stuff that you'd never think to use a "standard format" for.
Apart from being text format, I'm not sure how well JSON-RPC handles doubles vs long integers and other types, where protobuf can be directed to handle them appropriately. That is a problem in JSON itself, so you may neeed to encode some numbers using... "string"
The opaque API brings some niceties that other languages have, specifically about initialization. The Java impl for protobuf will never generate a NullPointerException, as calling `get` on a field would just return the default instance of that field.
The Go OpenAPI did not do this. For many primative types, it was fine. But for protobuf maps, you had to check if the map had been initialized yet in Go code before accessing it. Meaning, with the Opaque API, you can start just adding items to a proto map in Go code without thinking about initialization. (as the Opaque impl will init the map for you).
This is honestly something I wish Go itself would do. Allowing for nil maps in Go is such a footgun.
It's interesting, to everyone but but the mega shops like Google, protobuf is a schema declaration tool. To the megashops its a performance tool.
For most of my projects, I use a web-framework I built on protobuf over the years but slowly got rid of a lot of the protobufy bits (besides the type + method declarations) and just switched to JSON as the wire format. http2, trailing headers, gigantic multi-MB files of getters, setters and embedded binary representations of the schemas, weird import behaviors, no wire error types, etc were too annoying.
Almost every project I've tracked that tries to solve the declarative schema problem seems to slowly die. Its a tough problem an opinionated one (what to do with enums? sum types? defaults? etc). Anyone know of any good ones that are chugging along? OpenAPI is too resty and JSONSchema doesn't seem to care about RPC.
Protobuf 3 was bending over backwards to try to make the Go API make sense, but in the process it screwed up the API for C++, with many compromises. Then they changed course and made presence explicit again in proto 3.1. Now they are saying Go gets a C++-like API.
What I'd like is to rewind the time machine and undo all the path-dependent brain damage.
I work mainly in Python, it's always seemed really bad that there are 3 main implementations of Protobufs, instead of the C++ being the real implementation and other platforms just dlopen'ing and using it (there are a million software engineering arguments around this; I've heard them all before, have my own opinions, and have listened to the opinions of people I disagree with). It seems like the velocity of a project is the reciprocal of the number of independent implementations of a spec because any one of the implementations can slow down all the implementations (like what happened with proto3 around required and optional).
From what I can tell, a major source of the problem was that protobuf field semantics were absolutely critical to the scaling of google in the early days (as an inter-server protocol for rapidly evolving things like the search stack), but it's also being used as a data modelling toolkit (as a way of representing data with a high level of fidelity). And those two groups- along with the multiple language developers who don't want to deal with native code- do not see eye to eye, and want to drive the spec in their preferred direction.
(FWIW nowadays I use pydantic for type descriptions and JSON for transport, but I really prefer having an external IDL unrelated to any specific programming language)
I still use proto2 if possible. The syntactic sugar around `oneof` wasn't nice enough to merit dealing with proto3's implicit presence -- maybe it is just because I learned proto2 with C++ and don't use Go, but proto3 just seemed like a big step back and introduced footguns that weren't there before. Happy to hear they are reverting some of those finally.
Nice to see my comments on their proto3 design doc vindicated, lol. There were a lot of comments on that doc, far more than what you'd usually see. Some of those comments dealt with the misguided decision to basically drop nullability (that is, the `has_` methods) that proto2 had. The team then just deleted all the comments and disabled commenting on the doc and proceeded with their original design much to the consternation of their primary stakeholders.
> syntax = "proto2" uses explicit presence by default
> syntax = "proto3" used implicit presence by default (where cases 2 and 3 cannot be distinguished and are both represented by an empty string), but was later extended to allow opting into explicit presence with the optional keyword
> edition = "2023", the successor to both proto2 and proto3, uses explicit presence by default
The root of the problem seems to be go's zero-values. It's like putting makeup on a pig, your get rid of null-panics, but the null-ish values are still everywhere, you just have bad data creeping into every last corner of your code. There is no amount of validation that can fix the lack of decoding errors. And it's not runtime errors instead of compile-time errors, which can be kept in check with unit tests to some degree. It's just bad data and defaulting to carry on no matter what, like PHP back in the day.
> It's like putting makeup on a pig, your get rid of null-panics
How so? In Go, nil is the zero value for a pointer and is ripe for panic just like null. Zero values do not avoid that problem at all, nor do they intend to.
I don't think the reason for zero values has anything to do with "avoiding null panics". If you want to inline the types, that is avoid using most of your runtime on pointer chasing, you can't universally encode a null value. If I'm unclear, ask yourself: What would a null int look like?
If what you wanted was to avoid null-panics, you can define the elementary operations on null. Generally null has always been defined as aggressively erroring, but there's nothing stopping a language definition from defining propagation rules like for float NaN.
From what I remember, proto3 behavior happened to map to objective c since iOS maps coincidentally happened at around the same time so they could be loud.
It was partially reverted with proto3 optional and fully reverted finally. Go's implementation happened to come around the same time as proto3 so allowed struct access, despite behaving quite differently when accessing nil fields. That is also finally reverted. Hopefully more lessons already learned from the Java days will come sooner than later going forward...
Yes, as much as I love Go and love working with it every day. This inner workings of Go with zero-values has been an design issue that comes up again and again and again.
I hate this API and Go's handling of protocol buffers in general. Especially preparing test data for it makes for some of the most cumbersome and unwieldy files that you will ever come across. Combined with table driven testing you have thousands upon thousands of lines of data with an unbelievably long identifiers that can't be inferred (e.g. in array literals) that is usually copy pasted around and slightly changed. Updating and understanding all of that is a nightmare and if you miss a coma or a brace somewhere, the compiler isn't smart enough to point you to where so you get lines upon lines of syntax errors. But, being opaque has some advantages for sure.
I recently used code-gen'd protobuf deser objects as the value type for an in-memory db and was considering flattening them into a more memory-efficient representation and using bitfields. That was for java though, not sure if they are doing the same thing there
Glad to see this change, for that use case it would've been perfect
Surprisingly I saw this on the front page mere minutes after deciding to use protobufs in my new project.
Currently I'm not quite sold on RPC since the performance benefits seem to show up on a much larger scale than what I am aiming for, so I'm using a proto schema to define my types and using protoc codegen to generate only JSON marshaling/unmarshaling + types for my golang backed and typescript frontend, with JSON transferred between the two using REST endpoints.
Seems to give me good typesafety along with 0 headache in serializing/deserializing after transport.
One thing I also wanted to do was generate SQL schemas from my proto definitions or SQL migrations but haven't found a tool to do so yet, might end up making one.
Would love to know if any HN folk have ideas/critique regarding this approach.
Oh, this is great. I just did an implementation in gRPC in Go whereby I had to churn through 10MB/s of data. I could not implement any kind of memory pool and thus I had a lot of memory allocation issues which lead to bad memory usage and garbage collection eating up my CPU.
why is code generation under-utilized? protobufs and other go tooling are great for code generation. Yet in practice i see few teams using it at scale.
Lots of teams creating rest / json APIs, but very few who use code generation to provide compile-time protection.
The whole FlatBuffers toolchain is wildly immature compared to Protobuf. Last I checked, flatc doesn’t even have a plugin system - all code generators had to be upstreamed into the compiler itself.
Great, now there's an API per struct/message to learn and communicate throughout the codebase, with all the getters and setters.
A given struct is probably faster for protobuf parsing in the new layout, but the complexity of the code probably increases, and I can see this complexity easily negating these gains.
I can’t wait to try this new Protobuf Enterprise Edition, with its sea of getters and setters ad nauseam. /s
However I can get behind it for the lazy decoding which seems nice, though I doubt its actual usefulness for serious software (tm). As someone else already mentioned, an actual serious api (tm) will have business-scope types to uncouple the api definition from the implementation. And that’s how you keep sane as soon as you have to support multiple versions of the api.
Also, a lot of the benefits mentioned for footgun reductions smell like workarounds for the language shortcomings. Memory address comparisons, accidental pointer sharing and mutability, enums, optional handling, etc are already solved problems and where something like rust shines. (Disclaimer: I run several grpc apis written in rust in prod)
The article goes into great detail about the benefits of an opaque api vs open structs. Somewhat unintuitively open structs are not necessarily the “fastest” largely due to pointers requiring heap allocations.
Opaque APIs can also be “faster” due to lazy loading and avoiding memcpy altogether. The latter appears in libraries like flat buffers but not here IIRC.
It's true that this would perform better, and greatly reduce allocations. But:
- Messages (especially opaque ones) are not supposed to be copied. The
recommendation is to use `m := &mypb.Message{}`.
- This would make migrating to use the opaque API more difficult, if the
getters don't return the same type as the old open API fields, much more
code needs to be rewritten, or some wrapper that allocates a new slice on
every get.
- Users expect that `subm := m.GetSubMessages()[2] ;
m.SetSubMessages(append(m.GetSubMessages(), anotherm))) ; subm.SetInt(42) ;
assert(subm.GetInt() == m.GetSubMessages()[2].GetInt())`. This would not be
the case if the API returned a slice of values.
- ...
Effectively, a slice of pointers is baked into the API, and the way people use
protocol buffers in Go. For these reasons, it's not clear to me this would end
up performing better or causing less work.
If we had returned an iterator (new in Go 1.23) instead of an actual slice, then
it would've been possible to vary the underlying representation
(slice-of-pointers, slice-of-value-chunks, ...). But there are other downsides
to that too:
- Allocations when passing iterators to functions that expect a slice.
- Extra API surface for modifying the list (append, getn, len, ...).
Not that clear of a win either.
Another thing that could be considered is: when decoding, allocate a slice of
values ([]mypb.Message), *and* a slice with pointers (or do it lazily):
[]*mypb.Message. Then initialize:
for i := range valuel {
ptrl[i] = &valuel[i] // TODO: verify that this escape doesn't cause disjoint allocations.
}
That might be beneficial due to grouping allocations, and the user would be none
the wiser.
[+] [-] dpeckett|1 year ago|reply
Been increasingly using LSP style JSON-RPC 2.0, sure it's got it's quirks and is far from the most wire/marshaling efficient approach but JSON codecs are ubiquitous and JSON-RPC is trivial to implement. In-fact I recently even wrote a stack allocated, server implementation for microcontrollers in Rust https://github.com/OpenPSG/embedded-jsonrpc.
Varlink (https://varlink.org/) is another interesting approach, there's reasons why they didn't implement the full JSON-RPC spec but their IDL is pretty interesting.
[+] [-] elcritch|1 year ago|reply
Take JSON-RPC and replace JSON with MsgPack for better handling of integer and float types. MsgPack/CBOR are easy to parse in place directly into stack objects too. It's super fast even on embedded. I've been shipping it for years in embedded projects using a Nim implementation for ESP32s (1) and later made a non-allocating version (2). It's also generally easy to convert MsgPack/CBOR to JSON for debugging, etc.
There's also an IoT focused RPC based on CBOR that's an IETF standard and a time series format (3). The RPC is used a fair bit in some projects.
1: https://github.com/elcritch/nesper/blob/devel/src/nesper/ser... 2: https://github.com/EmbeddedNim/fastrpc 3: https://hal.science/hal-03800577v1/file/Towards_a_Standard_T...
[+] [-] bccdee|1 year ago|reply
Honestly, if protobuf just serialized to a strictly-specified subset of json, I'd be happy with that. I'm not in it for the fast ser/de, and something human-readable could be good. But when multiple services maintained by different teams are passing messages around, a robust schema language is a MASSIVE help. I haven't used Avro, but I assume it's similarly useful.
[+] [-] perezd|1 year ago|reply
[+] [-] crabmusket|1 year ago|reply
Me too. My context is that I end up using RPC-ish patterns when doing slightly out-of-the-ordinary web stuff, like websockets, iframe communications, and web workers.
In each of those situations you start with a bidirectional communication channel, but you have to build your own request-response layer if you need that. JSON-RPC is a good place to start, because the spec is basically just "agree to use `id` to match up requests and responses" and very little else of note.
I've been looking around for a "minimum viable IDL" to add to that, and I think my conclusion so far is "just write out a TypeScript file". This works when all my software is web/TypeScript anyway.
[+] [-] girvo|1 year ago|reply
[+] [-] hansvm|1 year ago|reply
State of the art for both gzipped json and protobufs is a few GB/s. Details matter (big strings, arrays, and binary data will push protos to 2x-10x faster in typical cases), but it's not the kind of landslide victory you'd get from a proper binary protocol. There isn't much need to feel like you're missing out.
[+] [-] ajross|1 year ago|reply
There are just very few spots that actually "need" protobuf at a level of urgency that would justify walking away from self-describing text formats (which is a big, big disadvantage for binary formats!).
[1] Something very well served by JSON
[2] Network routing, stateful packet inspection, on-the-fly transcoding. Stuff that you'd never think to use a "standard format" for.
[+] [-] malkia|1 year ago|reply
[+] [-] mirekrusin|1 year ago|reply
[+] [-] kyrra|1 year ago|reply
The Go OpenAPI did not do this. For many primative types, it was fine. But for protobuf maps, you had to check if the map had been initialized yet in Go code before accessing it. Meaning, with the Opaque API, you can start just adding items to a proto map in Go code without thinking about initialization. (as the Opaque impl will init the map for you).
This is honestly something I wish Go itself would do. Allowing for nil maps in Go is such a footgun.
[+] [-] parhamn|1 year ago|reply
For most of my projects, I use a web-framework I built on protobuf over the years but slowly got rid of a lot of the protobufy bits (besides the type + method declarations) and just switched to JSON as the wire format. http2, trailing headers, gigantic multi-MB files of getters, setters and embedded binary representations of the schemas, weird import behaviors, no wire error types, etc were too annoying.
Almost every project I've tracked that tries to solve the declarative schema problem seems to slowly die. Its a tough problem an opinionated one (what to do with enums? sum types? defaults? etc). Anyone know of any good ones that are chugging along? OpenAPI is too resty and JSONSchema doesn't seem to care about RPC.
[+] [-] jeffbee|1 year ago|reply
What I'd like is to rewind the time machine and undo all the path-dependent brain damage.
[+] [-] sa46|1 year ago|reply
Is there a design doc with the rationale for switching back to explicit presence for Edition 2023?
The closest docs I've found are https://buf.build/blog/protobuf-editions-are-here and https://github.com/protocolbuffers/protobuf/tree/main/docs/d....
[+] [-] jcdavis|1 year ago|reply
The implicit presence garbage screwed up the API for many languages, not just C++
What is wild is how obviously silly it was at the time, too - no hindsight was needed.
[+] [-] dekhn|1 year ago|reply
From what I can tell, a major source of the problem was that protobuf field semantics were absolutely critical to the scaling of google in the early days (as an inter-server protocol for rapidly evolving things like the search stack), but it's also being used as a data modelling toolkit (as a way of representing data with a high level of fidelity). And those two groups- along with the multiple language developers who don't want to deal with native code- do not see eye to eye, and want to drive the spec in their preferred direction.
(FWIW nowadays I use pydantic for type descriptions and JSON for transport, but I really prefer having an external IDL unrelated to any specific programming language)
[+] [-] sbrother|1 year ago|reply
[+] [-] ein0p|1 year ago|reply
[+] [-] boulos|1 year ago|reply
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] the_gipsy|1 year ago|reply
> syntax = "proto3" used implicit presence by default (where cases 2 and 3 cannot be distinguished and are both represented by an empty string), but was later extended to allow opting into explicit presence with the optional keyword
> edition = "2023", the successor to both proto2 and proto3, uses explicit presence by default
The root of the problem seems to be go's zero-values. It's like putting makeup on a pig, your get rid of null-panics, but the null-ish values are still everywhere, you just have bad data creeping into every last corner of your code. There is no amount of validation that can fix the lack of decoding errors. And it's not runtime errors instead of compile-time errors, which can be kept in check with unit tests to some degree. It's just bad data and defaulting to carry on no matter what, like PHP back in the day.
[+] [-] 9rx|1 year ago|reply
How so? In Go, nil is the zero value for a pointer and is ripe for panic just like null. Zero values do not avoid that problem at all, nor do they intend to.
[+] [-] delusional|1 year ago|reply
If what you wanted was to avoid null-panics, you can define the elementary operations on null. Generally null has always been defined as aggressively erroring, but there's nothing stopping a language definition from defining propagation rules like for float NaN.
[+] [-] tech132|1 year ago|reply
It was partially reverted with proto3 optional and fully reverted finally. Go's implementation happened to come around the same time as proto3 so allowed struct access, despite behaving quite differently when accessing nil fields. That is also finally reverted. Hopefully more lessons already learned from the Java days will come sooner than later going forward...
[+] [-] knodi|1 year ago|reply
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] kubb|1 year ago|reply
[+] [-] remram|1 year ago|reply
I call this Battlefield versioning, after the Battlefield video game series [1]. I bet the next version will be proto V.
[1]: in order: 1942, 2, 2142, 3, 4, 1, V, 2042
[+] [-] matrix87|1 year ago|reply
Glad to see this change, for that use case it would've been perfect
[+] [-] fofoz|1 year ago|reply
[+] [-] h4ch1|1 year ago|reply
Currently I'm not quite sold on RPC since the performance benefits seem to show up on a much larger scale than what I am aiming for, so I'm using a proto schema to define my types and using protoc codegen to generate only JSON marshaling/unmarshaling + types for my golang backed and typescript frontend, with JSON transferred between the two using REST endpoints.
Seems to give me good typesafety along with 0 headache in serializing/deserializing after transport.
One thing I also wanted to do was generate SQL schemas from my proto definitions or SQL migrations but haven't found a tool to do so yet, might end up making one.
Would love to know if any HN folk have ideas/critique regarding this approach.
[+] [-] favflam|1 year ago|reply
[+] [-] alecthomas|1 year ago|reply
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] cyberax|1 year ago|reply
Now you can not use normal Go struct initialization and you'll have to write reams of Set calls.
[+] [-] tonymet|1 year ago|reply
Lots of teams creating rest / json APIs, but very few who use code generation to provide compile-time protection.
[+] [-] alakra|1 year ago|reply
[+] [-] mort96|1 year ago|reply
[+] [-] kyrra|1 year ago|reply
[+] [-] neonsunset|1 year ago|reply
[+] [-] aktau|1 year ago|reply
[+] [-] g0ld3nrati0|1 year ago|reply
[+] [-] akshayshah|1 year ago|reply
[+] [-] strawhatguy|1 year ago|reply
A given struct is probably faster for protobuf parsing in the new layout, but the complexity of the code probably increases, and I can see this complexity easily negating these gains.
[+] [-] tuetuopay|1 year ago|reply
However I can get behind it for the lazy decoding which seems nice, though I doubt its actual usefulness for serious software (tm). As someone else already mentioned, an actual serious api (tm) will have business-scope types to uncouple the api definition from the implementation. And that’s how you keep sane as soon as you have to support multiple versions of the api.
Also, a lot of the benefits mentioned for footgun reductions smell like workarounds for the language shortcomings. Memory address comparisons, accidental pointer sharing and mutability, enums, optional handling, etc are already solved problems and where something like rust shines. (Disclaimer: I run several grpc apis written in rust in prod)
[+] [-] Naru41|1 year ago|reply
[+] [-] schmichael|1 year ago|reply
[+] [-] akira2501|1 year ago|reply
To bake endianess and alignment requirements into your protocol.
[+] [-] cyberax|1 year ago|reply
This one small change can result in an order of magnitude improvement.
[+] [-] aktau|1 year ago|reply
If we had returned an iterator (new in Go 1.23) instead of an actual slice, then it would've been possible to vary the underlying representation (slice-of-pointers, slice-of-value-chunks, ...). But there are other downsides to that too:
Not that clear of a win either.Another thing that could be considered is: when decoding, allocate a slice of values ([]mypb.Message), *and* a slice with pointers (or do it lazily): []*mypb.Message. Then initialize:
That might be beneficial due to grouping allocations, and the user would be none the wiser.