Pagination is one of those things I feel like so many of these things get wrong. LIMIT/OFFSET (or as MS likes to call it, TOP/SEEK) style results in O(n²) operations; an automated tool trying to pull the entirety of a large collection in such a scenario is not good. I have to again recommend the excellent "Pagination done the Right Way" presentation[1] from Use the Index, Luke (an equally excellent site).
Just return a link to the "next page" (make the next page opaque); while this removes the ability to of the client to go to an arbitrary page on its own, in practice, I've never seen that matter, and the generalized URL format allows you to seek instead of by LIMIT/OFFSET by something indexable by the DB; HTTP even includes a standardized header for just this purpose[2].
I also think the generalized URL is more in line with Fielding's definition of REST, in that the response links to another resource. (I don't know if being in the header invalidates this; to me, it does not.)
If you get the metadata out of the data section of your response, and move it to the headers, where it belongs, this usually then lets you keep the "collection" of items you are return as an array (because if you need to put metadata along side it, you need:
["the", "data", "you", "wanted", "in a sensible format"]
)
(I've seen metadata-pushed-into-the-body turn API endpoints that literally need to return no more than "true" or "false" into object that then force a client to know and look up the correct key in an object… sigh.)
From the parent Microsoft guideline [1], they make the explicit differentiation between Server-driven and Client-driven paging, the Server-driven seems to be exactly what you describe in your first paragraph:
> Paginated responses MUST indicate a partial result by including a continuation token in the response. The absence of a continuation token means that no additional pages are available.
> Clients MUST treat the continuation URL as opaque, which means that query options may not be changed while iterating over a set of partial results.
They then explain Client-driven paging, if the client really wants to do it:
> Clients MAY use $top and $skip query parameters to specify a number of results to return and an offset. The server SHOULD honor the values specified by the client; however, clients MUST be prepared to handle responses that contain a different page size or contain a continuation token.
But if you want to obtain the entire collection, you would not use Client-driven paging, but only rely on Server-driven paging
Big +1 to killing offset in favor of order-by and an "after" or "before" criteria.
However, -1 to the idea of sticking any important "metadata" in headers. For one, it's dramatically easier to interact with response bodies than headers at the terminal or with simple programming libraries and tools. Having only one way of representing data requires fewer special cases. JSON is far superior to key/value pairs in terms of ability to represent structure. Lastly, if you ever decide to use a different "transport" protocol other than HTTP, you'll be glad you are already using a common representation without a side channel.
Making the next page opaque as you said is a good thing also because limit and offset will often become unstable or incorrect if the dataset is changing without a proper timestamp. That is you may get more or less records then you are expecting which maybe important if you need to allocate size.
Almost all of my rest services I require sending a timestamp for batch processing (that is only records created/updated/removed before given timestamp are shown).
Microsoft came up with OData about 10 years ago, but it's now steered by a technical committee with a lot of participants. I wouldn't necessarily be so quick to assume that the OASIS committee - or Microsoft's representatives on it - speak for all of Microsoft. Let alone an internal guidelines committee that they apparently convened fairly recently (given the age of the report) when they use that slogan.
OData is a big enough tent these days that there are good and bad (relative) bits even inside OData. Fortunately one gets to pick-and-choose what bits you use, so just avoid the bad bits.
The example cited as a bad url makes use of OData functions and parameters, which is definitely a more esoteric part of the spec and has spotty implementation (if at all) amongst OData consumers - so discouraging this kind of API seems perfectly reasonable for a REST-centric guideline.
OTOH the OData query syntax is IMO a lot more reasonable; outside of structured queries built in the form of URI path fragments, if you want to provide a generic query mechanism on top of a store you need some kind of generic query language. $filter is a reasonable such language - it is easy to parse, easy to emit, and relatively easy to read. Yes it has some gaps and a couple bizarre edge cases, but they don't get in the way of mainline consumption scenarios - and it's hard to beat being able to provide a reasonable REST API that clients can construct queries by hand for, and also have these same APIs "just work" when plugged into OData consumers (of which there are quite a few in the enterprise).
That particular example was chosen not because of OData, but because it required someone to type in a horribly long set of alphanumeric codes.
There's no way a human can effectively type that, nor is there any real way to look at it and figure out what it's doing. That the URL happens to use ODAta (because that's what Exchange does) is not really relevant.
The overriding factor, as called out in the doc, is that URLs should be human understandable. That's not to say canonical id's can't be used, but a big base64 encoded string is not recommended.
I'd love to see some guidance on how you're supposed to do file uploads (think: uploading an image avatar for a user) and fit it into the REST structure. These guidelines don't even mention "x-www-form-urlencoded" except in a bit of fluff about CORS.
Made more frustrating by Microsoft's own WebAPI2 basically having zero support for file uploads, meaning we had to go way outside the library to code support for it.
Not sure why that's such a blind spot. Doesn't every web service need to upload files eventually?
It turns out that uploading files is hard. Files come in a variety of sizes, networks sometimes don't work, browser processes come-and-go, some users expect restart semantics, some don't...
Well presented. It would be great if there was a language / framework that made this guaranteed. As-is everything just returns 500 error on any exception, lets you return errors with 200, allows update on GET, etc. Even the Haskell ones.
Handling errors over REST API is something I've struggled with. What's the best way to handle errors? Data validation errors will be different from system/server errors. Tough to establish a universally applicable error response structure.
I used to be in favor of sending 200 responses with error codes but now gravitating back towards relaying the most relevant HTTP error & letting the clients handle it.
It's great to see Microsoft release these guidelines. It's good work, a broad document with a lot of interesting topics covered. You can always debate the details (the pagination discussion here is very interesting), but having seen first hand at Zalando how much dedication and effort goes into API guidelines to support a large number of service teams, plus releasing them into the public, it's no small feat and they deserve credit for doing this.
There's naturally been some discussion around REST/HTTP APIs and design styles. One of the things we've tried to do with the Zalando guidelines (https://zalando.github.io/restful-api-guidelines) is go into more detail on using HTTP and JSON, and how REST properties are used. Zalando, like everyone else, has had to think through versioning and compatibility, so it was interesting to read what Microsoft do here. The Zalando guidelines take a different approach, using media types for versioning, document just one set of compatibility rules, plus a deprecation model, and it's working very well in practice so far (http://zalando.github.io/restful-api-guidelines/compatibilit...).
Btw, in case anyone from Microsoft working on the guidelines is reading and ever wanted to swap guideline notes or ideas, that would be awesome. And once again, great job releasing the doc :)
In Kibana, everything is done over GET as get parameters, and I find that extremely annoying and a poor design.
A lot of public APIs also don't honor or have any intentions in supporting or using PATCH. Most APIs I have worked with only use PUT for modification. Anything resembles "creation" is automatically a POST.
I have worked with some systems that do not expose all http methods. Having a fallback way to invoke a method as a GET or POST makes your API more usable to more clients.
Very cool document. I kind of got stuck at delta queries, though. How do you implement that? I can't find any reference to delta/removed queries on Mongo, Postgres, or MySQL. Do you just keep all records in the database and add a "removed" field? How would that solution work with user privacy & users truly wanting to remove data?
Funny thing: I've been thinking a bit about API versioning quite a bit lately, and the best solution I've come up with is the ONE thing not at all covered in this: put an `api-version` header into the request. I've seen both of the schemes recommended here, and I like neither very much. So what's wrong with my (header) solution?
For instance, if client application A talks to the service using version 1.0 of the API and client application B talks to the service using version 2.0 of the API, then those client applications can't interoperate because they are seeing two different sets of resources.
Your solution isn't far off the approach recommended by everybody who rejects the different URI approach. You don't need an API version header. When your representation format for a resource changes, provide a `version` parameter for the media type. For example: `Content-Type: application/vnd.example.foo+json;version=2`.
This is exactly how HTTP and the `Content-Type` header are supposed to work – if your representation for a resource changes, you don't change the URI, you change the media type.
1. Older proxies used to remove/strip headers they don't understand.
2. Frameworks and libraries don't always give access to n non-standard headers, meaning they just can't used.
3. It's harder for humans to look at a request and see what's going on.
An API version in an URI is fine. But better would be to use a new domain, if the API brakes backward compatibility entirely.
If the API does not break the clients entirely, just have new resources for incompatible formats:
/user
/user2
For the first resource, you can just add a "X-Deprecated" header with a link to the deprecation information. If you already know the date when the resource is abandoned, you can also add a header that has a date of removal in it. If removed, you can either have /user serve the new format or have it serve a HTTP 303 to the new format.
Being that guy again, (and sacrificing my karma) but...
This is not REST, it contains nothing about hypermedia, entities having knowledge of their URIs, or any way of discovering features of an API programmatically.
While I'm sure there's plenty of good stuff in here (it looks otherwise fairly comprehensive), APIs will continue to be a disparate system that requires custom code for every integration until we can embrace the full benefits of REST, and develop the technology and conventions/standards for integrating RESTful services.
Edit: for an example of what's possible with 'real' REST, check out this talk: https://vimeo.com/20781278 – better documentation, easy parsing, API optimisation without client changes, discovery, and the API is fully human readable/usable as well.
Not sure about your karma sacrifice, but if someone has doubts:
> What needs to be done to make the REST architectural style clear on the
> notion that hypertext is a constraint? In other words, if the engine of
> application state (and hence the API) is not being driven by hypertext,
> then it cannot be RESTful and cannot be a REST API. Period. Is there
> some broken manual somewhere that needs to be fixed?
REST seems like an elephant in a room with some blind folks. Everybody who touches it thinks it's something different than the next guy, and they're all describing only one element of the thing.
That said, I don't actually know what this elephant looks like, either, because everybody I've read on the subject seems to have only a partial understanding of it...thus, I have a partial understanding of it.
If you know what the whole elephant looks like, and have good resources for what makes something actually REST, I'd be interested.
> or any way of discovering features of an API programmatically.
So, this is one of my issues with HATEOAS. When does this _actually_ come up in practice? I mean really, how do we normally consume APIs? We read the docs to figure out which endpoints we need to call, call them in the right order, and do the right stuff with the result. Nothing about that process is automatic; it takes a human to figure out what's available and the right way to use it.
To me the hypermedia part of REST has always seemed like a pointless academic exercise. Yes it's really cool to be able to point a service browser at your service's root resource and have it map the whole thing out but in practice, when is this ever anything but a neat party trick?
I don't know why everyone wants to downvote this comment. It's absolutely right.
I would add that REST has nothing to do with pretty URIs, JSON formatting, versioning, query strings, and more misconceptions that people seem to mistake for the architectural style. There is nothing that isn't RESTful with URIs like "/a89f30e9" instead of "/:type/:id". Even TBL thinks URIs should be opaque: https://www.w3.org/DesignIssues/Axioms.html#opaque
To be frank, main issue is people are arguing about semantics, one side thinks Fielding's definition is the only thing that can be called REST, and the other larger(?) side could care less about that academic definition and are describing some simple rules for HTTP APIs. While it would be nice if there were two terms, just saying "This is NOT REST" really doesn't clarify anything.
I'm fine with APIs continuing to be disparate systems if they have a simple, sensible design and have comprehensive documentation. Let's just stop calling them "REST" since they're not as you've pointed out.
No worries. There are people out there, like me, who agree with you. You have my vote on this.
Lets say Microsoft at least created their own format, that allows some discovery. That is a good approach, because a big upfront design can be harmful. There are few decisions in HTML you can take as an example.
I am currently working on having a format for my company, that is based on Siren. Its no easy thing to do, because I want it to be easy to be used (which is not simple).
Otherwise, what I've read so far looks like a really good start. Say what one will about Microsoft's products, but there are a lot of smart folks there.
[+] [-] deathanatos|9 years ago|reply
Just return a link to the "next page" (make the next page opaque); while this removes the ability to of the client to go to an arbitrary page on its own, in practice, I've never seen that matter, and the generalized URL format allows you to seek instead of by LIMIT/OFFSET by something indexable by the DB; HTTP even includes a standardized header for just this purpose[2].
I also think the generalized URL is more in line with Fielding's definition of REST, in that the response links to another resource. (I don't know if being in the header invalidates this; to me, it does not.)
If you get the metadata out of the data section of your response, and move it to the headers, where it belongs, this usually then lets you keep the "collection" of items you are return as an array (because if you need to put metadata along side it, you need:
vs. )(I've seen metadata-pushed-into-the-body turn API endpoints that literally need to return no more than "true" or "false" into object that then force a client to know and look up the correct key in an object… sigh.)
[1]: http://use-the-index-luke.com/no-offset
[2]: https://tools.ietf.org/html/rfc5988
[+] [-] vmarsy|9 years ago|reply
> Paginated responses MUST indicate a partial result by including a continuation token in the response. The absence of a continuation token means that no additional pages are available.
> Clients MUST treat the continuation URL as opaque, which means that query options may not be changed while iterating over a set of partial results.
> Example:
They then explain Client-driven paging, if the client really wants to do it:> Clients MAY use $top and $skip query parameters to specify a number of results to return and an offset. The server SHOULD honor the values specified by the client; however, clients MUST be prepared to handle responses that contain a different page size or contain a continuation token.
But if you want to obtain the entire collection, you would not use Client-driven paging, but only rely on Server-driven paging
[1] https://github.com/Microsoft/api-guidelines/blob/master/Guid...
[+] [-] brandonbloom|9 years ago|reply
However, -1 to the idea of sticking any important "metadata" in headers. For one, it's dramatically easier to interact with response bodies than headers at the terminal or with simple programming libraries and tools. Having only one way of representing data requires fewer special cases. JSON is far superior to key/value pairs in terms of ability to represent structure. Lastly, if you ever decide to use a different "transport" protocol other than HTTP, you'll be glad you are already using a common representation without a side channel.
[+] [-] foota|9 years ago|reply
(edit: may be mitigated in newer browsers)
[+] [-] agentgt|9 years ago|reply
Almost all of my rest services I require sending a timestamp for batch processing (that is only records created/updated/removed before given timestamp are shown).
[+] [-] draw_down|9 years ago|reply
Yeah, that's the whole problem with this approach.
[+] [-] mythz|9 years ago|reply
https://github.com/Microsoft/api-guidelines/blob/master/Guid...
Whilst continuing to praise OData as "the best way to REST" - http://www.odata.org
[+] [-] bunderbunder|9 years ago|reply
[+] [-] pdrayton|9 years ago|reply
The example cited as a bad url makes use of OData functions and parameters, which is definitely a more esoteric part of the spec and has spotty implementation (if at all) amongst OData consumers - so discouraging this kind of API seems perfectly reasonable for a REST-centric guideline.
OTOH the OData query syntax is IMO a lot more reasonable; outside of structured queries built in the form of URI path fragments, if you want to provide a generic query mechanism on top of a store you need some kind of generic query language. $filter is a reasonable such language - it is easy to parse, easy to emit, and relatively easy to read. Yes it has some gaps and a couple bizarre edge cases, but they don't get in the way of mainline consumption scenarios - and it's hard to beat being able to provide a reasonable REST API that clients can construct queries by hand for, and also have these same APIs "just work" when plugged into OData consumers (of which there are quite a few in the enterprise).
[+] [-] ChrisAtWork|9 years ago|reply
There's no way a human can effectively type that, nor is there any real way to look at it and figure out what it's doing. That the URL happens to use ODAta (because that's what Exchange does) is not really relevant.
The overriding factor, as called out in the doc, is that URLs should be human understandable. That's not to say canonical id's can't be used, but a big base64 encoded string is not recommended.
[+] [-] erjjones|9 years ago|reply
[+] [-] blakeyrat|9 years ago|reply
Made more frustrating by Microsoft's own WebAPI2 basically having zero support for file uploads, meaning we had to go way outside the library to code support for it.
Not sure why that's such a blind spot. Doesn't every web service need to upload files eventually?
[+] [-] ChrisAtWork|9 years ago|reply
[+] [-] gedy|9 years ago|reply
?
What additional guidance would you expect?
[+] [-] beachy|9 years ago|reply
[+] [-] daxfohl|9 years ago|reply
[+] [-] tarequeh|9 years ago|reply
I used to be in favor of sending 200 responses with error codes but now gravitating back towards relaying the most relevant HTTP error & letting the clients handle it.
[+] [-] ZalandoTech|9 years ago|reply
[+] [-] dehora|9 years ago|reply
There's naturally been some discussion around REST/HTTP APIs and design styles. One of the things we've tried to do with the Zalando guidelines (https://zalando.github.io/restful-api-guidelines) is go into more detail on using HTTP and JSON, and how REST properties are used. Zalando, like everyone else, has had to think through versioning and compatibility, so it was interesting to read what Microsoft do here. The Zalando guidelines take a different approach, using media types for versioning, document just one set of compatibility rules, plus a deprecation model, and it's working very well in practice so far (http://zalando.github.io/restful-api-guidelines/compatibilit...).
Btw, in case anyone from Microsoft working on the guidelines is reading and ever wanted to swap guideline notes or ideas, that would be awesome. And once again, great job releasing the doc :)
[+] [-] yeukhon|9 years ago|reply
This discuss some option for the GET vs POST.
In Kibana, everything is done over GET as get parameters, and I find that extremely annoying and a poor design.
A lot of public APIs also don't honor or have any intentions in supporting or using PATCH. Most APIs I have worked with only use PUT for modification. Anything resembles "creation" is automatically a POST.
[+] [-] dozzie|9 years ago|reply
Kibana (4.x) being a website to display queries and some charts and weighing over hundred megabytes is itself a clear example of a poor design.
[+] [-] ams6110|9 years ago|reply
[+] [-] balls187|9 years ago|reply
[+] [-] iheart2code|9 years ago|reply
[+] [-] captn3m0|9 years ago|reply
Nice to see them support such stuff, still.
[+] [-] spdustin|9 years ago|reply
[0]: http://www.odata.org
[+] [-] mikro2nd|9 years ago|reply
[+] [-] JimDabell|9 years ago|reply
For instance, if client application A talks to the service using version 1.0 of the API and client application B talks to the service using version 2.0 of the API, then those client applications can't interoperate because they are seeing two different sets of resources.
Your solution isn't far off the approach recommended by everybody who rejects the different URI approach. You don't need an API version header. When your representation format for a resource changes, provide a `version` parameter for the media type. For example: `Content-Type: application/vnd.example.foo+json;version=2`.
This is exactly how HTTP and the `Content-Type` header are supposed to work – if your representation for a resource changes, you don't change the URI, you change the media type.
[+] [-] beckler|9 years ago|reply
https://www.troyhunt.com/your-api-versioning-is-wrong-which-...
[+] [-] ChrisAtWork|9 years ago|reply
1. Older proxies used to remove/strip headers they don't understand. 2. Frameworks and libraries don't always give access to n non-standard headers, meaning they just can't used. 3. It's harder for humans to look at a request and see what's going on.
[+] [-] einrealist|9 years ago|reply
If the API does not break the clients entirely, just have new resources for incompatible formats:
For the first resource, you can just add a "X-Deprecated" header with a link to the deprecation information. If you already know the date when the resource is abandoned, you can also add a header that has a date of removal in it. If removed, you can either have /user serve the new format or have it serve a HTTP 303 to the new format.[+] [-] aligajani|9 years ago|reply
[+] [-] fooyc|9 years ago|reply
People following this also tend to follow it like a dogme, and find that their APIs are slow, too meta/abstract, and hard to consume.
[+] [-] danpalmer|9 years ago|reply
This is not REST, it contains nothing about hypermedia, entities having knowledge of their URIs, or any way of discovering features of an API programmatically.
While I'm sure there's plenty of good stuff in here (it looks otherwise fairly comprehensive), APIs will continue to be a disparate system that requires custom code for every integration until we can embrace the full benefits of REST, and develop the technology and conventions/standards for integrating RESTful services.
Edit: for an example of what's possible with 'real' REST, check out this talk: https://vimeo.com/20781278 – better documentation, easy parsing, API optimisation without client changes, discovery, and the API is fully human readable/usable as well.
[+] [-] rimantas|9 years ago|reply
So if it is not HATEOAS it is not REST. Alas, I am not aware about any term which can be used for these Richardson maturity model Level 2 services.
[+] [-] SwellJoe|9 years ago|reply
That said, I don't actually know what this elephant looks like, either, because everybody I've read on the subject seems to have only a partial understanding of it...thus, I have a partial understanding of it.
If you know what the whole elephant looks like, and have good resources for what makes something actually REST, I'd be interested.
[+] [-] xienze|9 years ago|reply
So, this is one of my issues with HATEOAS. When does this _actually_ come up in practice? I mean really, how do we normally consume APIs? We read the docs to figure out which endpoints we need to call, call them in the right order, and do the right stuff with the result. Nothing about that process is automatic; it takes a human to figure out what's available and the right way to use it.
To me the hypermedia part of REST has always seemed like a pointless academic exercise. Yes it's really cool to be able to point a service browser at your service's root resource and have it map the whole thing out but in practice, when is this ever anything but a neat party trick?
[+] [-] daliwali|9 years ago|reply
I would add that REST has nothing to do with pretty URIs, JSON formatting, versioning, query strings, and more misconceptions that people seem to mistake for the architectural style. There is nothing that isn't RESTful with URIs like "/a89f30e9" instead of "/:type/:id". Even TBL thinks URIs should be opaque: https://www.w3.org/DesignIssues/Axioms.html#opaque
[+] [-] gedy|9 years ago|reply
[+] [-] varenc|9 years ago|reply
[+] [-] einrealist|9 years ago|reply
Lets say Microsoft at least created their own format, that allows some discovery. That is a good approach, because a big upfront design can be harmful. There are few decisions in HTML you can take as an example.
I am currently working on having a format for my company, that is based on Siren. Its no easy thing to do, because I want it to be easy to be used (which is not simple).
[+] [-] ihsw|9 years ago|reply
Frankly I would prefer adhering to this standard.
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] pstuart|9 years ago|reply
[+] [-] raz32dust|9 years ago|reply
[+] [-] zeveb|9 years ago|reply
Otherwise, what I've read so far looks like a really good start. Say what one will about Microsoft's products, but there are a lot of smart folks there.
[+] [-] mianos|9 years ago|reply
[+] [-] horacios|9 years ago|reply
[deleted]
[+] [-] intellix2|9 years ago|reply
[+] [-] intellix2|9 years ago|reply