Hyper-Schema and non-HTTP-compliant APIs

[handrews]

HTTP APIs are not necessarily RESTful, and do not necessarily strictly adhere to the official HTTP semantics (most of which are laid out in RFC 7231).

An example is Elasticsearch's use of a request body with GET, where they gleefully interpret the following passage from RFC 7231:

A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request.

to mean "do whatever you want!" which is definitely not what "no defined semantics" is intended to convey. Other examples include many APIs which wrap resource representations in status envelopes in GET responses. The analogous PUT (and other) requests may or may not be wrapped similarly (in the linked API they are not wrapped).

This goes all they way down to APIs that just use POST for everything and always return HTTP 200 even if they have an error.

This topic has come up before in various forms, here is a summary of past discussions:

---

In PR #179 @Relequestual asserted that we should support these APIs with hyper-schema.

[Only supporting RESTful APIs] would be limiting. [GET with a request body] is not an invalid use of HTTP, it's just not recomended, but there may be valid reasons for doing so.

---

In issue #88 I had previously asked about this and @awwright commented:

If people can't be bothered to use HTTP correctly in the first place, for what reason would they potentially end up using the hypermedia functionality of JSON Schema?

(then we wandered off into a discussion of JSON Home which is irrelevant, which is why I'm not re-opening that issue).

---

In issue #107 I asked about response envelopes. Despite also trying to get discussion going by posting about it to the google group, I got zero meaningful responses on the topic. I couldn't figure out a good solution anyway, and #88 had seemed to indicate this shouldn't be supported, so I gave up and closed it.

---

We need to decide this one way or the other. It has massive implications for what is and isn't out of scope for JSON Hyper-schema.

In particular, "service definition" formats like Swagger/OpenAPI are well-suited to non-RESTful HTTP APIs because they describe every aspect of interaction.

Hyper-schema is not a service definition format, and I strongly believe that it should not become such a thing. While having keywords to help identify and describe HTTP (or other protocol) mis-use would allow very valuable use with a broader range of APIs, I would hate to lose the simplicity that is enabled by proper protocol usage.

Ideally, any features added to support protocol mis-use would be ignorable by hyper-schema authors writing hyper-schemas for properly RESTful APIs.

[donaldpipowitch]

Could you explain what is a service definition format?

[handrews]

@donaldpipowitch it's possible I'm abusing the term :-)

What I mean is something like OpenAPI which intends to exhaustively document every aspect of each resource and the requests/responses defined for it. This includes error responses, headers, auth, etc. etc.

Service definitions usually do not seem to include information needed for HATEOAS (based on the last time I checked, anyway). The philosophy seems to be to document every detail of individual resource interaction, and assume humans will make the necessary connections (filling out documented templates with values, rather than enabling the resources to produce the links automatically).

Since most HTTP APIs are not properly RESTful, this approach is popular. In a RESTful API, for instance, once you've documented the representation, you don't need to explicitly document the GET response or PUT request. Those are representations, and you already documented the representation. At most, you just need to document that GET and PUT are supported.

If your PUT response has a body and sets the Content-Location header to the request URI, then you don't need to document that either- per the HTTP spec, that is also a representation (specifically, it's exactly what you would have gotten from a GET made at that time). You can just state that you use Content-Location to return an updated representation on successful PUT.

You shouldn't need to document what a PATCH request looks like. Just note what media type PATCH expects as a request (it should be set in the Accept-Patch header anyway). That media type plus the representation tells you everything you need to know to construct a request.

Most HTTP response codes should not need explicit documentation. Seriously, what is there to say about a 404? If you use a media type like application/problem+json for error response bodies, then you can get away with even less documentation, because the media type defines the format for you.

But most APIs don't do things like use Content-Location properly. They don't use a standard media type for error responses. They implement PATCH requests as plain old application/json which makes no sense, and don't set Accept-Patch anyway. Some wrap extra information around responses, so GET doesn't return an actual representation. So since they ignore most of the HTTP and media type features that are designed to let you not need documentation, they have to document everything.

Which produces a giant pile of documentation that no one likes to read (and therefore it's often ignored or at best skimmed).

I'm waving my hands here a lot. There are things that a good documentation package should have that JSON Hyper-Schema is unlikely to provide. I just think that service definitions and the documentation that they produce go too far into the details which encourages people to not learn to use standards properly. (oh hey, is this a soapbox I'm standing on? look at that...)

[Relequestual]

Giving GET request bodies meaning is in fact not a violation of the spec. HTTP/1.1 specifies that servers SHOULD ignore the body, but RFC 2119 specifies that implementers are allowed to ignore "SHOULD" clauses if they have good reason to do so. Rather, a client does violate the spec if it assumes that changing the GET body will not change the response.

(http://stackoverflow.com/questions/978061/http-get-with-request-body#comment56145237_983458)

[Relequestual]

@handrews I think what you're getting at is pretty well laid out in this interesting article you should read https://blog.goodapi.co/future-of-apis-c84a76bc9c85#.ewz8ghdij - mentions HATEOAS and JSON-LD but no love for Hyper Schema =[

[donaldpipowitch]

@handrews Thank you for your definition. Yes, we use JSON Hyper Schema more like a "service definition format". Our JSON Hyper Schema is highly dynamic and personalized. It can change at runtime and therefor a static documentation is not enough for us.

[handrews]

@donaldpipowitch perhaps I didn't explain it as well as I thought. A service definition format, as I mean it, is more of a static thing geared towards generating documentation. So a hyper-schema that changes at runtime (I assume by this you mean that you attach different schemas with different URIs to a specific resource's response at runtime, not that the schema at a particular URI changes at runtime) is not a service definition approach. That is more of a hyper-media approach, which I think JSON Hyper-Schema can provide but service definitions struggle with (although again, I haven't delved deeply into the state-of-the-art of service definitions recently, so I may be missing some things).

[handrews]

@Relequestual I did not say in this issue that a GET with a body is in violation, I just gave it as an example of something that "[does] not necessarily strictly adhere to the official HTTP semantics". The passage I quoted has no SHOULD (nor does it have a MUST or a MAY). It just states that a particular behavior cannot be relied upon, and that such a request may be rejected entirely as a result.

So when an API defines semantics for something that the standard warns may be rejected entirely (for instance by intermediaries between the client and server), such an API is definitely not within HTTP's defined semantics. This is different from POST, where the HTTP standard specifically states that the resource defines the semantics. "The resource defines this" is different from "there is no defined behavior, so the behavior may be that it does not work at all").

Generally, you're going to be running the Elasticsearch server and client within your own local network, so you can control whether it goes through any intermediaries that may not handle a GET request with a body. But over arbitrary internet connections, if you end up going through a proxy or cache that rejects GET requests with bodies, those proxies or caches would not be in violation of HTTP. You're not violating HTTP to send a request body, but neither are they violating HTTP to refuse such a request.

[handrews]

@Relequestual the "SHOULD" that you reference is in RFC 2616 which was obsoleted by RFC 7231 a few years ago. And in any event, this is the passage in question:

A server SHOULD
read and forward a message-body on any request; if the request method
does not include defined semantics for an entity-body, then the
message-body SHOULD be ignored when handling the request.

and the SHOULD here is about forwarding the body. So that means that an implementation can choose to not forward the body.

[handrews]

@Relequestual that link is great, thanks! Yes, exactly the sort of thing I'm getting at, I just forwarded it to people at work :-)

As a side note, I think JSON-LD can complement JSON Hyper-Schema. Both do hyperlinking, so that's something to be ironed out, but Hyper-Schema is flexible enough to work in terms of JSON-LD. And JSON-LD's ability to map standard vocabularies is great and outside of the scope of our work here.

[Relequestual]

In addition... I'm a muppet. With the API I was talking about, we DO actually do a POST, not a GET. I probably event recomended such. AND I won't litter this issue any further... =]

[handrews]

@Relequestual LOLOL! I noticed that "because GET with a request body is not universally supported, the search API also accepts POST requests"

To be fair, they are correct that GET makes more sense for data retrieval. While I understand HTTP's philosophy (if you are retrieving a resource, you should only need its identifier), I do agree that the limitations in what you can encode in a URI can make GET's lack of body problematic at times.

[awwright]

I think another way to put this is that JSON Hyper-schema is used to describe links between JSON documents. It is not a service description language, and how you interact with those resources remains the responsibility of the protocol (e.g. HTTP).

What's the proposed course of action for this issue?

[handrews]

@awwright yes that's the underlying principle for this topic.

The course of action is to decide the extent of our flexibility when it comes to gray areas (GET with a body is neither required nor forbidden by RFC 7231), or more importantly valid protocol uses that don't fit within the most common patterns.

For instance, RFC 7231 defines a PUT response is defined as "a representation of the status of the action." If that representation is not also a representation of the resource (as indicated by Content-Location), there is no way for JSON Schema to provide an advisory schema for that response.

Of course, once you have that response at runtime, it presumably identifies its own schema. But should we, and if so how would we provide an advisory schema for a PUT response when it is not a representation of the resource?

Let's start with that. I can expand more if needed depending on the responses.

[handrews]

I think that the improved language around context and target in RFC 5988bis section 2 has given me an idea on how to convey what @awwright was getting at with this bit:

I think another way to put this is that JSON Hyper-schema is used to describe links between JSON documents. It is not a service description language, and how you interact with those resources remains the responsibility of the protocol (e.g. HTTP).

I'm going to write it up as a trial balloon PR because it makes the most sense if you see it in the actual spec environment. As usual, I expect to have to significantly rework trial balloon PRs before acceptance, if accepted at all.

[handrews]

As a result of the trial balloon PR #366, I filed issue #377 to rework our language around the abstract model of RFC 5988bis. This, plus revisiting JSON Home (which is better-defined than a year ago) and the "linkset" I-D has helped me determine where I think the line is between a hypermedia media type (which is what Hyper-Schema is) and other sorts of descriptions.

I cover that distinction in detail in json-schema-org/json-schema-vocabularies#1, which proposes an API Documentation vocabulary to address those non-hypermedia media type concerns.

Therefore I am going to close this. I think the answer is that Hyper-Schema itself will not go out of its way to accommodate non-RESTful/non-protocol-compliant systems, but the API Doc vocabulary is the correct way to bridge that gap.

[dlax]

I cover that distinction in detail in json-schema-org/json-schema-vocabularies#3, which proposes an API Documentation vocabulary to address those non-hypermedia media type concerns.

Did you mean json-schema-org/json-schema-vocabularies#1?

[handrews]

@dlax yes, thanks! Fixed.