DRY Specifications via $ref

[BenjamenMeyer]

So I realize there is some discussions regarding some "similar" things, but I'm not quite sure they're similar enough - if so, please say so and I'll contribute to the other discussions accordingly and close this one out. Other similar but not quite as extensive discussions I have found:

• Extra JSON Reference properties ignored - reduces reusability of references #556 : polymorphism around $ref
• Support for multiple $refs inside a single object #417 : paths and $ref

Basically, one should be able to write their OpenAPI specification in as DRY a manner as possible. Presently this is not possible because:

1. $ref is not valid at all the right places
2. allOf is not valid at the right places (which could make up for point 1)
3. $ref can't be specified numerous times next to itself appropriately

Let's give this some better clarity via an example. Discussing API Header Fields provides a good example such as the following which similar to #417 but I'm going more general where that seems to be focused on path objects:

paths:
    /:
        get:
            parameters:
                - $ref: `#/components/headers/x-my-header-1`
                - $ref: `#/components/headers/x-my-header-3`
            responses:
                '401':
                    headers:
                        $ref: `#/components/headers/x-my-header-1`
                        $ref: `#/components/headers/x-my-header-3`
                '403':
                    headers:
                        $ref: `#/components/headers/x-my-header-1`
                        $ref: `#/components/headers/x-my-header-3`
                '200':
                    headers:
                        $ref: `#/components/headers/x-my-header-1`
                        $ref: `#/components/headers/x-my-header-2`
                        $ref: `#/components/headers/x-my-header-3`
               '301':
                    headers:
                        allOf:
                            - $ref: `#/components/headers/x-my-header-1`
                            - $ref: `#/components/headers/x-my-header-2`
                            - $ref: `#/components/headers/x-my-header-3`
                default:
                    headers:
                        $ref: `#/components/headers/x-my-header-1`
                        $ref: `#/components/headers/x-my-header-3`
        post:
            parameters:
                - $ref: `#/components/headers/x-my-header-1`
                - $ref: `#/components/headers/x-my-header-2`
            responses:
                '401':
                    headers:
                        $ref: `#/components/headers/x-my-header-1`
                        $ref: `#/components/headers/x-my-header-3`
                '403':
                    headers:
                        $ref: `#/components/headers/x-my-header-1`
                        $ref: `#/components/headers/x-my-header-3`
                '200':
                    headers:
                        $ref: `#/components/headers/x-my-header-1`
                        $ref: `#/components/headers/x-my-header-2`
                        $ref: `#/components/headers/x-my-header-3`
                default:
                    headers:
                        $ref: `#/components/headers/x-my-header-1`
                        $ref: `#/components/headers/x-my-header-3`
    /howdy:
        delete:
            parameters:
                - $ref: `#/components/headers/x-my-header-1`
                - $ref: `#/components/headers/x-my-header-2`
            responses:
            '401':
                headers:
                    $ref: `#/components/headers/x-my-header-1`
                    $ref: `#/components/headers/x-my-header-3`
            '403':
                headers:
                    $ref: `#/components/headers/x-my-header-1`
                    $ref: `#/components/headers/x-my-header-3`
            '200':
                headers:
                    $ref: `#/components/headers/x-my-header-1`
                    $ref: `#/components/headers/x-my-header-2`
                    $ref: `#/components/headers/x-my-header-3`
            default:
                headers:
                    $ref: `#/components/headers/x-my-header-1`
                    $ref: `#/components/headers/x-my-header-3`
components:
    headers:
        x-my-header-1:
            required: true
            description: my first header
            schema:
            type: string
            format: uuid
        x-my-header-2:
            required: true
            description: my second header
            schema:
            type: string
        x-my-header-3:
            required: true
            description: my third header
            schema:
            type: integer

The above is missing a lot of stuff, but only in order to show the re-use and avoid having to put a complete spec here. I know some of the above can be fixed by moving up a layer in the referencing, e.g create a component for the header set and reference that instead; however that can then dictate a bad spec by forcing all headers to be pushed to header sets which then can also be an issue for 1-off header combinations.

Goal here is to enable spec writers to manage their specifications by creating re-usable modules that can be continuously re-used. To achieve this, $ref would need to be valid in essentially ever object to replace anything in the object. F.e if you have a description field you want to re-use, define it once and use it 10 times.

Two parallel $ref values should be valid and able to point to separate objects, neither being discarded, or allOf could be used to combine the contents of both $ref references to create the same effect (good solution for one-offs).

When building larger APIs using tooling like OpenAPI being able to be a DRY as possible is key to keeping bugs from creeping into the specs as it reduces the ability for any one instance to be mis-typed. For instance, if you had to type x-my-header-3 for every single Request object and x-my-header-2 for every single Response object - one typo of my-header-3 or xmy-header-3 could easily create something hard to detect where using the $ref objects would make it fail validation and be easily caught in gate checks (PR builders, etc).

To re-iterate - please let me know if I need to file this with JSON Reference/Schema too. I did not find anything suitably talking about these aspects while reading through any of the repos I came across, but have in general there does seem to be some of sentiment that this is an issue.

[handrews]

@BenjamenMeyer I'm one of the editors of the JSON Schema spec.

There are a couple of things going on here, some due to JSON and YAML, and others specific to OAS. As phrased, none of this is actually about JSON Schema.

For all practical purposes, JSON does not allow allow duplicate keys in objects. They do not technically violate the RFC, but the RFC notes that the behavior is undefined and therefore not interoperable. Furthermore, YAML absolutely requires unique keys. So your number 3, putting multiple $ref keys in the same object, is not possible.

The correct way to have multiple $refs is to simply wrap them in an allOf:

allOf:
  - $ref: '#/components/headers/x-my-header-1'
  - $ref: '#/components/headers/x-my-header-2'
  - $ref: '#/components/headers/x-my-header-3'

which reduces your 3 to a combination of 1 and 2. Those are limitations of OAS. JSON Schema allows $ref anywhere a schema is expected, and allows allOf as a keyword in any schema object. There are a number of issues and discussions going on about whether and how to converge OAS's schema variant and JSON Schema proper, although there is no particular timeline on that right now.

[MikeRalphson]

@BenjamenMeyer @handrews

Just to clarify, OASv3 also allows a $ref anywhere a schema object is expected:

Alternatively, any time a Schema Object can be used, a Reference Object can be used in its place. This allows referencing definitions instead of defining them inline.

and also allows allOf as a keyword in any schema object, with the caveat:

allOf - Inline or referenced schema MUST be of a Schema Object and not a standard JSON Schema.

What is not allowed is using a $ref where a primitive type (like a string description) is expected (and in some other objects which have no reusable components to $ref to or there are no defined semantics for combining them). Nor is using allOf (which is a JSON schema keyword) outside of a schema object.

If you wish to experiment with allowing allOf (and $ref) anywhere, I think the first stage would be to look at a pre-processor of some description. That would allow you to discover problems and edge cases with the approach.

PS: I appreciate the above is just an example, but it is not recommended to $ref a header object from the parameters array where only parameter objects (or $refs to them) are expected.

[handrews]

@MikeRalphson OK the reference object rules in OAS match how JSON Schema works. In older drafts it was outside of the spec and therefore could be used anywhere, but we changed that in draft-05. We have strong reasons for not allowing $refs to things other than schemas but it's also a long story so I'll leave it at that :-)

[BenjamenMeyer]

Sorry for delay - trying to figure out what I wanted to say more properly...

@MikeRalphson per your "PS" that's kind of my point here is that $ref should be valid there; $ref should be more generic in usage so that anything can be pointed to; it should still be valid for the kind of contents that go in that section, but it should be valid. IOW one can write:

base:
     howdy:
        field: string

Then:

base:
    howdy:
        $ref: '#/value_types/field'

values_types:
    field: value

or event

base:
    $ref: '#/collections/howdy`

collections:
    howdy:
        $ref: '#/value_types/field'

values_types:
    field: value

kind of thing ought to be valid so field could be re-used as appropriate as often as possible. I would think that'd qualify as a Schema Object of the appropriate type for the $ref and the OAS validator/generator should be able to use internal polymorphism appropriate too if the reference could be interpreted as different kinds of Schema Objects.

Again - I'm new OAS so trying to understand how all this applies to maximize DRY/re-use in writing a big spec. From what I've been told (outside of here), using $ref is a lot more restricted; but if I'm understanding you all correctly, reality is somewhere between that extremely restrictive view and what I'm asking here.

@handrews If it's not possible, I'd be interested in reading up. Do you have a link to an PR/Issue/etc I could follow for the background?

[handrews]

@BenjamenMeyer I'm not sure I follow your example as the only recognizable JSON Schema keyword is $ref. Could you please show a bit more context? In particular, please make clear what is OAS (because I don't have the whole OAS spec memorized) vs JSON Schema vs new proposals vs example names (e.g. I'm assuming "collection" is an example name, not a keyword of any sort)

Sorry I did not notice that you'd updated this, I try to keep on top of at-mentions but this one slipped through.

[BenjamenMeyer]

@handrews no problem: I'm not on github.com as much myself nowadays either so I may take a bit longer to respond too.

To answer your question, my example is very generic, kind of on purpose - so everything falls into being an example name.

To try to bring some more clarity...

1. I don't have the work I was doing accessible at the moment so going off the cuff for this...
2. I don't have the spec down either, so I used https://github.com/OAI/OpenAPI-Specification/blob/master/examples/v3.0/petstore.yaml as an example to draw some stuff from for the below.

But to take a stab at it against what I was doing it'd be something like:

# openapi.yaml
openapi: "3.0.0"
info:
    version: 1.0.0
    title: Example
paths:
    /:
        $ref: "root.yaml#/root"
...
------------------------------------------------------
# root.yaml
root:
    get:
        summary: hello
        operationId: getHello
        responses:
             '200':
                  $ref: "response.yaml#/helloWorld"
             '401':
                   $ref: "responses.yaml#error/auth_401"
    post:
        $ref: "submission.yaml#post/hello"
...
------------------------------------------------------
# responses.yaml
helloWorld:
    description: Hello World
    content:
        $ref: "#/content/json/hello/"
content:
     json:
         application/json:
              schema:
                  $ref: "#/schemas/hello"
...

Essentially, once past the basic path, almost anything could use a $ref to re-use the information.

And yeah, I know - I'm not following the general structure of the field naming at the top level; but this should be able to be structured in a way that makes sense to document writers and enables them to easily expand the document out and re-use material.

[handrews]

@BenjamenMeyer sounds like you are looking to add more $ref for things outside of the schema object, so that's outside of what I do with my JSON Schema hat on.

Since allOf is very much a JSON Schema construct relating primarily to constraints rather than definitions, I'm not sure it would make much sense outside of JSON Schema. I could possibly be convinced otherwise, but I'd want to really understand how the semantics are consistent with the JSON Schema usage. Otherwise, a different keyword would be preferred.

With my "interested party but not all that proficient in OpenAPI" hat on, I generally like allowing $ref for any coherent object, but not for splicing objects.

@BenjamenMeyer you asked about the history of only allowing $ref where a schema is allowed. In draft-04, JSON Reference was its own specification, and JSON Schema technically wasn't aware of it at all. So there was no notion of being able to restrict its usage.

The problem with this is that if you can use $ref in other places, it gets much harder to reason about schema objects in code. You can't, for instance, figure out the property set that is relevant to additionalProperties without following a $ref if the whole value of properties is a $ref. And doing that doesn't give you much over $ref-ing a schema that is just a properties keyword anyway.

If you go further and say that you can do something like {"properties": {"allOf": [{"$ref": "..."}, {"$ref": "..."}]}}, in addition to being confusing (although not impossible) to distinguish from a property named "allOf", you get into schema merging and splicing.

That is an incredibly complex topic spread across numerous issues across at least two repositories with well over 500 comments in total. If you really want to know I'll point you to them, but I recommend against it. The TL;DR is "it doesn't solve as many problems as you think, and it introduces new and extremely difficult problems as well".

In place of schema merging/splicing, we are pursuing json-schema-org/json-schema-spec#556. It may not be obvious how those relate, but again, the backstory is extremely involved.

[BenjamenMeyer]

@handrews thanks for the insight.

I can certainly understand the issue with the properties, and appreciate the concern of complexity.

[jam01]

@handrews @MikeRalphson but what about combining objects other than schemas, since they are json (yaml?) instances can't we just do something like merge ? For example, in order to extend an operation with a new response couldn't we do something like

  get:
    $merge
      - $ref: openApi-registry.com/foo#components/operation/greatOperation
      - responses:
          418:
            ...

Which would just do a deep copy of both json instances... Jackson implemented this here FasterXML/jackson-databind#1399

[handrews]

$merge is deeply problematic. There are well over 500 comments on this spread across four or five issues in the current JSON Schema repo, plus one issue in the old JSON Schema repo (and I am not exaggerating- one issue alone has over 230 comments). I don't want to get into it, but just no.

[fenollp]

Just to point out that YAML has anchors and you can work around this issue today. Example

Repeated nodes (objects) are first identified by an anchor (marked with the ampersand - “&”), and are then aliased (referenced with an asterisk - “*”) thereafter.

Now I know it's a different story for JSON, but most examples in this thread were in YAML. Hopefully this helps.

[handrews]

@fenollp $ref works just fine as a JSON replacement for YAML anchors, as far as I know. It's just a question of whether OAS wants to allow a $ref here, and maybe whether there's some way to apply specific combination semantics along the lines of allOf.

The reason allOf (and the draft-08 unevaluatedProperties keyword) work and $merge does not is that the former keywords are defined in terms of JSON Schema semantics, so they are consistent with the overall processing model. $merge is simply syntactic splicing, so the semantics of the input and output are not predictably related. In some ways that's not much of a practical concern, but it turns out there are also several situations where syntactic splicing does not help (involving oneOf, anyOf, and if, all of which require dynamic evaluation with an instance to fully determine their effects- since $merge is static, it cannot work well with those keywords).