OAS and Overlays integration ideas

[handrews]

I'm splitting this off from #5116 as it has gotten by far the most response so far. I will port over the relevant comments.

The part of the original post of that discussion that kicked this off is:

Next, enable / encourage implementations to work together.

• Determine what is needed from the other specs to avoid feature gaps
• Allow OADs to indicate they are unusable without first applying overlay(s)
• Allow OADs to indicate that they should be used via Arazzo workflows

The latter two points require understanding the use cases, both for whether they should be done at all (Arazzo probably, Overlays maybe) and how (tight vs loose coupling).

[handrews]

@mikekistler replied:

---

@handrews Thanks for getting this started!

Regarding:

Determine which requested OAS features are best handled by Overlays

Something I think is worth exploring is integrating some Overlays features into OAS. In particular, I think that if we allowed an OAS spec to include "actions" array, we could incorporate "traits" directly into OAS.

[handrews]

To which I replied:

@mikekistler so.. I have like 10x these notes written down but didn't want to overwhelm everyone with details. Under "Next ecosystem step" above, I wrote:

Allow OADs to indicate they are unusable without first applying overlay(s)
Allow OADs to indicate that they should be used via Arazzo workflows

The latter two points require understanding the use cases, both for whether they should be done at all (Arazzo probably, Overlays maybe) and how (tight vs loose coupling).

The more expansive version of this was:

---

Later steps: An integrated ecosystem

This part requires thinking through the UX of using a set of OADs, Overlay documents, and/or Arazzo documents that are intended to be used together.

• For Overlays, this means applying them before the OAD is usable, and getting an error if the OAD is used without applying the Overlays first
• For Arazzo, this means that the OAD author intends users to work primarily or exclusively from Arazzo documents, not directly from the OAD

There are several possible levels of this:

• A simple flag saying Overlays needed / Arazzo workflows available
• A link to a repository where such overlays/workflows might be found
• Lists of refs to specific overlays that MUST be applied or workflows that SHOULD be preferred over raw OAD use
• Inline overlays / Arazzo documents (note: consider the pros and cons of JSON Schema bundling, as there would be similar issues)

---

So yes, incorporating one or more Overlays directly into an OAD is something I was considering as well. My preference would be to more-or-less literally embed Overlay documents so that they can be delegated to a separate (or very loosely coupled) Overlays implementation for processing, and so that the Overlay version can vary independently from the OAS version (beyond some minimal expectation of, say, OAS 3.3+ and Overlays 1.1+).

I would have two lists of Overlays in the OpenAPI Object: One that is document-scoped, meaning that those Overlays have to be applied in order to the document when it is read, before anything else happens, and another that is OAD-scoped, meaning that this second list is only used from the entry document, but is applied to every document in the OAD, after that document's document-scoped Overlays are applied.

However, my main requirement (long-term) would be the ability of an OAD author to say that overlays MUST be applied, and ensure that tools error-out if that has not happened. There are ways to do that other than by embedded (whether literally inline or by-reference) overlays, such as having a boolean flag that is cleared when overlays are applied. So I don't want to preclude an alternate solution here.

My personal preference would be to allow embedded overlays, but that's one of several options.

---

There are some analogous questions around referencing/incorporating Arazzo content, although it's also an interesting question what directions those connections should go. Overlays happen before the resulting OAD is used, but Arazzo builds on top of an OAD's descriptions, rather than manipulating its text to change the description.

[handrews]

And @SensibleWood replied:

---

For Overlays, this means applying them before the OAD is usable, and getting an error if the OAD is used without applying the Overlays first

@handrews @mikekistler trying not to do more than make worthy notes, but this I like immensely. Almost you want a "policy" thing that says:

requiresPolicy: securitySchemeApplies

And that drives the consumption of the OAD i.e. if there is no Security Requirement, the OAD is effectively dead until the Security Scheme Policy (via Overlay) is applied.

[handrews]

@baywet started another reply thread with:

---

Adding on the "encourage implementations to work together" aspect. Another option (don't do anything, add a boolean, embed the related arazzo/overlay files in the original specification) is to design a new "manifest" format.

Not only this would help the discovery (potentially automated) of the description itself, it'd maintain a separation of concerns, and we have precedents for that. https://www.ietf.org/archive/id/draft-miller-api-manifest-01.html https://apisjson.org/

Remixing those to document "this thing contains a description, and two overlays to apply in this order, and one arazzo that gives you more context about this and that" might be a good approach.

[handrews]

@baywet Having had more time to consider this, I think there are really two distinct things going on here:

• Leveraging the central nature of the OAS, in terms of how our community thinks of things, to ensure that when Overlays are needed or Arazzo workflows are available, that users who do not know about Overlays or Arazzo are informed of the dependencies
• Packaging, distributing, and discovering groups of documents for coordinated use

A manifest file helps the second but not the first. If we were starting all of the specs at once, having a manifest as the primary entry point would make sense.

But one of the complaints about deferring things to Overlays is that people want the OAS to be the entry point because that's how they think about it. A manifest won't help this, but approaches that put something into OADs would.

Now, we could put a manifest in the OAD. In fact, certain of the proposed options more-or-less do that, and the notion of a "manifest" section added to the OAS rather than a couple of independent mechanisms might be a best-of-both-worlds option. It need not preclude a standalone manifest either, just like if we were to go with @mikekistler 's notion of embedding overlay content directly into OADs, that would not stop people from also using Overlays completely separately.

[lornajane]

I see the manifest as a separate entity because at least an Overlay, perhaps less an Arazzo Description, can apply to many OpenAPI documents (for example, an overlay to remove DELETE operations because your data retention rules require an alternative archiving policy). Overlay needs a manifest file since they are applied in a specific order (and at the moment people seem to prefix the files with numbers which makes me think of 20 GOTO 10 every time :) - it makes sense to consider other specifications to use within or alongside the overlay manifest, definitely.

[handrews]

@lornajane all of that makes sense. The "inline manifest" option (not sure if that's really the best terminology) would specifically signal that an OAD is not usable without the overlays applied in order. If that inline manifest references overlays in external files, there's nothing to prevent them from also being used in other, independent manifests, or without a manifest.

So I think we're saying compatible things here? Which is not to say we must do both inline and separate manifests, but, at least technically, we could?