AsyncAPI Initiative
Reference tooling discussion and requirements #485
Replies: 17 comments 58 replies
-
Discussion around U1AsyncAPI + JSON Schema >= draft 2019-09 with bundling. Let's try to have all the discussions related to this use case below here to keep the discussion specific to one case at a time. Otherwise, we might get lost very quickly! |
Beta Was this translation helpful? Give feedback.
-
|
Yes, if I understand this correctly it is not just allowed but required (although the normative language here is a little weird- some of it might be a SHOULD when it ought to be a MUST, we're working on cleaning that up). References at least SHOULD be resolved internally (using embedded |
Beta Was this translation helpful? Give feedback.
-
|
So as I understand what you are saying this is definitely something from JSON Schema side that is a MUST to support, however, would you say this is clear from an AsyncAPI side (for schemas under components)? While I agree it's definitely possible to do (implementation-wise), I have not been convinced the specs enforce it in such a way you can argue it's a bug in the implementation if it's not supported. I see this as a huge problem because if you design your documents with that "feature" in mind you will be sorely disappointed when you try to use those documents in any tools. So to me, I feel like we are missing some concrete sections in the spec document that highlight this situation in some way or another 🤔 |
Beta Was this translation helpful? Give feedback.
-
|
Yeah this is tricky because JSON Reference does not have any awareness of |
Beta Was this translation helpful? Give feedback.
-
|
yes exactly. But also such bundling techniques are only relevant for JSON Schema formats, if you use Avro, Protobuf, or whatever, such behavior is non-existing (I would assume). The alternative is that it's scoped to the schema at hand instead of implicit for the entire AsyncAPI document. asyncapi: '2.5.0'
info:
title: Account Service
version: 1.0.0
description: This service is in charge of processing user signups
channels:
user/signedup:
subscribe:
message:
$ref: '#/components/messages/UserSignedUp'
components:
messages:
UserSignedUp:
schemaFormat: 'application/schema+json;version=draft-2019/09'
payload:
$defs:
UserSignedUp:
$schema: "https://json-schema.org/draft/2019-09/schema"
$id: "https://my-schema-registry.org/UserSignedUp"
type: object
properties:
displayName:
type: string
description: Name of the user
email:
type: string
format: email
description: Email of the user
$ref: 'https://my-schema-registry.org/UserSignedUp'So the schemas within the AsyncAPI document under Could also be: asyncapi: '2.5.0'
info:
title: Account Service
version: 1.0.0
description: This service is in charge of processing user signups
channels:
user/signedup:
subscribe:
message:
$ref: '#/components/messages/UserSignedUp'
components:
messages:
UserSignedUp:
schemaFormat: 'application/schema+json;version=draft-2019/09'
payload:
$defs:
UserSignedUp:
$id: "https://my-schema-registry.org/UserSignedUp"
$ref: '#/components/schemas/UserSignedUp'
$ref: 'https://my-schema-registry.org/UserSignedUp'
schemas:
UserSignedUp:
$schema: "https://json-schema.org/draft/2019-09/schema"
type: object
properties:
displayName:
type: string
description: Name of the user
email:
type: string
format: email
description: Email of the userNot sure if that would create more confusion or less 🤷 |
Beta Was this translation helpful? Give feedback.
-
Discussion around U2AsyncAPI + JSON Schema >= draft 2019-09 with bundling and relative reference. Let's try to have all the discussions related to this use case below here to keep the discussion specific to one case at a time. Otherwise, we might get lost very quickly! |
Beta Was this translation helpful? Give feedback.
-
|
Yes to this one as well. |
Beta Was this translation helpful? Give feedback.
-
|
This one is directly related to #485 (comment) and based on that discussion it will directly influence this - as the relative references are a feature in JSON Schema. |
Beta Was this translation helpful? Give feedback.
-
Discussion around U3AsyncAPI + JSON Schema < draft 2019-09 with bundling. Let's try to have all the discussions related to this use case below here to keep the discussion specific to one case at a time. Otherwise, we might get lost very quickly! |
Beta Was this translation helpful? Give feedback.
-
|
@handrews I get it. I completely understand it after @jonaslagoni explained it yesterday at the Spec 3.0 meeting. It may be "normal" behavior in accordance with any standard but what I'm saying is that it's highly confusing and counterintuitive. And even worse, I'd say it's an unexpected behavior for the majority of the AsyncAPI users (and probably JSON Schema users too but that I don't know). So yeah, it may make sense in general for JSON Schema but it definitely makes AsyncAPI unnecessarily complex. To be clear, I'm speaking about this:
So, effectively, in this case, the Now, if you tell me, for these cases we're gonna use a different keyword: {
"$localRef": "https://my-schema-registry.org/UserSignedUp"
}Alright, I can still find it confusing but, at least, there's something there telling me that this reference is local so I, as a developer, would go to the docs just to see what the heck Don't get me wrong. I'm not proposing we introduce |
Beta Was this translation helpful? Give feedback.
-
|
@fmvilas this is why both JSON Schema and JSON Reference are very careful to always say "URI" and not "URL". JSON Schema even explains this distinction in the sections on referencing, identification, and vocabularies. Sometimes people criticize the URI/URL distinction as academic stuffiness, but it's important for understanding how to think about these features. Treating these URIs as URLs is not feasible for several reasons:
Once you start thinking in terms of URIs as identifiers rather than locators, all of this is much easier to think about:
(I actually have a blog article on URI vs URL that I ought to publish soon)
So is local resolution of
But the larger point here, and the thing that folks need to understand when working with specs, is that URIs aren't URLs and shouldn't be thought about the same way. It's an adjustment, yes, but it carries substantial benefits. |
Beta Was this translation helpful? Give feedback.
-
In my whole experience with all the conversations I've had with people who use or wanted to use AsyncAPI, I've had any of these issues raised as a concern or even just mentioned. This makes me think that, yes, it's most probably "academic stuffiness". I didn't know it was meant to be URIs instead of URLs and I've been doing this for 6 years now + a few more editing OpenAPI files. So I can't imagine someone who's not working on this full-time. I'm sorry but all these reasons are valid from a theoretical standpoint but not really a concern in real-life use cases.
And substantial drawbacks. My feeling all this time working with AsyncAPI and OpenAPI is that people prefer simplicity because this is just another tool in their toolchain. Requesting them to learn this kind of stuff to simply use a |
Beta Was this translation helpful? Give feedback.
-
This raises some questions for me about the scope of AsyncAPI and the products in which it is expected to be used. Air-gapped products, or those that are still connected but behind an aggressively restricted firewall, are a huge market, particularly in the government and defense sectors. I've spent many hours in meetings with PMs on how to manage APIs for products that are deployed in this way, including how to package, distribute, and load schemas. There are a lot of companies and customers out there who would be surprised to not be considered "real-life use cases". I've also spent a fair amount of time working on large-scale test automation of enterprise products, where we needed to create test environments with a complex topology of products running different released and pre-release versions of the API software. Being able to separate location (
I've been working with JSON Schema for more than a decade now. In my experience, it is no harder to explain referencing local
If we're going to frame this in terms of what folks want, then we need to address the full set of use cases and not dismiss various markets or usage scenarios as "academic" when they actually have millions if not billions of dollars attached to them. Likewise for the security concerns involving fetching remote resources at runtime: How would software that does such a thing hold up in a security audit? Auditors don't tend to accept "that's not a real-world scenario" as an answer if they think something is a security problem. It's their job to come up with the unlikely-but-possible scenarios that hackers exploit. Fundamentally, this is a problem of education, and we can make very, very, good education materials. Compared to everything else involved in working with APIs, this is a very small thing. |
Beta Was this translation helpful? Give feedback.
-
|
And keep in mind that no one is being forced to make their Only tooling implementers and spec authors (meaning us, not people writing API descriptions) need to understand this. In a larger organization, there will probably be a tech lead or architect who understands all of these details and works with the more junior team members who don't have to learn everything up front (I've been in this role multiple times and understand the degree to which people at different experience levels do and do not absorb spec minutia that is not immediately relevant to their work). |
Beta Was this translation helpful? Give feedback.
-
Discussion around U4AsyncAPI + JSON Schema >= draft 2019-09 with bundling and internal definitions. Let's try to have all the discussions related to this use case below here to keep the discussion specific to one case at a time. Otherwise, we might get lost very quickly! |
Beta Was this translation helpful? Give feedback.
-
|
We consolidated information on duplicate URIs in a recent PR TL;DR: Defining the same URI twice (with any combination of The PR I linked emphasizes that if a tool notices duplicate URIs, it SHOULD raise an error. |
Beta Was this translation helpful? Give feedback.
-
|
That is a fair assessment 🤔 Damn, that's why we encountered it with the bundled schemas and had to write a specific section about it: https://github.com/asyncapi/spec-json-schemas/blob/master/migrations/Migrate%20to%20version%203.md#bundled-schemas Now it makes sense from a standard perspective. But what about those scenarios where it is not possible to change the |
Beta Was this translation helpful? Give feedback.
-
|
Hmm.. actually if you identify the same schema with the same URI that should be fine. Of course, that requires comparing the schemas, but if you literally just copied them then tools ought to recognize that you're just telling them something they already know. Although there is no language about this case in the spec (there is always a debate as to how much to highlight these things in the spec). But that "SHOULD raise an error" is about identifying different schemas with the same URI. |
Beta Was this translation helpful? Give feedback.
-
|
I am unsure what to write here 😆 Let's try to continue and accept it as the expected behavior. Should it be allowed to overwrite this behavior i.e. ignoring schemas with duplicate URIs and just using the first one encountered? Especially for JSON Schema < draft 2019-09 ? 🤔 |
Beta Was this translation helpful? Give feedback.
-
Discussion around U5AsyncAPI + JSON Schema contradicting formats. Let's try to have all the discussions related to this use case below here to keep the discussion specific to one case at a time. Otherwise, we might get lost very quickly! |
Beta Was this translation helpful? Give feedback.
-
|
Basically, when you embed a resource with |
Beta Was this translation helpful? Give feedback.
-
|
So this means that ultimately the Do we think the specs define this as the expected behavior well enough to be expected behavior? 🤔 |
Beta Was this translation helpful? Give feedback.
-
|
I think it's clear in 2020-12. Prior to that we did not allow |
Beta Was this translation helpful? Give feedback.
-
Oh! I did not know that tbh... Or I forgot, that might also be the case 😆 But if you are using references, would there not be cases where a referenced document is the root in some cases and in others not?
interesting! |
Beta Was this translation helpful? Give feedback.
-
That is exactly why you can change |
Beta Was this translation helpful? Give feedback.
-
Discussion around U6AsyncAPI using OpenAPI Schema documents which expect a base URI. Let's try to have all the discussions related to this use case below here to keep the discussion specific to one case at a time. Otherwise, we might get lost very quickly! |
Beta Was this translation helpful? Give feedback.
-
Yes, any tooling supporting relative references needs to support this. There isn't a PR for it yet, but I've been meaning to make this more explicit in the JSON Schema spec (ironically, someone argued with me over this on the grounds that it should be obvious from RFC 3986, but I don't think most people find it obvious).
You may not have a way to explicitly set it per RFC 3986 §5.1.1, but §5.1.2, 5.1.3, and 5.1.4 all apply and mean that an AsyncAPI document does have a base URI. Most often, it's the retrieval URI per §5.1.3. But it is a good idea to allow setting it within the format, because then you can have consistent behavior whether you're loading it from a filesystem, or from a test server, or wherever. |
Beta Was this translation helpful? Give feedback.
-
Alright, let's add this as a requirement 👍 I think it makes sense as there might even be other cases where this is useful.
There are many parts about standards that are not very clear, especially when it comes to more everyday questions it can be very hard to map that to the standard and their explanation. At least it is for someone that has not written the standard, nor spend endless hours with it (speaking for myself of course) 😄
That's true and might be added at some point in the future, it's definitely not on the roadmap at the moment, and nobody has requested it. For this use-case, I think it makes sense to describe this dilemma somewhere in AsyncAPI especially when it comes to the reuse of schemas from OpenAPI documents. It just comes down to where? |
Beta Was this translation helpful? Give feedback.
-
Discussion around U7OpenAPI + JSON Schema contradicting base-URI's Let's try to have all the discussions related to this use case below here to keep the discussion specific to one case at a time. Otherwise, we might get lost very quickly! |
Beta Was this translation helpful? Give feedback.
-
|
I was wrestling with these two sections last week a bit. Which is funny b/c I wrote the URI one so I ought to know how it fit with the URL one, but TBH I don't remember what I thought at the time. My interpretation is:
|
Beta Was this translation helpful? Give feedback.
-
|
That makes sense 👍 I guess we can move this to |
Beta Was this translation helpful? Give feedback.
-
Discussion around U8AsyncAPI + non-JSON references Let's try to have all the discussions related to this use case below here to keep the discussion specific to one case at a time. Otherwise, we might get lost very quickly! |
Beta Was this translation helpful? Give feedback.
-
|
For this one, I'm going to import some of my questions from the existing PR A few questions for clarifying requirements. By secondary resource, I more-or-less mean "thing you'd use a fragment to reference if fragment behavior were defined for the media type."
|
Beta Was this translation helpful? Give feedback.
-
Initially, my answer was yes, that might happen. However, I think I want to change it to a no. The reason being
I would say that would be preferred, otherwise, implementations can come to the wrong conclusion of what that second resource is, or not even be able to find it.
I would say no here. But I dont have a complete experience with all the standards so I cant possibly say tbh. Maybe asyncapi/spec#622 can give you some insight.
I would say yes because of asyncapi/spec#622.
If you take a look at JTD, it uses its own referencing format i.e. simple mapping between references and definitions. See https://www.rfc-editor.org/rfc/rfc8927#name-ref |
Beta Was this translation helpful? Give feedback.
-
Discussion around U9AsyncAPI + JSON Schema contradicting formats for remote references Let's try to have all the discussions related to this use case below here to keep the discussion specific to one case at a time. Otherwise, we might get lost very quickly! |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
I would suggest we need to clarify it one way or another. If we can build the behavior on target attributes, I think that would be preferred tbh! |
Beta Was this translation helpful? Give feedback.
-
Discussion around U10AsyncAPI + JSON Schema double contradicting formats for remote references Let's try to have all the discussions related to this use case below here to keep the discussion specific to one case at a time. Otherwise, we might get lost very quickly! |
Beta Was this translation helpful? Give feedback.
-
|
This is essentially the same as the previous case from a target attribute hinting point of view. The forthcoming JSON Schema + OpenAPI media type RFC addresses this sort of scenario directly. The other part of this is that |
Beta Was this translation helpful? Give feedback.
-
But if the version is not part of the media type, how will implementations know what they are loading? Especially if schemas don't define |
Beta Was this translation helpful? Give feedback.
-
|
You can use a media type parameter, but That was one of many reasons that we introduced vocabularies in 2019-09. In 2019-09 and 2020-12, even if the meta-schema URI is not recognized, the core vocabulary declared in Unraveling this is a topic of much debate as we work to register the media type. It is also a major factor in the JSON Schema spec development lifecycle proposal. In the near future, we do not want people to be thinking about JSON Schema "versions" as this situation of a trail of distinct draft versions, each of which requires separate support, is not sustainable. As you might imagine, this is a very complex (and urgent) discussion. |
Beta Was this translation helpful? Give feedback.
-
|
@jonaslagoni I have literally been spending this morning surveying various non-JSON Schema JSON Reference implementations and compiling use cases, so this is great and timely for me! To add a wrinkle: JSON Schema will support direct use of IRIs in the next release (instead of requiring them to be converted to URIs by punycoding, etc.), and I expect IRI usage will increase over time. Of course that does not require anyone to use IRIs that are not valid URIs without conversion, but it's worth keeping in mind. |
Beta Was this translation helpful? Give feedback.
-
Even more reason why the following requirement is needed: I.e. the tool needs to be able to adapt to various standards 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Yes, fortunately RFC3987 literally says "apply the RFC3986 algorithm". I'm not aware of any situation regarding URIs/IRIs that is not covered by RFC 3986, though. Do you have a specific example in mind? |
Beta Was this translation helpful? Give feedback.
-
No I have no experience with the standard 🙂 I just wanted to point out that from an architectural perspective, the tool should not be completely tied to one or another resource identifier standard. Of course not more than what the standards define, not saying it should be built to support another standard for JSON Schema, but in general whenever a reference is encountered, it should be based on of the context how it's processed. |
Beta Was this translation helpful? Give feedback.
-
It's hard to have different referencing and identifier standards interact. Most things use URIs/IRIs (they're equivalent for our purposes in this discussion, as every IRI can be encoded as a URI). If, for example, something used plain UUIDs, how would a URI-based system interact with that? How would the plain UUID system interact with the URI-based system? There's a solution to this from the URI side which is that the It is probably possible to accommodate this, as it's arguably a generalization of needing non-fragment ways to address the internals of resources without a defined fragment syntax, but it would be tricky. Things are a lot easier as URIs, and it's probably better to just define an appropriate URI scheme so that URIs can be used throughout than to stitch together arbitrary systems. But that's just my initial thought, I haven't really tried to solve the problem. |
Beta Was this translation helpful? Give feedback.
-
Note also that putting JSON Schema annotations adjacent to |
Beta Was this translation helpful? Give feedback.
-
|
Just copying from the PR 🙂
Yes, but of course not for all formats. We probably have to add this as a requirement 😄
It is only from a tooling perspective. Because, as far as we can determine, the use-case is not to know it's a reference but what that referenced resource is, so you can iterate/interact with the resolved resource. Especially when it comes to parsers.
Should the tool take that into account if the standard says no-adjacent keywords are allowed? I would say no... Otherwise, what is that requirement doing there? Cause you might as well define those keywords within the referenced resource or maybe even wrap the reference within We could, of course, start a discussion about whether there should be options to slightly alter the restrictive behavior i.e. that keywords may not be adjacent to |
Beta Was this translation helpful? Give feedback.
-
I'll have more to say about this hopefully sometime this week, but the short answer is that there's no need to allow keywords adjacent to Whether or not that is true for any other |
Beta Was this translation helpful? Give feedback.
-
|
Oh, and if you weren't aware, the forthcoming YAML media type RFC directly addresses JSON compatibility and using JSON Pointer fragments with YAML. |
Beta Was this translation helpful? Give feedback.
-
|
Can you elaborate a little on use cases for this tool and what useful information you expect it to provide, or functions you expect it to perform? One of the things to remember is that
JSON Schema does not use JSON Reference. It was dropped in draft-05 because it's incompatible with the behavior of URI References, as described in RFC3986, and this is a problem inherent to the idea of a JSON Reference standard. It might seem like you could write a tool that will transparently dereference $ref references, but URI References are just too powerful to be able to reason about without a parser specific to the media type/format in question. In all drafts of JSON Schema with $ref, you cannot generally substitute the reference for the document in JSON Schema, this might change the base URI that a URI Reference in the document is relying on. At the very least you have to add or change the If you describe the specific contexts where this library is being deployed and what answers it's providing for you, I can suggest some better alternatives. |
Beta Was this translation helpful? Give feedback.
-
Particularly when managing sets of (possibly cyclicly) linked documents is not the primary purpose of any of these specifications. It's just a thing that needs to be supported to handle any non-trivial set of structured data. JSON Schema now defines semantics for combining the results of a
These keywords can also largely be pre-processed to build a cache of URI-identified resources and pre-resolve the relative URI-references in |
Beta Was this translation helpful? Give feedback.
-
The only JSON Schema specific about them, is they are defined within that specific specification 😄 And there is nothing protective of the keywords, so they could mean something else in another spec. This is where context becomes paramount, that the library knows which type of input (standard) it has to comply with - and where.
Yep, you hit the nail right on the head I think 😄 Some of these tricky parts might be needed to specify in the standards themselves. Some may just need to be mapped out and explained more accurately then what the specs offer. Still TBD. |
Beta Was this translation helpful? Give feedback.
-
Yes, I have many thoughts on these points, stay tuned :-) As noted in the referencing and linking PR, we are discussing all of this within the JSON Schema core team. However, one team member is OOTO and we need to wait until they can weigh in before we can share anything publicly as a proposal from the org. |
Beta Was this translation helpful? Give feedback.
-
But they're defined in a way that cannot be generalized: they aren't merely reserved property names, they're schema keywords and only work where a schema is expected. The parser must be able to distinguish |
Beta Was this translation helpful? Give feedback.
-
|
@awwright I think it's do-able. It's not necessary to make every usage completely interoperable. As @jonaslagoni noted, there's a division of responsibility between the generic referencing functionality and context-specific behavior. A tool that replaces the entire object containing |
Beta Was this translation helpful? Give feedback.
-
|
I feel like referencing use cases break down into three broad categories, and I'm curious if others agree. All of these are defined across a set of documents. Documents that support "Same behavior" in this context means that whatever the underlying document set is supposed to do has the same outcome. For JSON Schemas, this means the same validation outcome, and the same annotations (except that the evaluation path can vary a bit as it shows where Document transformationsTake a document set and produce another document set with the same behavior. This has several variations:
There are probably some others I'm forgetting, but those are the main ones. In-memory data structuresThis involves using lazy proxy objects of some sort so that you can traverse the overall structure in memory as if it were a single big (possibly infinite/cyclic) resource. There are two variations: making the reference totally invisible (JSON Reference complete object replacement model) or making the reference the proxy so you would still see a Caching and serving reference targets"Serving" here might be as simple as "there's a JS object/Python dict/Perl hash/whatever mapping URIs to parsed resources". This is even more useful with
|
Beta Was this translation helpful? Give feedback.
-
|
Here is another question regarding tooling, interoperability, and base URIs. Since references can be relative, a tool fully supporting RFC 3986 defines a four-step precedence ladder for determining the correct base URI:
And of course, you might have a reason to supply a non-RFC3986-compliant base URI, e.g. because you would normally use the retrieval URI, but in your development environment technically the retrieval URI is Or maybe you're not even faking part of the RFC 3986 source ladder, but you want to supply a totally erroneous base URI to test some sort of error handling. So RFC 3986 normatively governs the determination of the "correct" base URI, regardless of what is or isn't written in some other specification. But as noted above, tooling requirements can include the need to go outside of that process.
So I have a two-part question about how a spec should handle this in terms of normative requirements: Question 1: what normative requirements, regardless of level, should exist?
Question 2: what level of requirement should it be (assuming you chose a normative requirement)
I have had some surprisingly intense debates on this topic, so I thought I'd see what expectations folks who are focused on interoperable tooling have. Because in my view, RFC 3986's normative requirements for base URIs alone do not impose clear or consistent direct requirements on tooling, and my perusal of |
Beta Was this translation helpful? Give feedback.
-
|
My main priority is that we should never end up in a case where a document works in one tool and not in another where both say they are spec compliant because you all of the sudden have a feature in one of the tools that the other is not aware of. For example, this is somewhat happening to bundling, as if you create your schema documents with this feature in mind, you cannot take any off-the-shelf reference implementation that says they are draft-7 compliant, because not everyone will even know what bundling is (especially if they haven't followed up with supporting never schema version). So depending on that feature becomes a huge problem down the line (unless the tooling ecosystem really is mature enough to provide you with implementations that understand it, so you don't end up having to create your own). So because of that, I would say it should be as restrictive as necessary to not end up in such a situation as described above. Not sure if this is the answer you were looking for @handrews? |
Beta Was this translation helpful? Give feedback.
-
|
To me that says there ought to be a normative requirement that is at least a SHOULD with clear guidance on when it might be acceptable to disregard it. It does not answer whether the requirement should be for a source-agnostic base URI (option 2) or if it should mimic the RFC 3986 precedence ladder plus override (option 4), but it's OK if you don't have an opinion on that. It does seem clear to me that option 1 (no requirement) and option 3 (only RFC 3986-compliant sources) wouldn't work assuming you might have use cases that don't fit RFC 3986. Of course you could pass another base URI in as if it were an RFC 3986-sourced one, but then you have to pick which source to fake and that seems like a needless complication to me. |
Beta Was this translation helpful? Give feedback.
-
|
OK, I have finally submitted a PR for a standalone JSON Referencing and Identification proposal at the JSON Schema Org's new referencing repository. We are hosting this discussion but encouraging submissions and participation from other projects. There will be at least one other proposal from another JSON Schema team member (and @jonaslagoni if you want to submit the one from asyncapi/spec#825 you are most welcome to do so! Also paging @whitlockjc ). The JRI proposal should address the concerns here, although some things are not entirely resolved. It also tries to bridge the various usages and use cases, so it's more complex in an effort to accommodate a variety of uses. Some of the ideas (like media type parameters for indicating Please feel free to open issues in that repo along with continuing discussions here. |
Beta Was this translation helpful? Give feedback.
-
In AsyncAPI we have issues with references, where basic use cases work, but everything gets fussy as soon as you dive into more advanced cases. This makes the life of building tools around them very tricky if not almost impossible.
We have been unable to find a library that can fully handle the use-cases and requirements so we can integrate it with our JS parser (Currently we use json-schema-ref-parser), because of the very use-cases and requirements highlighted here.
This proposal revolves around how we can build a core building block (across specifications) that makes sure it's easy to work with documents that contain references, where each reference follows different standards. This spans across (and is not limited by) the standards AsyncAPI, JSON Schema, and OpenAPI. It will be one of the very core building blocks for implementing parsers and other processing tools.
This discussion goes hand in hand with asyncapi/spec#825 for AsyncAPI 3.0 and this should be used to further progress the discussion points and requirements for the specification changes.
This is a first step effort to map out requirements to a library, that is not to say that we have to build this ourselves, cause if we can leverage other already existing tools that would probably be preferred! However, if that is the case, they MUST be willing to adapt to these requirements. Whether to do A or B is NOT the focal point in this discussion as this is ONLY about the requirements and use cases of said tool.
Requirements
These are the requirements that I expect are needed for any reference tool:
The API of the library also has different requirements, which come down to in which context it's used. This is what I can seem to gather:
Reference specifics
I want to give a quick overview of the different reference behaviors we see across standards (including those I know the most AsyncAPI, OpenAPI, and JSON Schema). If you find anything that is wrong or would make sense to redefine feel free to point it out 👌
AsyncAPI 2.x
Resource: https://www.asyncapi.com/docs/reference/specification/v2.4.0#referenceObjectURI resolvement: Base URI is not possible to set.
Resource resolution: Replace the entire containing object with the resolved resource. Uses references through the reference object, that in turn is relying on JSON Reference.
AsyncAPI 3.x
Referencing JSON (Avro, JSON Schema) and non-JSON data (GraphQL, Protobuf, XSD), are still TBD, discussion happening in https://github.com/asyncapi/spec/pull/825OpenAPI 3.0
Resource: https://spec.openapis.org/oas/v3.0.3URI resolvement: Uses the server object for the base URI for relative references.
Resource resolution: Replace the entire containing object with the resolved resource. Uses JSON Reference
OpenAPI 3.1
Resource: https://spec.openapis.org/oas/v3.1.0URI resolvement: Have multiple resolvement statements based on where the reference is located. Relative references within Reference Objects, PathItem Object $ref fields, Link Object operationRef fields, and Example Object externalValue fields, are resolved using the referring document as Base URI. A special case is resolving the fully qualified URL for relative references, which revolves around external documentation, license, contact and oath flow urls (all properties that is a url).
Relative references in Schema Objects, including any that appear as $id values, use the nearest parent $id as a Base URI, as described by JSON Schema Specification Draft 2020-12.
Resource resolution: custom behavior (but somewhat similar to JSON Reference), cannot find anything specific about the resource resolution.
Must adhere to Bundling (a feature in draft 2020-12): Whenever OpenAPI 3.1 documents need to resolve references, the tool need to understand the bundling behavior, loading all schemas in
/components/schemasbefore trying to resolve the resource.JSON Schema draft 6 + 7
Resource draft 6: https://datatracker.ietf.org/doc/html/draft-wright-json-schema-01
Resource draft 7: https://datatracker.ietf.org/doc/html/draft-handrews-json-schema-01
URI resolvement: Uses the nearest
$idas the base URI for relative references.Resource resolution: Replace the entire containing JSON object with the resolved resource.
Bundling: Even though bundling was not well defined until draft 2019-09, it should be the default behaviour for handling schemas < draft 2019-09. It should however be possible to disable this behavior.
JSON Schema draft 2019-09
JSON Schema draft 2019-09: https://json-schema.org/specification-links.html#draft-2019-09-formerly-known-as-draft-8URI resolvement: Uses the nearest
$idas the base URI for relative references. Uses JSON Pointer for URI fragments.Resource resolution: Replace the reference value with the resolved Schema object. Adapts to bundling.
Bundling: Implementations must lookup
$refin$defsbefore trying to access the resource externally.JSON Schema draft 2020-12
JSON Schema draft 2020-12: https://datatracker.ietf.org/doc/html/draft-bhutton-json-schema-00URI resolvement: Uses the nearest
$idas base URI for relative references.Resource resolution: Replace the string reference with the resolved Schema object. Must adhere to bundling.
Bundling: Implementations must lookup
$refin$defsbefore trying to access the resource externally.Use-cases
Here is a list of use-cases that I have been able to gather, that either have undefined/unknown behavior (unknown meaning the standard does not define the expected behavior) or behaviour we know and that we would possibly want to change (or not).
If any of these use cases seem wrong based on your experience, please point out where in the standards the behavior is explained so we can keep it as organized as possible.
known behavior cases
These are all the cases where I expect that we can conclude the expected behavior based on the specifications.
U3 - AsyncAPI + JSON Schema < draft 2019-09 with bundling
Discussion can be found here: #485 (comment)
Original question and answer: As far as I can understand it was not until draft 2019-09 that it enabled compound schemas (i.e. bundled schemas where any within
#/components/schemaswill automatically be loaded in implementations and so it does not matter that there exist no remote resource onhttps://my-schema-registry.org/UserSignedUpas it will be locally matched with#/components/schemas/UserSignedUpbecause of the$id), i.e. this is not supported with draft-7. Expected behavior: Failure to locatehttps://my-schema-registry.org/UserSignedUp.Expected behavior: Bundling should be defaulted to but can be turned off through options for schema formats JSON Schema < draft 2019-09 because it was not well defined until draft 2019-09, so it is hard put your foot down and say this MUST be the expected behavior.
U4 - AsyncAPI + JSON Schema >= draft 2019-09 with bundling and internal definitions
Discussions found here: #485 (comment)
If bundling is allowed, what is the accurate behavior for when schemas inline definitions within AsyncAPI? Take notice how the properties in in
#/$defs/UserSignedUphas type number instead of string. Would the message payload validate accurately against{displayName: 0, email: 0}or{displayName: "myUsername", email: "[email protected]"}?Expected behavior: By default duplicate URIs should raise an error, which was defined in newer JSON schema version.
U5 - AsyncAPI + JSON Schema contradicting formats
Discussions found here: https://github.com//discussions/485#discussioncomment-3788937If
$schemamust be adhered to, how can tooling interpret the following AsyncAPI document, where the formats are contradicting? Would the payload schema be interpreted as a draft-7 or draft-2019/09 schema? Who has the highest precedence? Who defines that this is the desired behavior?Expected behavior: Because
$schemaallows you to define the schema for the embeded resource,$schema"overwrites" whatever is defined withinschemaFormat.U7 - OpenAPI + JSON Schema contradicting base-URI's
Discussion found here: https://github.com//discussions/485#discussioncomment-3788946I am not 100% sure this is even a contradiction, are the servers url used for relative references in URIs? Here I am assuming it is, cause it creates this weird behavior. I think one could argue two things here, which is 1. the base-URI is determined by the closest application keyword. I.e. once we reach the JSON Schema document, its reference is determined by the
$id. 2. the OpenAPI base URI triumphs over the nested base URIs. I honestly think that the desired behavior is always 1. However, from reading the specs, I cannot whole-hearted say that is the expected behavior.Expected behavior: The server url has no effect as base URI for the schema object, as that definition has no baring on that part.
./UserSignedUp.yaml
./DisplayNames.yaml
Unknown behavior cases
These are all the cases where I simply cannot seem to find the information about what is supported and expected behavior when reading the standards from a tooling perspective.
U1 - AsyncAPI + JSON Schema >= draft 2019-09 with bundling
Discussion found here: https://github.com//discussions/485#discussioncomment-3788915Is bundling allowed within JSON Schema >= draft 2019-09 in AsyncAPI? Should it enable bundling within AsyncAPI so the reference
https://my-schema-registry.org/UserSignedUpwould accurately be loaded from#/components/schemas/UserSignedUpbefore trying to fetch it remotely?U2 - AsyncAPI + JSON Schema >= draft 2019-09 with bundling and relative reference
Discussion found here: https://github.com//discussions/485#discussioncomment-3788920If U1 is allowed, then this would subsequently also be allowed because the base-URI is fairly straightforward to resolve. With >= draft 2019-09 it enables bundling within AsyncAPI and should accurately load the schemas without reaching out externally to access the schema
https://my-schema-registry.org/Userand accurately make the relative reference./Useruse the base URI ofhttps://my-schema-registry.org/UserSignedUp(https://my-schema-registry.org) to resolve it tohttps://my-schema-registry.org/User.U6 - AsyncAPI using OpenAPI Schema documents which expect a base URI
Discussion found here: https://github.com//discussions/485#discussioncomment-3788941Since AsyncAPI does not have a way to specify base URIs for relative reference resolvement, how do you reuse your schema documents across specs?
asyncapi.yaml
openapi.yaml
./UserSignedUp.yaml
./DisplayNames.yaml
I know I might be pushing the limits of what a user might do because if they just sat an
$idfor theUserSignedUpschema you would not have had any issues.Maybe tooling always needs to provide an option for specifying the base URI around the application itself. So when used in AsyncAPI you still have the option of defining the base URI to
https://my-schema-registry.org.U8 - AsyncAPI + non-JSON references
Discussion found here: https://github.com//discussions/485#discussioncomment-3788951How should non-JSON references be interpreted from a tooling side? Is that up to the individual implementor? If so, how can you make sure that reference fragments are properly upheld as expected? i.e. what if the implementer converted something to an array, where the user and fragment were defined as an object? Or an entirely different interpretation?
Example here I assume I can access the message through
#UserSignedUphowever what if the internal JSON representation is the following:{ "messages": [ { "name": "UserSignedUp", "properties": { "displayName": { "type": "string", "id": 0 }, "email": { "type": "string", "id": 0 } } } ] }Then I would have to access it as
#/messages/0(just an example, dont like this conversion, but it's just to show it might be interpreted differently because we dont following a "standard").U9 - AsyncAPI + JSON Schema contradicting formats for remote references
Discussion found here: https://github.com//discussions/485#discussioncomment-3788954For remote references, if it's the media type that determines how a resource is processed, what happens when they are contradicted?
The mime-type received from
https://schema_registry.com/UserSignedUpisapplication/schema+json;version=draft-07, contradicting what was defined within the AsyncAPI document. Which one should tooling use?U10 - AsyncAPI + JSON Schema double contradicting formats for remote references
Discussion found here: https://github.com//discussions/485#discussioncomment-3788956Extending U9, with the specific schema that defined its own
$schemaformat.The mime-type received from
https://schema_registry.com/UserSignedUpisapplication/schema+json;version=draft-07, and the$schemadefined within the schema itself ishttps://json-schema.org/draft/2020-12/schema. Double contradicting what the AsyncAPI document defined and what the remote server returned.One could say this is just badly managed schemas, but how can tooling and users know what to do in this situation?
Relevant resources
These are some of the relevant resources that are somewhat related to this.
$ref/$idjson-schema-org/json-schema-spec#724$idconform to RFC 3986 suggestion for base URI elements json-schema-org/json-schema-spec#729Beta Was this translation helpful? Give feedback.
All reactions