[FEATURE] `Bundler` to have more flexible and configurable behavior for circular refs

[412b]

Why do we need this improvement?

Despite being called bundler it does dereferencing, which blocks the use of it for schemas with circular references.
json-schema-ref-parser used under the hood has mutiple options for handling circular refs and even a dedicated bundle function for actually bundeling.
moreover, further down the asyncapi stack circular refs are not a problem, e.g. modelina can handle schemas with circular refs and generate code for those.

How will this change help?

it will unblock the use of bundler for cases with circular refs and make the tool more close functions-wise to its name.

Screenshots

No response

How could it be implemented/designed?

default behavior can remain the same, and additional option may enable a behavior of really just bundling
also mixed dereference via dereference.circular=ignore can be achieved.

from the first look at the code base it can be a single additional cli arg (e.g. named strategy) describing the desired behaviours:

• dereference all (default and current behavior)
• dereference circular
• dereference all except circular
• bundle (via using json-schema-ref-parser's bundle)

no real complexity of the change, but quite some additional value for the tool.

🚧 Breaking changes

No

👀 Have you checked for similar open issues?

• I checked and didn't find a similar issue

🏢 Have you read the Contributing Guidelines?

• I have read the Contributing Guidelines

Are you willing to work on this issue?

Yes I am willing to submit a PR!

[github-actions]

Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

[aeworxet]

@412b
Could you please provide examples of the original schema(s) and the expected results for each case?

[412b]

@412b Could you please provide examples of the original schema(s) and the expected results for each case?

sure, it's a more of expected behavior here, 'cause I don't have time to test it in real-life, but conceptually it should work as follows:
Input
asyncapi.yml

asyncapi: 3.0.0
info:
  title: Test model
  version: 1.0.0
components:
  messages:
    btree:
      title: BTree
      payload:
        $ref: "./btree.yaml"

btree.yaml

$schema: "https://json-schema.org/draft-07/schema#"
title: BTree
type: object
properties:
  root:
    $ref: "./node.yaml"

node.yaml

$schema: "https://json-schema.org/draft-07/schema#"
title: Node
type: object
properties:
  left:
    $ref: "#"
  right:
    $ref: "#"

Result

1. dereference all
   ReferenceError: Circular $ref pointer found at

2. dereference circular
   here I'm not 100% sure, how exactly the schema will be dereferenced, because docs don't give enough of examples

3. dereference all except circular

asyncapi: 3.0.0
info:
  title: Test model
  version: 1.0.0
components:
  messages:
    btree:
      title: BTree
      payload:
        $schema: "https://json-schema.org/draft-07/schema#"
        title: BTree
        type: object
        properties:
          root:
            $schema: "https://json-schema.org/draft-07/schema#"
            title: Node
            type: object
            properties:
              left:
                $ref: "#/components/messages/btree/payload/properties/root"
              right:
                $ref: "#/components/messages/btree/payload/properties/root"

4. bundle
   here I'm again not 100% sure of the output due too lack of docs, but should be pretty close to pt. 3

asyncapi: 3.0.0
info:
  title: Test model
  version: 1.0.0
components:
  messages:
    btree:
      title: BTree
      payload:
        $schema: "https://json-schema.org/draft-07/schema#"
        title: BTree
        type: object
        properties:
          root:
            $schema: "https://json-schema.org/draft-07/schema#"
            title: Node
            type: object
            properties:
              left:
                $ref: "#/components/messages/btree/payload/properties/root"
              right:
                $ref: "#/components/messages/btree/payload/properties/root"

examples are lacking precision of course, but in worst case I see only number of options being reduced in the PR, while writing the tests and checking the real-life behavior.

[412b]

thus it may even end up with only 2 options:

• all
• non-circular

[aeworxet]

@412b

Comments on your requests:

• dereference all (default and current behavior)

  • No changes are needed.

• bundle (via using json-schema-ref-parser's bundle)

  • Bundler uses onDereference function to save the properties x-origin. Therefore, to avoid introducing complex conditions - such as 'option bundle disables the possibility to save the x-origin property, so you can use only either one or the other' - bundle functionality can be implemented only after the onBundle callback, which has similar functionality to onDereference is implemented in @apidevtools/json-schema-ref-parser (in development since 2024-04-27).

• dereference all except circular

  • Functionality can be achieved by setting dereference.circular to ignore. Should a separate flag be introduced?

• dereference circular

  • There is no immediate answer on how to achieve this in this particular case; the topic needs to be researched.

[412b]

@aeworxet I tend to exclude bundle and introduce option to control dereference.circular.
that seems to be a good-trade-off between complexity and features.
I will spin up a PR for it soon.

[412b]

@aeworxet

real-life renderings via dereference:

• circular = true. breaks the toolchain, can be discarded completely due to anchors.

asyncapi: 3.0.0
info:
  title: Test model
  version: 1.0.0
components:
  messages:
    btree:
      title: BTree
      payload:
        $schema: https://json-schema.org/draft-07/schema#
        title: BTree
        type: object
        properties:
          root: &ref_0
            $schema: https://json-schema.org/draft-07/schema#
            title: Node
            type: object
            properties:
              left: *ref_0
              right: *ref_0
            x-origin: ./node.yml
        x-origin: ./btree.yml

• circular = 'ignore'. does nothing for example provided, can be discarded.

    asyncapi: 3.0.0
    info:
      title: Test model
      version: 1.0.0
    components:
      messages:
        btree:
          title: BTree
          payload:
            $ref: ./btree.yml

more useful results we're obtained by using bundle before dereference.

• circular = 'ignore'. perfect resolution, best option so far.

asyncapi: 3.0.0
info:
  title: Test model
  version: 1.0.0
components:
  messages:
    btree:
      title: BTree
      payload:
        $schema: https://json-schema.org/draft-07/schema#
        title: BTree
        type: object
        properties:
          root:
            $schema: https://json-schema.org/draft-07/schema#
            title: Node
            type: object
            properties:
              left:
                $ref: '#/components/messages/btree/payload/properties/root'
              right:
                $ref: '#/components/messages/btree/payload/properties/root'

there are only 2 side-effects of using bundle:

1. no x-origin, because nobody implemented the callback feature for bundle in json-schema-ref-parser
2. one test showed, that $ref changed, needs a bit of check, if the result is still valid.

the research is 90% of the PR, that I am ready to submit (does the feature need approval still?) and lead it to the finish line.

[aeworxet]

@412b

Commenting on the PR #202 here for better visibility.

• I have explicitly explained in [FEATURE] Bundler to have more flexible and configurable behavior for circular refs #201 (comment) why the bundle() function of @apidevtools/json-schema-ref-parser cannot be used in its current form. The usage of https://github.com/asyncapi/bundler/pull/202/files#diff-abe2818e9c29aadf84401cb7ef8286c7cbce9253eba09f6fa52e22ca4e57b17eR80-R81 breaks the functionality by obliterating historical $refs, preventing their values from being assigned to x-origin afterward, despite users' desires. If you need to use bundle(), please implement the onBundle callback in @apidevtools/json-schema-ref-parser, as outlined in Possibility to add onBundle callback? APIDevTools/json-schema-ref-parser#346, first.

• In the excludedPathMatcher() function, there are specified objects that MUST be ONLY Reference Objects according to the AsyncAPI Specification v3.0.0; therefore, they are excluded from any dereferencing. The messages property of the Channel Object is not one of them, so if the schema used for internal purposes contains a $ref, it can be freely dereferenced before being sent out.

• You don't jump over two major versions of a mission-critical package without consulting the known maintainers of the packages that use yours and the known target users.

Given these considerations, the only truly valuable change in the entire PR is the control of the behavior of dereference.circular. However, even in this case, you would still need to consult with the CLI maintainers - @Souvikns, @Shurtu-gal, and @AayushSaini101 - to clarify whether a separate option is required or if simply modifying the default behavior of the Bundler would be sufficient and would not break anything further down the chain.

In the future, please refrain from opening PRs or initiating coding efforts until everything is settled in plain words beforehand.

[412b]

@aeworxet let' go ver those one by one :)
1.

is implemented in @apidevtools/json-schema-ref-parser (in development since 2024-04-07).

that's a false statement, there's an issue created, there's no development as there's no PR.
it also intersects with your pt. 3, which means, that even in case of having callback supported the loop will get longer and longer.

breaks the functionality by obliterating historical $refs

here I fully agree, but there can be a compromise of keeping the exact historical behavior as well for default case and warning the user, that for circular refs x-origin is not supported. that is still better than no support for circular refs at all.

2. that's an easy revert. the change itself was born due to obvious declaration duplication. with your clarification it now looks more a question to the spec then.

3. somewhat touched, easy revert as well, but there were important changes related in particular to circular references. though concern is understandable.

given your answer above, it's obvious, that you haven't read the rationale, why bundle is a must for a good enough support of circular refs and why using only dereference is not going to work.

on a serious note: that all doesn't look like welcoming contributors, which the whole toolchain needs in my opinion.

[aeworxet]

Given the high demand for the functionality of the onBundle callback, I started coding it myself.
This functionality will resolve all the matters mentioned in this issue at once.

[aeworxet]

@412b

bundle method of @apidevtools/json-schema-ref-parser cannot currently be used also because it lacks the excludedPathMatcher callback, which the dereference method has. This absence will lead to the dereferencing of all $refs that are supposed to be only $refs (same as in previous comment), that is why I included it in my example code in APIDevTools/json-schema-ref-parser#346 (comment) when proposing the merge.

Therefore, delete the use of bundle in https://github.com/asyncapi/bundler/pull/202/files#diff-abe2818e9c29aadf84401cb7ef8286c7cbce9253eba09f6fa52e22ca4e57b17eR78-R82, please.

[412b]

@aeworxet bundle doesn't dereference local refs:

Bundles all referenced files/URLs into a single schema that only has internal $ref pointers. This lets you split-up your schema however you want while you’re building it, but easily combine all those files together when it’s time to package or distribute the schema to other people. The resulting schema size will be small, since it will still contain internal JSON references rather than being fully-dereferenced.

docs

thus bundle will replace external refs with internal refs only.
is there an edge-case, that you see there, that I don't?

[aeworxet]

bundle method still replaces existing local references with different ones.

To avoid breaking functionality for users, I have submitted a PR to the https://github.com/APIDevTools/json-schema-ref-parser repository that introduces the callback onBundle, which will address all matters mentioned in this issue at once.

Work on this issue will continue after that functionality is merged into @apidevtools/json-schema-ref-parser.