feat: add new custom resolver to fetch the reference from private repo #1810

AayushSaini101 · 2025-06-21T12:35:01Z

Resolves: #1796

Create a custom resolver to fetch the reference from the private repo
Add new command auth add for storing the repo name and repo token.
Tests and documents need to be updated

…and add new command auth add

github-actions · 2025-06-21T12:36:48Z

Changeset has been generated for this PR as part of auto-changeset workflow.

Please review the changeset before merging the PR.

---
'@asyncapi/cli': minor
---

feat:  add new custom resolver to fetch the reference from private repo

- 0213926: Add new custom resolver to fetch the reference from private template and add new command auth add
- 26037eb: Merge remote-tracking branch 'remote/master' into 1796-b
- 6473326: chore: separate cli and api

Signed-off-by: Shurtu-gal <[email protected]>
- d484f2f: update
- ab58365: remove invalid changes
- 932c09c: update package-lock.json
- ec5eb66: update package-lock.json
- 7e4b523: fix github jobs
- c7501a1: Fix elint issue
- 925b90a: remove uncessary files
- 5d651f7: remove invalid file
- 5d76066: update-changeset
- 5d89ce7: update-doc
- 3461288: update-suggestions
- 4f63af9: add new testcase
- 5d83ac7: add new testcases
- f1a4d39: add new test case for the validationService
- af11ce4: update test case
- 824bdb9: update changeset
- cf4d070: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 150865c: update validation service for the github
- 4c5be56: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- fcc864d: Update src/apps/api/middlewares/validation.middleware.ts

Co-authored-by: Fran Méndez <[email protected]>
- 2f69485: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 5328d62: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 407bacc: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 3b7159b: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- f601cf7: Update src/domains/services/validation.service.ts

Co-authored-by: Fran Méndez <[email protected]>
- 8943766: updateValidationservice
- aa37d14: remove invalid comment
- 3b882ed: Add new custom resolver to fetch the reference from private template and add new command auth add

# Conflicts:
#	package-lock.json
#	src/core/parser.ts
- 811bc08: update

# Conflicts:
#	asyncapiTest.yml
#	index.js
#	src/apps/api/middlewares/validation.middleware.ts
#	src/apps/cli/commands/diff.ts
#	src/apps/cli/commands/generate/models.ts
#	src/apps/cli/commands/validate.ts
#	src/domains/services/config.service.ts
#	src/domains/services/validation.service.ts
- bb0fb6c: remove config service test file
- 32298a9: remove invalid statements
- ad3b3b8: add missing dependencies
- 7662fca: remove warnings
- 40b4b50: update validation-service
- d7db432: remove invalid file
- 98dfd4c: fix lint issue
- 75dda7b: update validation-service
- 5c23530: fix lint issue

If you are a maintainer or the author of the PR, you can change the changeset by clicking here

Tip

If you don't want auto-changeset to run on this PR, you can add the label skip-changeset to the PR or remove the changeset and change PR title to something other than fix: or feat:.

changeset-bot · 2025-06-21T12:36:51Z

🦋 Changeset detected

Latest commit: d6d0402

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@asyncapi/cli	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

AayushSaini101 · 2025-06-21T12:44:54Z

Tests and documentation need to be updated, i will update after getting ack from this approach : )

Spec containing private repo:

Trying to parse using the existing version of the asyncapi cli:

New feature:

Add auth value for the github token:

Repo URL format: https://github.com/AayushSaini101/AayushSaini101/tree/main

Trying to fetch the private ref:

cc: @fmvilas @Souvikns @Shurtu-gal

fmvilas · 2025-06-21T13:30:16Z

src/core/parser.ts

+  schema: 'https',
+  order: 1,
+
+  canRead: (uri: URI) => uri.hostname() === 'raw.githubusercontent.com',


Why checking for GitHub here? 🤔 References can be in many places, not just GitHub, and even in such a case the GitHub deployment might be in a private domain, which is pretty common in medium to large companies.

fmvilas · 2025-06-21T13:33:53Z

src/core/parser.ts

+
+  read: async (uri: URI) => {
+    const url = uri.toString();
+    const normalized = normalizeRawGitUrl(uri);


What if the URL is not pointing to a GitHub repo? This could perfectly be a Schema Registry URL, an S3 bucket, some random URL, or even a GitLab or BitBucket URL.

@fmvilas, we can limit this PR for the GitHub repo only because we know proper mapping for this for example, when the user provide the URL in this format 'https://github.com/AayushSaini101/AayushSaini101/blob/main/user-signedup.yaml#/payload', we know we have to modify the URL in this format n https://api.github.com/repos/${owner}/${repo}/contents/${path}?ref=${branch} to obtain the data from the private template, but we don't about the Gitlabm. or other BitBucket and other services modification, we can start with the github private url approach, then we can start updating bit by bit according to the requirements from the users thanks

mmm 🤔 this is being applied to all URLs and normalizeRawGitUrl assumes it's a GitHub URL. I'm fine with only doing the mapping for GitHub for now but other URLs shouldn't be modified. You know what I mean?

yes, at this time we can consider with GithubURL only, in future we can update according the requirements, like for Gitlab and other services based on the requirements cc: @fmvilas

Just to be sure we're understanding each other. If I'm understanding it correctly, right now we're assuming all URLs are GitHub URLs, which is wrong because that would make parser stop working with custom URLs. In the normalizeRawGitUrl function we should be checking if it's actually a GitHub URL, otherwise, leave it intact. Mappings for GitLab and Bitbucket can wait but breaking all URLs except the GitHub ones is definitely not acceptable.

yes, i need to change the functionnaming properly i will make it generic so that we can normalise other url as well in the future based on the requirements

src/core/parser.ts

AayushSaini101 · 2025-07-02T06:33:32Z

/u

sonarqubecloud · 2025-07-02T06:34:26Z

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

AayushSaini101 · 2025-07-05T16:18:15Z

@fmvilas For the private Github URL should be in this format: like https://api.github.com/repos/AayushSaini101/AayushSaini101/contents/user-signedup.yaml, not direct URL like we simple copy the path of the URL from the private repo, this is the github url, not sure about the private repo, in short we need to compare each and every url and create special check to find whether it is matching with the stored pattern or not, we can start with the github and gitlab pattern then we can slowly slowly add the comparison format as we grow thanks

aeworxet · 2025-07-21T10:32:29Z

@asyncapi/bounty_team

AayushSaini101 · 2025-08-04T06:13:01Z

/u

fmvilas · 2025-08-04T19:39:51Z

@fmvilas For the private Github URL should be in this format: like https://api.github.com/repos/AayushSaini101/AayushSaini101/contents/user-signedup.yaml, not direct URL like we simple copy the path of the URL from the private repo, this is the github url, not sure about the private repo, in short we need to compare each and every url and create special check to find whether it is matching with the stored pattern or not, we can start with the github and gitlab pattern then we can slowly slowly add the comparison format as we grow thanks

@AayushSaini101 I'm not sure I follow. I think you can translate from one to another, right? For instance, if I give you https://github.com/AayushSaini101/a-repo/blob/master/asyncapi.yaml (which is the URL I can copy from the browser), you can transform it to https://api.github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml or whatever format you need it to be, right? The idea is to make it really easy for the user.

Likewise, if my company has a Github installation behind the https://source.mycompany.com URL, then a URL like https://source.mycompany.com/AayushSaini101/a-repo/blob/master/asyncapi.yaml should also be possible to resolve.

In short, picture this, you're working at a company and have an AsyncAPI or JSON/YAML file in a private repo. You want to reference it using $ref and it should just work out of the box. Sure, the user has to provide the necessary details to resolve these URLs but it should be fairly easy once it's configured.

AayushSaini101 · 2025-08-05T07:10:40Z

github.com/repos/AayushSaini101/a-repo/contents/asyncapi.yaml

yes, i can tranform/configured the URL for the Github because i know the format for the accepted URL in the case of the Github, but for the case of another services like registry or any private registry what should be the correct URL that we need to consider for the custom resolver, that's why for this issue we can enable feature Github then in future according to the requirements we can add bit by bit @fmvilas

AayushSaini101 · 2025-08-11T12:57:52Z

/u

sonarqubecloud · 2025-10-19T12:03:50Z

Quality Gate passed

Issues
17 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
2.0% Duplication on New Code

See analysis details on SonarQube Cloud

AayushSaini101 · 2025-10-19T12:06:28Z

Need to open new PR, squashing and cleaning the commits something got wrong : (

Add new custom resolver to fetch the reference from private template …

0213926

…and add new command auth add

AayushSaini101 requested review from Amzani, Shurtu-gal, Souvikns and asyncapi-bot-eve as code owners June 21, 2025 12:35

github-project-automation bot added this to CLI - Kanban Jun 21, 2025

github-project-automation bot moved this to To Triage in CLI - Kanban Jun 21, 2025

AayushSaini101 changed the title ~~Add new custom resolver to fetch the reference from private repo~~ feaat: add new custom resolver to fetch the reference from private repo Jun 21, 2025

AayushSaini101 changed the title ~~feaat: add new custom resolver to fetch the reference from private repo~~ feat: add new custom resolver to fetch the reference from private repo Jun 21, 2025

chore: add changeset for PR asyncapi#1810

c685ac2

fmvilas requested changes Jun 21, 2025

View reviewed changes

github-project-automation bot moved this from To Triage to Backlog in CLI - Kanban Jun 21, 2025

Merge branch 'master' into 1796-b

181dbb9

asyncapi-bot added the bounty AsyncAPI Bounty program related label label Jul 21, 2025

aeworxet added this to Bounty Program Jul 21, 2025

aeworxet moved this to In Progress in Bounty Program Jul 21, 2025

Merge branch 'master' into 1796-b

82ac26b

aeworxet mentioned this pull request Aug 6, 2025

[FEATURE] Pass authentication/authorization to $ref HTTP resolvers #1796

Closed

2 tasks

Merge branch 'master' into 1796-b

0a4c2ed

remove invalid statements

32298a9

AayushSaini101 force-pushed the 1796-b branch from 3b322ce to 32298a9 Compare October 19, 2025 11:15

asyncapi-bot pushed a commit to AayushSaini101/cli that referenced this pull request Oct 19, 2025

chore: add changeset for PR asyncapi#1810

ace5aa8

add missing dependencies

ad3b3b8

AayushSaini101 force-pushed the 1796-b branch from ace5aa8 to ad3b3b8 Compare October 19, 2025 11:21

asyncapi-bot pushed a commit to AayushSaini101/cli that referenced this pull request Oct 19, 2025

chore: add changeset for PR asyncapi#1810

b3c7f49

remove warnings

7662fca

AayushSaini101 force-pushed the 1796-b branch from b3c7f49 to 7662fca Compare October 19, 2025 11:32

asyncapi-bot pushed a commit to AayushSaini101/cli that referenced this pull request Oct 19, 2025

chore: add changeset for PR asyncapi#1810

c97944d

update validation-service

40b4b50

AayushSaini101 force-pushed the 1796-b branch from c97944d to 40b4b50 Compare October 19, 2025 11:43

asyncapi-bot pushed a commit to AayushSaini101/cli that referenced this pull request Oct 19, 2025

chore: add changeset for PR asyncapi#1810

c8f8a89

remove invalid file

d7db432

AayushSaini101 force-pushed the 1796-b branch from c8f8a89 to d7db432 Compare October 19, 2025 11:45

asyncapi-bot pushed a commit to AayushSaini101/cli that referenced this pull request Oct 19, 2025

chore: add changeset for PR asyncapi#1810

523a3b0

fix lint issue

98dfd4c

AayushSaini101 force-pushed the 1796-b branch from 523a3b0 to 98dfd4c Compare October 19, 2025 11:56

asyncapi-bot pushed a commit to AayushSaini101/cli that referenced this pull request Oct 19, 2025

chore: add changeset for PR asyncapi#1810

b53fb27

AayushSaini101 added 2 commits October 19, 2025 17:31

update validation-service

75dda7b

fix lint issue

5c23530

AayushSaini101 force-pushed the 1796-b branch from b53fb27 to 5c23530 Compare October 19, 2025 12:02

chore: add changeset for PR asyncapi#1810

d6d0402

AayushSaini101 closed this Oct 19, 2025

AayushSaini101 deleted the 1796-b branch October 19, 2025 12:04

github-project-automation bot moved this from Backlog to Done in CLI - Kanban Oct 19, 2025

github-project-automation bot moved this from In Progress to Completed in Bounty Program Oct 19, 2025

asyncapi-bot removed the bounty AsyncAPI Bounty program related label label Oct 21, 2025

aeworxet removed this from Bounty Program Oct 21, 2025

Uh oh!

feat: add new custom resolver to fetch the reference from private repo #1810

feat: add new custom resolver to fetch the reference from private repo #1810

Uh oh!

Conversation

AayushSaini101 commented Jun 21, 2025

Uh oh!

github-actions bot commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changeset has been generated for this PR as part of auto-changeset workflow.

Uh oh!

changeset-bot bot commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

AayushSaini101 commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New feature:

Uh oh!

fmvilas Jun 21, 2025

Choose a reason for hiding this comment

Uh oh!

fmvilas Jun 21, 2025

Choose a reason for hiding this comment

Uh oh!

AayushSaini101 Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

fmvilas Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

AayushSaini101 Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fmvilas Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

AayushSaini101 Sep 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AayushSaini101 commented Jul 2, 2025

Uh oh!

sonarqubecloud bot commented Jul 2, 2025

Quality Gate passed

Uh oh!

AayushSaini101 commented Jul 5, 2025

Uh oh!

aeworxet commented Jul 21, 2025

Uh oh!

AayushSaini101 commented Aug 4, 2025

Uh oh!

fmvilas commented Aug 4, 2025

Uh oh!

AayushSaini101 commented Aug 5, 2025

Uh oh!

AayushSaini101 commented Aug 11, 2025

Uh oh!

sonarqubecloud bot commented Oct 19, 2025

Quality Gate passed

Uh oh!

AayushSaini101 commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

github-actions bot commented Jun 21, 2025 •

edited

Loading

changeset-bot bot commented Jun 21, 2025 •

edited

Loading

AayushSaini101 commented Jun 21, 2025 •

edited

Loading

AayushSaini101 Sep 2, 2025 •

edited

Loading

AayushSaini101 commented Oct 19, 2025 •

edited

Loading