Skip to content

chore: improve OpenAPI spec generation and add dev docs #114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sallymoc wants to merge 2 commits into
base: main
Choose a base branch
from
Open

chore: improve OpenAPI spec generation and add dev docs #114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
15bae06
chore: move OpenAPI adjustments from Makefile to CI workflow
sallymoc Mar 5, 2026
c12b266
ci: add check to verify OpenAPI spec is up to date
sallymoc Mar 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions .github/workflows/api-docs-update-v2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,37 @@ jobs:
api-docs-update-trigger:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Apply OpenAPI adjustments
run: |
cd v2/api/archive-query-service/v2
cp query_services.openapi.yaml query_services.openapi.adjusted.yaml
yq -i --indent 4 'del(.paths."/health")' query_services.openapi.adjusted.yaml
yq -i --indent 4 'del(.paths."/getEvents")' query_services.openapi.adjusted.yaml
yq -i --indent 4 '(.tags[] | select(.name == "ArchiveQueryService")).x-scalar-ignore = true' query_services.openapi.adjusted.yaml
yq -i --indent 4 'del(.tags[] | select(.name == "Events (Beta)"))' query_services.openapi.adjusted.yaml
yq -i --indent 4 '(.paths[][].tags) -= ["ArchiveQueryService"]' query_services.openapi.adjusted.yaml

- name: Encode adjusted spec
id: encode
run: |
CONTENT=$(base64 -w 0 v2/api/archive-query-service/v2/query_services.openapi.adjusted.yaml)
echo "spec_base64=$CONTENT" >> "$GITHUB_OUTPUT"

- name: Dispatch to qubic/integration
uses: peter-evans/repository-dispatch@v3
with:
token: ${{ secrets.GH_PAT }}
repository: qubic/integration
event-type: api-docs-update
event-type: query-api-update
client-payload: '{"openapi_spec_base64": "${{ steps.encode.outputs.spec_base64 }}"}'

- name: Dispatch to qubic/docs
uses: peter-evans/repository-dispatch@v3
with:
token: ${{ secrets.GH_PAT }}
repository: qubic/docs
event-type: query-api-update
event-type: query-api-update
client-payload: '{"openapi_spec_base64": "${{ steps.encode.outputs.spec_base64 }}"}'
23 changes: 23 additions & 0 deletions .github/workflows/test-v2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,29 @@ jobs:
with:
version: v2.6
working-directory: v2
check-openapi:
name: Check OpenAPI spec is up to date
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-go@v4
with:
go-version: 1.25.x
- name: Install protoc
uses: arduino/setup-protoc@v3
with:
version: '28.x'
- name: Install protoc-gen-openapi
run: go install github.com/google/gnostic/cmd/protoc-gen-openapi@latest
- name: Regenerate OpenAPI spec
working-directory: v2
run: make openapi-v3-gen
- name: Check for differences
run: |
if ! git diff --exit-code v2/api/archive-query-service/v2/query_services.openapi.yaml; then
echo "::error::OpenAPI spec is out of date. Run 'make openapi-v3-gen' and commit the result."
exit 1
fi
govulncheck:
runs-on: ubuntu-latest
name: Run govulncheck v2
Expand Down
5 changes: 0 additions & 5 deletions v2/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,6 @@ openapi-v3-gen:
cd "$(PROTO_DIR)" && \
protoc --openapi_out=output_mode=source_relative,default_response=false:. $(OPT_ARGS) -I "$(PROTO_DIR)" \
query_services.proto
@echo "Cleaning up OpenAPI spec..."
cd "$(PROTO_DIR)" && \
yq -iy 'del(.paths."/health")' query_services.openapi.yaml && \
yq -iy '(.tags[] | select(.name == "ArchiveQueryService"))."x-scalar-ignore" = true' query_services.openapi.yaml && \
yq -iy '(.paths[][].tags) -= ["ArchiveQueryService"]' query_services.openapi.yaml

test-cover:
@echo "Performing Go cover test..."
Expand Down
9 changes: 9 additions & 0 deletions v2/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,5 +223,14 @@ See [messages.proto](api/archive-query-service/v2/messages.proto)
and [query_services.proto](api/archive-query-service/v2/query_services.proto)
for full details.

## Regenerating protobuf and OpenAPI files

Run `make` (or `make all`) to regenerate all protobuf and OpenAPI files.

### Dev dependencies

- `protoc` with Go plugins (`protoc-gen-go`, `protoc-gen-go-grpc`, `protoc-gen-grpc-gateway`)
- `protoc-gen-openapi` (`go install github.com/google/gnostic/cmd/protoc-gen-openapi@latest`)

Copy link
Contributor Author

@sallymoc sallymoc Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@qubicmio I don't have linux but seems you are using the python-based version that has a different syntax. Please try 'snap install yq' that installs the Go-based mikefarah/yq v4+

Copy link
Member

@qubicmio qubicmio Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made it work by using a different syntax but I don't think we should have such fragile constructs in our Makefile. It will work on some machines while it won't on others. Wouldn't it be better to have the conversion somewhere else in the pipeline?

Copy link
Contributor

@0xluk 0xluk Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point @qubicmio

Copy link
Contributor Author

@sallymoc sallymoc Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@qubicmio There are two different tools both called yq — the Go-based mikefarah/yq and the Python-based kislyuk/yq — with incompatible syntax. Can you confirm the last one is the one you have installed?

I've added a check-yq target in the Makefile that runs before openapi-v3-gen and detects if the wrong yq (or none) is installed, showing a clear error with install instructions.

As a next step, I could add a CI validation workflow that regenerates the OpenAPI spec and compares it with what's committed. If they differ, the PR fails with a message to run make openapi-v3-gen. We can also add it as a required check in branch protection so it blocks the merge until the YAML is up to date.

This is better than auto-generating the YAML after merge because:

  • The developer sees the final YAML in the PR diff before merging, so reviewers can catch issues
  • No surprise commits or merge conflicts from CI pushing changes back
  • The committed YAML is always the source of truth

Copy link
Member

@qubicmio qubicmio Mar 3, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's the wrong place to handle this here because the yaml is correctly generated. only a certain user (ci pipeline or another repo idk in detail) needs a different format. I think the changes to the file should happen there so that the ease of development is not affected by tooling.

## Caching
See [CACHE.md](CACHE.md) for details about enabling and using caching in Archive Query Service v2.
Loading
Loading