Skip to content

RELEASING.md

Latest commit

 

History

History
535 lines (420 loc) · 22.3 KB

RELEASING.md

File metadata and controls

535 lines (420 loc) · 22.3 KB
Raw

Releasing NVIDIA NeMo Fabric

This document is the maintainer playbook for cutting NeMo Fabric releases. It describes the release contract, the version files that must be updated, the tag format that CI accepts, the package surfaces that are published, and the checks to run before and after a tag push.

Source Of Truth

This section defines where release history and release-facing details are maintained.

  • There is no CHANGELOG.md in this repository.
  • The documentation site has a release-notes landing page for current documentation-visible release status.
  • The source of truth for complete release history and tag-specific release notes is always GitHub Releases for this repository.

Do not copy full GitHub Release notes into CHANGELOG.md or the docs site. The docs release-notes page can summarize support status and point users to GitHub Releases.

Published Surfaces

The release process publishes these package surfaces from canonical and package-specific tag pushes:

Ecosystem Published Surface
crates.io nemo-fabric-core, nemo-fabric-cli
npm nemo-fabric-adapter-contract, nemo-fabric-adapters-common, nemo-fabric-adapters-pi
GitHub Actions nemo-fabric, nemo-fabric-runtime, nemo-fabric-collector, nemo-fabric-adapters-common, nemo-fabric-adapters-claude, nemo-fabric-adapters-codex, nemo-fabric-adapters-deepagents, nemo-fabric-adapters-hermes, and nemo-fabric-adapters-nooa wheel artifacts
Fern The documentation site

Version Model

NeMo Fabric versions are anchored on the workspace SemVer in the repository root Cargo.toml.

  • The root Cargo.toml workspace.package.version is the canonical release version for the Rust workspace.
  • The root Cargo.toml workspace.dependencies entry for nemo-fabric-core must stay aligned with that same version.
  • sdk/python/nemo-fabric/pyproject.toml, sdk/python/nemo-fabric-collector/pyproject.toml, adapter-contract/python/pyproject.toml, and every adapters/python/*/pyproject.toml carry the Python package versions and internal dependency pins and must stay aligned with the same release version. The root pyproject.toml is a private development coordinator and remains at 0.0.0.
  • The TypeScript contract, Common adapter support package, Pi adapter, private adapter workspace, exact internal dependency pins, and their lockfiles all use the canonical NeMo Fabric release version. The package version is independent of the fabric.adapter/v1alpha2 wire contract version.
  • The nemo-fabric-runtime Python package version is derived at packaging time. sdk/python/nemo-fabric-runtime/pyproject.toml stays dynamic = ["version"] in the repository, and Maturin derives the version from crates/fabric-python/Cargo.toml, which inherits the workspace version.

Release Tags

Release tags use SemVer with a leading v.

  • Use v0.1.0 for stable releases.
  • Use v0.1.0-rc.1 for prereleases.
  • Do not use tags such as 0.1.0 or 0.1.0-rc.1.

CI rejects tags that do not match the required format.

The tag text must match the version that the packaging jobs publish.

Release tags for a frozen release line should be created from the matching release/* branch, not from main.

Patch Releases

Cut a patch release from the existing release branch for that major and minor line. Do not create another release branch or run the code-freeze workflow.

Set the exact patch version, previous stable tag, and existing release branch:

export RELEASE_VERSION=0.1.1
export PREVIOUS_RELEASE_TAG=v0.1.0
export RELEASE_BRANCH=release/0.1

git fetch upstream "${RELEASE_BRANCH}" --tags
git log --oneline "${PREVIOUS_RELEASE_TAG}..upstream/${RELEASE_BRANCH}"

Open the fix or release-preparation PR against ${RELEASE_BRANCH}. The PR must contain the intended patch changes and run just set-version <release-version> so the release branch contains the final package version before tagging. Verify the commit range from ${PREVIOUS_RELEASE_TAG} contains only changes intended for the patch release. Changes required on main should be handled separately; do not mix a forward merge into the patch release.

Code Freeze

When code freeze begins for a target release, create a release branch from the latest main commit. Name the branch from the target release major and minor version. The prepare-code-freeze skill automates this process.

These examples assume upstream is the NVIDIA repository remote (NVIDIA/NeMo-Fabric). The origin remote is usually a maintainer's personal fork.

git fetch upstream main
git checkout -b release/0.2 upstream/main
git push upstream release/0.2

After creating the release branch, open a PR against main that does the following:

  1. Bump all package versions on main to the next release line:

    just set-version <next-version>

New PRs that must go into the upcoming release must target the new release/* branch. Changes intended for later releases should continue to target main.

Cut An RC Tag

RC Tags should be created when soon after code freeze to allow for QA testing of the upcoming release. The create-rc-tag skill automates this process.

export RELEASE_VERSION=0.1.0-rc.1
export RELEASE_BRANCH=release/0.1
export RELEASE_TAG="v${RELEASE_VERSION}"
echo "Cutting release tag ${RELEASE_TAG} for release branch ${RELEASE_BRANCH}"

git fetch upstream "${RELEASE_BRANCH}" --tags
git switch "${RELEASE_BRANCH}"
git pull --ff-only upstream "${RELEASE_BRANCH}"

test -z "$(git status --porcelain)"
RELEASE_SHA="$(git rev-parse HEAD)"
REMOTE_RELEASE_SHA="$(git rev-parse "upstream/${RELEASE_BRANCH}^{commit}")"
test "${RELEASE_SHA}" = "${REMOTE_RELEASE_SHA}"
test "$(just normalize-release-tag "${RELEASE_TAG}")" = "${RELEASE_VERSION}"
BASE_RELEASE_VERSION="${RELEASE_VERSION%%-*}"
CURRENT_VERSION="$(sed -n 's/^version = "\(.*\)"$/\1/p' Cargo.toml | head -n 1)"
test "${CURRENT_VERSION}" = "${BASE_RELEASE_VERSION}"

if git ls-remote --exit-code --tags upstream "refs/tags/${RELEASE_TAG}" >/dev/null; then
  echo "Error: remote tag ${RELEASE_TAG} already exists" >&2
  exit 1
fi

git tag -s -a \
  -m "NVIDIA NeMo Fabric ${RELEASE_VERSION}" \
  "${RELEASE_TAG}" \
  "${RELEASE_SHA}"

git tag -v "${RELEASE_TAG}"
git show "${RELEASE_TAG}"
test "$(git rev-parse "${RELEASE_TAG}^{commit}")" = "${RELEASE_SHA}"

git push upstream "refs/tags/${RELEASE_TAG}"

Before You Cut A Release

Before you create a release tag, confirm the following:

  1. The intended release commit is already on the release branch you intend to tag. For frozen release lines, tag the matching release/* branch.
  2. The release commit contains the final stable base version, docs updates, and any public API changes that belong in the release. Prerelease versions are derived from beta or RC tags in disposable workflow checkouts.
  3. The working tree you use for local validation is clean or disposable.

Prepare The Release Commit

Update the versioned source files in the release PR or release-prep commit to the stable base version for the release line. Prefer the repository helper:

just set-version <stable-base-version>
# For example: just set-version 0.1.0

For beta and RC tags, keep the committed source metadata at the corresponding stable base version. The tag workflows normalize the prerelease tag and stamp ecosystem-specific package metadata in their disposable checkouts. Do not commit RC-specific versions or internal dependency pins to the release branch.

The helper updates:

  1. The root Cargo.toml workspace version.
  2. The root Cargo.toml workspace.dependencies versions for nemo-fabric-core.
  3. sdk/python/nemo-fabric/pyproject.toml, adapter-contract/python/pyproject.toml, every adapters/python/*/pyproject.toml, and their internal dependency pins to the same release version.
  4. The TypeScript contract and adapter package manifests, their exact internal dependency pins, and both npm lockfiles.
  5. Cargo.lock, uv.lock, and every Python project lockfile.

Review docs and snippets that mention explicit versions, including:

Do not commit a static Python package version into sdk/python/nemo-fabric-runtime/pyproject.toml just to cut the release. Maturin derives that version from Cargo during the build.

Local Validation

Run the checks that match the surfaces affected by the release. For a normal repository release, the safest baseline is:

uv run pre-commit run --all-files
just test-rust
just test-python
just test-typescript
just docs

If you want to validate the Python packaging recipe before pushing a tag, run:

just set-version 0.1.0
just wheels

Be aware that the set-version helper intentionally rewrites version fields in place. In a disposable CI workspace that is fine. In a local checkout, restore those temporary manifest edits before continuing if you are not committing them.

Bootstrap npm Trusted Publishing

An npm package must exist before npm can bind it to a GitHub trusted publisher. The following packages were published once at 0.0.0 as inert registry bootstraps:

  • nemo-fabric-adapter-contract
  • nemo-fabric-adapters-common
  • nemo-fabric-adapters-pi

The first supported versions must be published by the trusted GitHub Actions workflows. Do not publish or tag 0.0.0 again. npm versions are immutable.

The bootstrap was published with --tag next. The registry also initialized latest to 0.0.0 because it was the package's first version and rejected removing that only latest tag. Leave both tags on the inert bootstrap: RC and beta releases move next, and the first stable release replaces latest.

Before the first supported TypeScript package release:

  1. Add at least one more NVIDIA maintainer to the package so registry administration does not depend on the bootstrap publisher's account:

    for package in \
  nemo-fabric-adapter-contract \
  nemo-fabric-adapters-common \
  nemo-fabric-adapters-pi; do
  npm owner add <second-nvidia-maintainer> "$package"
done

    npm sends each new maintainer an email invitation. The maintainer must accept it before access is granted. Verify every expected account appears in npm owner ls <package-name> before the release. Maintainers must use account-level two-factor authentication.

  2. Create and protect the GitHub npmjs environment. Require the release approvers who should authorize registry publication, and restrict deployment tags to canonical release tags matching v*.

  3. Configure each package's single trusted publisher in npm with the common values below and the package-specific workflow filename shown in the table. The values are case-sensitive.

    • Organization or user: NVIDIA
    • Repository: NeMo-Fabric
    • Environment: npmjs
    • Allowed action: npm publish
    Package Workflow filename
    nemo-fabric-adapter-contract publish_typescript.yml
    nemo-fabric-adapters-common publish_typescript.yml
    nemo-fabric-adapters-pi publish_typescript.yml

  4. Cut the canonical release tag. The publishing workflow uses that tag to publish the contract, Common adapter, and Pi adapter packages in dependency order, then verifies them in that order.

  5. Approve the npmjs environment when prompted. The workflows test, pack, and publish through OIDC; do not manually pre-publish a supported version. On a retry, a workflow exits without republishing only when its repack has byte-identical integrity and the expected dist-tag is exact. A mismatch fails closed because npm versions are immutable; do not overwrite or weaken the check.

  6. Confirm each release's provenance on npm. In the npm package settings, require two-factor authentication and disallow token publication. Then remove or revoke any local credentials used for the bootstrap.

The workflows publish stable versions with the latest dist-tag and beta and RC versions with next. Alpha versions are validated but are not published to the public npm registry. Stable releases do not move the prerelease dist-tag. A retry skips only when the immutable package version, packed artifact integrity, and expected dist-tag all match. If any of them differs, the workflow fails so a maintainer can inspect and repair the registry state explicitly. Publication also fails rather than moving latest or next backward when cutting a patch from an older release line.

Prepare Release Notes

Before cutting the final release tag, prepare the release notes (OK to skip for RC and alpha tags). You can perform these steps manually or use the draft-release-notes skill.

Confirm the exact target release version from the release branch and package metadata, then gather read-only evidence with explicit release refs. For a patch release, use the previous stable tag as --previous. For a new release line, use the previous release branch or tag:

python3 .agents/skills/draft-release-notes/scripts/collect_release_evidence.py \
  --previous <previous-release-tag-or-branch> \
  --current HEAD \
  --version <release-version>

Treat the report as an evidence index, not publication-ready copy. Verify every candidate claim in the changed public docs, API types, command help, or source. Prioritize breaking changes, migrations, user-visible features, and ongoing support limitations.

Draft the authoritative GitHub Release body for every stable release. Summarize the user-visible changes, compatibility or migration requirements, known limitations, and verified fixes. For a patch release, state the affected behavior and whether public APIs, configuration, or dependency contracts changed.

Update docs/about-nemo-fabric/release-notes.mdx only when the release changes the documentation-visible summary, compatibility guidance, support status, or limitations. A patch release that does not change those surfaces can leave the page unchanged. Preserve the MDX front matter and JSX SPDX comment when editing it, and state that the full history is available in GitHub Releases.

Review product names, commands, package names, support claims, and links. If the documentation page changed, validate it with:

git diff --check
just docs

Cut The Tag

After the release commit is merged and validated, create and push the signed, annotated release tag. Set the exact stable release version and matching release branch:

export RELEASE_VERSION=0.1.0
export RELEASE_BRANCH=release/0.1
export RELEASE_TAG="v${RELEASE_VERSION}"
echo "Cutting release tag ${RELEASE_TAG} for release branch ${RELEASE_BRANCH}"

git fetch upstream "${RELEASE_BRANCH}" --tags
git switch "${RELEASE_BRANCH}"
git pull --ff-only upstream "${RELEASE_BRANCH}"

test -z "$(git status --porcelain)"
RELEASE_SHA="$(git rev-parse HEAD)"
REMOTE_RELEASE_SHA="$(git rev-parse "upstream/${RELEASE_BRANCH}^{commit}")"
test "${RELEASE_SHA}" = "${REMOTE_RELEASE_SHA}"
test "$(just normalize-release-tag "${RELEASE_TAG}")" = "${RELEASE_VERSION}"
CURRENT_VERSION="$(sed -n 's/^version = "\(.*\)"$/\1/p' Cargo.toml | head -n 1)"
test "${CURRENT_VERSION}" = "${RELEASE_VERSION}"

if git ls-remote --exit-code --tags upstream "refs/tags/${RELEASE_TAG}" >/dev/null; then
  echo "Error: remote tag ${RELEASE_TAG} already exists" >&2
  exit 1
fi

git tag -s -a \
  -m "NVIDIA NeMo Fabric ${RELEASE_VERSION}" \
  "${RELEASE_TAG}" \
  "${RELEASE_SHA}"

git tag -v "${RELEASE_TAG}"
git show "${RELEASE_TAG}"
test "$(git rev-parse "${RELEASE_TAG}^{commit}")" = "${RELEASE_SHA}"

git push upstream "refs/tags/${RELEASE_TAG}"

Publish the TypeScript Adapter Packages

Pushing the canonical beta, RC, or stable tag publishes the contract, Common adapter, and Pi adapter packages in dependency order. The workflow submits all three packages before it starts deployment verification, then verifies them in the same order. Do not create package-specific npm tags. The nightly workflow runs the TypeScript tests against alpha tags without publishing to npm.

What CI Does on a Canonical Tag Push

Pushing a valid canonical tag triggers:

Workflow Trigger
.github/workflows/ci_python.yml For all tags including alpha
.github/workflows/publish_rust.yml For RC, beta and release tags
.github/workflows/ci_typescript.yml For nightly alpha tags and normal pull request/main validation
.github/workflows/publish_typescript.yml Publishes and verifies the contract, Common, and Pi packages for beta, RC, and stable tags
.github/workflows/fern-docs.yml For RC, beta and release tags

The release pipeline then:

  1. Normalizes and validates the tag format, then stamps ecosystem-specific package metadata in each disposable workflow checkout.
  2. Builds platform nemo-fabric-runtime wheels and pure-Python nemo-fabric, nemo-fabric-collector, and adapter wheels with the exact tag version, then uploads them as GitHub Actions artifacts.
  3. Publishes nemo-fabric-core and nemo-fabric-cli to crates.io through trusted publishing for stable, beta, and RC tags. Alpha tags are not published to crates.io.
  4. Publishes the contract, Common, and Pi packages to npm from the canonical tag in dependency order, then verifies the deployment in that order. Stable releases use the latest dist-tag and beta and RC releases use next. Alpha tags validate the packages without publishing them.
  5. Publishes Fern documentation versions for stable, beta, and RC tags. Alpha tags do not publish a separate documentation version.

The workflow boundary is split intentionally:

Publish The GitHub Release Entry

  1. Open Releases from the repository page.
  2. Select Tags, select the tag that you pushed, and select Create release from tag.
  3. Set the release title to NVIDIA NeMo Fabric <release-version>.
  4. Paste the verified GitHub Release body prepared before tagging. You can use Generate release notes as an evidence source for contributors, included pull requests, and the full comparison link, but review and curate the generated text before publishing.
  5. Confirm the selected tag and target commit match the signed tag that you verified locally. Publish a stable release as the latest release and mark a prerelease appropriately.

Post-Release Checks

After the release is live, verify:

  1. The nemo-fabric-core and nemo-fabric-cli crates are visible on crates.io.

  2. The Python wheels are available on PyPI:

  3. The Python wheels are available on NVIDIA PyPI:

  4. The TypeScript contract, Common, and Pi packages are visible on npm with the expected version, dist-tag, and provenance:

    (
  set -euo pipefail
  npmjs_registry="https://registry.npmjs.org/"
  for package in \
    nemo-fabric-adapter-contract \
    nemo-fabric-adapters-common \
    nemo-fabric-adapters-pi; do
    npm view "${package}@<release-version>" version \
      --registry="$npmjs_registry"
    npm view "$package" dist-tags --registry="$npmjs_registry"
  done
  verification_dir="$(mktemp -d)"
  trap 'rm -rf "$verification_dir"' EXIT
  cd "$verification_dir"
  npm init --yes --registry="$npmjs_registry"
  npm install --ignore-scripts --save-exact \
    --registry="$npmjs_registry" \
    "nemo-fabric-adapter-contract@<release-version>" \
    "nemo-fabric-adapters-common@<release-version>" \
    "nemo-fabric-adapters-pi@<release-version>"
  npm audit signatures --registry="$npmjs_registry"
)

  5. The Fern documentation site shows the expected version and release notes.

  6. The GitHub Release page is complete and accurate.