RFC [KT] Add content-release command for kernel release automation #57

bmastbergen · 2026-01-30T14:23:08Z

Introduces a new 'content-release' command that automates the complete
kernel content release workflow through three stages: prepare, build,
and test. The command can run all stages sequentially (default) or
execute individual stages via --prepare, --build, and --test flags.

Prepare Stage:

Validates git user.name and user.email configuration in dist-git repo
Executes mkdistgitdiff.py to generate staging branch and release files
Staging branch format: {automation_tmp}_<src_branch>
Checks out the newly created staging branch
Extracts and displays the new release tag from script output

Build Stage:

Verifies mock command availability and user membership in mock group
Validates DEPOT_USER and DEPOT_TOKEN environment variables
Creates temporary mock config with depot credentials injected
Downloads kernel sources using getsrc.sh script
Builds source RPM with mock in build_files directory
Builds binary RPMs from the SRPM
Lists all created RPM packages
Cleans up temporary mock configuration

Test Stage:

Spins up VM for the kernel workspace (creates if needed)
Installs built kernel RPMs excluding .src.rpm, kernel-rt*, kernel-debug*
Reboots VM and verifies running kernel version matches installed version
Executes kselftests using /usr/libexec/kselftests/run_kselftest.sh
Counts and reports number of tests passed
Outputs logs: install.log, selftest-.log

Additional Changes:

Refactored VM implementation to extract reusable functions:
- load_vm_from_workspace(): Load config and VM from workspace name
- setup_and_spinup_vm(): Common VM setup and spinup logic
Increased VM disk size to 30GB during creation to accommodate kernel
packages and debug symbols
Updated KT.md with comprehensive documentation for the new command

The content-release command streamlines the kernel release process by
automating previously manual steps, ensuring consistency and reducing
the potential for human error in the release workflow.

And since its not in the diff, this requires the mock-configs and kernel-tools repos checked out in the kt workspace.

This works for lts-9.2. That is all I tested. More testing will be needed. This is really just an RFC at the moment.

Copilot

Pull request overview

This PR extends the kt vm command with a --content_release workflow that automates building and installing LTS kernels in a VM from dist-git, then running kselftests. It introduces configuration fields and YAML metadata needed to drive the per-kernel mock builds and depot authentication.

Changes:

Extend KernelInfo / kernels.yaml to track a per-kernel mock_config used for mock builds.
Extend Config with user, email, depot_user, and depot_token and adjust from_str_dict to only treat specific keys as paths.
Add VmInstance.content_release, a --content_release CLI flag, and additional VM dependencies (mock, python3-GitPython) plus a few supporting wiring changes.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`kt/ktlib/kernels.py`	Adds a required `mock_config` field to `KernelInfo` to match new `kernels.yaml` entries and support selecting the proper mock config for each kernel.
`kt/ktlib/config.py`	Adds user/depot-related config fields and refines `from_str_dict` to convert only specific keys to `Path`, changing how JSON configs must be structured.
`kt/data/kernels.yaml`	Annotates each kernel entry with its corresponding `mock_config` filename to be used by the content release workflow.
`kt/commands/vm/impl.py`	Implements the `VmInstance.content_release` orchestration, wires it into `main`, and consumes configuration/kernels metadata to run mkdistgitdiff, mock builds, install RPMs, and run kselftests.
`kt/commands/vm/command.py`	Exposes the new `--content_release` flag on the `kt vm` CLI and forwards it to the implementation.
`kernel_install_dep.sh`	Ensures `mock` and `python3-GitPython` are installed for supported Rocky versions and adds the invoking user to the `mock` group to allow mock builds within the VM.

Comments suppressed due to low confidence (1)

kt/commands/vm/impl.py:375

In the content_release path we log “Waiting for the deps to be installed” but the actual time.sleep(Constants.VM_DEPS_INSTALL_WAIT_SECONDS) call is commented out, unlike the test path above. This means that on a freshly created VM the content release workflow may start before kernel_install_dep.sh has finished installing mock, python3-GitPython, and other dependencies, leading to sporadic failures; either restore the sleep or otherwise ensure dependency installation has completed before invoking content_release.

    vm = Vm.load(config=config, kernel_workspace=kernel_workspace)
    if destroy:
        vm.destroy()
        return

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

kt/ktlib/kernels.py

kt/ktlib/config.py

kt/commands/vm/impl.py

roxanan1996

I left some comments inline.

I think this needs a separate kt command.
Since building the kernel package can be done locally (not in a vm).

And reuse some vm functionality to test the kernel. The current state could be refactored a bit to "export" some functionality to other kt commands. I would put those into kt/ktlib.

kt/commands/vm/impl.py

roxanan1996 · 2026-02-02T16:22:08Z

kt/commands/vm/impl.py

+
+        ssh_cmd = (
+            f"cd {self.kernel_workspace.dist_worktree.folder.absolute()} && "
+            f"mock -v -r ../${{USER}}_{mock_config} --resultdir={self.kernel_workspace.folder.absolute()}/build_files --buildsrpm "


Do we need to run mock in the vm? I think this is already using a chroot or sth similar.

So why not running this on the host?

And use the vm just for testing. I believe the content_release original script is like this.

you are absolutely right that the rpms could be built on the host. And I probably prefer it that way (which is why content_release.sh does it that way). The one nice thing about them being built in the vm is that the system running kt doesn't even have to care about mock, which is kind of nice from the perspective of minimizing the requirements for automation.

Yeah, but it kt won't run locally. It will run in a container in jenkins or idk. I think it's not an issue if we have more requiremements, as long as they are nicely installable. I will add support to install other packages than through pip to fix this issue.

you are absolutely right

We're all LLMs now

bmastbergen · 2026-02-03T15:20:31Z

I have revamped this to introduce a new content-release command that does the mkdistgitdiff and mock builds locally, and only spins up the vm to test. This is much closer to what content_release.sh does today, with the added flexibility of splitting the task up into prepare, build and test steps taht can be run individually if needed. See what you think.

Copilot

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

kt/commands/vm/impl.py:308

setup_and_spinup_vm always destroys and recreates the VM when override=True without checking if it exists, which can cause an unnecessary failure if the VM name has never been created. It would be more robust to check VirtHelper.exists(vm_name=...) before calling destroy() so --override works both for existing and first-time VMs.

    vm, config = load_vm_from_workspace(kernel_workspace_name)

    if override:
        vm.destroy()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

kt/commands/content_release/impl.py

kt/commands/vm/impl.py

kt/commands/content_release/impl.py

Copilot

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-04T16:02:20Z

kt/commands/content_release/impl.py

+    srpm_files = list(build_files_dir.glob("*.src.rpm"))
+    if not srpm_files:
+        raise RuntimeError(f"No SRPM found in {build_files_dir}")
+    srpm_file = srpm_files[0]


When selecting the SRPM to use for the binary RPM build, this code takes the first *.src.rpm it finds in build_files_dir. If there are multiple SRPMs present (e.g., from previous runs), this can end up rebuilding from a stale SRPM instead of the one just produced by the current mock --buildsrpm invocation. Consider either cleaning build_files_dir before the SRPM build, or selecting the SRPM deterministically (for example by using the most recently modified file or parsing the SRPM path from the mock_buildsrpm.log).

Suggested change

srpm_file = srpm_files[0]

# Select the most recently modified SRPM to avoid using a stale file

srpm_file = max(srpm_files, key=lambda p: p.stat().st_mtime)

Copilot · 2026-02-04T16:02:21Z

kt/commands/content_release/impl.py

+    # Verify DEPOT_USER and DEPOT_TOKEN environment variables are set
+    depot_user = os.environ.get("DEPOT_USER")
+    depot_token = os.environ.get("DEPOT_TOKEN")
+
+    if not depot_user:
+        raise RuntimeError(
+            "DEPOT_USER environment variable is not set. Please set it:\n  export DEPOT_USER=your_username"
+        )
+    if not depot_token:
+        raise RuntimeError(
+            "DEPOT_TOKEN environment variable is not set. Please set it:\n  export DEPOT_TOKEN=your_token"
+        )
+    logging.info("DEPOT_USER and DEPOT_TOKEN environment variables are set")


DEPOT_USER and DEPOT_TOKEN are validated unconditionally, even though the subsequent logic treats CBR workspaces (cbr-*) as not needing depot credentials and uses a non-depot mock config in that case. As written, run_build will still fail for CBR workspaces if these environment variables are unset, which both contradicts the intent described by the is_cbr comment and forces users to provide unused credentials. You may want to short-circuit the depot credential checks for CBR workspaces (or otherwise conditionally enforce them only for configurations that actually require depot).

Copilot · 2026-02-04T16:02:21Z

kt/commands/content_release/impl.py

+            logging.info("Depot client installed")
+
+            # Register depot with credentials
+            SshCommand.run(domain=vm_instance.domain, command=[f"sudo depot register -u {depot_user} -t {depot_token}"])


The depot registration command is built by interpolating depot_user and depot_token into a single shell string (f"sudo depot register -u {depot_user} -t {depot_token}") that is passed through SSH. If either value contains spaces or shell metacharacters, this can lead to command failures or, in the worst case, shell injection. It would be safer and more robust to pass these as separate arguments (e.g., command=["sudo", "depot", "register", "-u", depot_user, "-t", depot_token]) so that they are not subject to shell parsing on the remote side.

Suggested change

SshCommand.run(domain=vm_instance.domain, command=[f"sudo depot register -u {depot_user} -t {depot_token}"])

SshCommand.run(

domain=vm_instance.domain,

command=["sudo", "depot", "register", "-u", depot_user, "-t", depot_token],

)

Introduces a new 'content-release' command that automates the complete kernel content release workflow through three stages: prepare, build, and test. The command can run all stages sequentially (default) or execute individual stages via --prepare, --build, and --test flags. Prepare Stage: - Validates git user.name and user.email configuration in dist-git repo - Executes mkdistgitdiff.py to generate staging branch and release files - Staging branch format: {automation_tmp}_<src_branch> - Checks out the newly created staging branch - Extracts and displays the new release tag from script output Build Stage: - Verifies mock command availability and user membership in mock group - Validates DEPOT_USER and DEPOT_TOKEN environment variables - Creates temporary mock config with depot credentials injected - Downloads kernel sources using getsrc.sh script - Builds source RPM with mock in build_files directory - Builds binary RPMs from the SRPM - Lists all created RPM packages - Cleans up temporary mock configuration Test Stage: - Spins up VM for the kernel workspace (creates if needed) - Installs built kernel RPMs excluding .src.rpm, kernel-rt*, kernel-debug* - Reboots VM and verifies running kernel version matches installed version - Executes kselftests using /usr/libexec/kselftests/run_kselftest.sh - Counts and reports number of tests passed - Outputs logs: install.log, selftest-<version>.log Additional Changes: - Refactored VM implementation to extract reusable functions: * load_vm_from_workspace(): Load config and VM from workspace name * setup_and_spinup_vm(): Common VM setup and spinup logic - Increased VM disk size to 30GB during creation to accommodate kernel packages and debug symbols - Updated KT.md with comprehensive documentation for the new command The content-release command streamlines the kernel release process by automating previously manual steps, ensuring consistency and reducing the potential for human error in the release workflow.

Copilot

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-04T17:52:40Z

kt/commands/content_release/impl.py

+    # Load kernel workspace
+    kernel_workspace_obj, config = KernelWorkspace.load_from_name(kernel_workspace)
+
+    # Setup and spin up the VM (reuses common code from vm command)
+    vm_instance, config = setup_and_spinup_vm(kernel_workspace_name=kernel_workspace)


run_test calls KernelWorkspace.load_from_name(kernel_workspace) to obtain kernel_workspace_obj and config, and then calls setup_and_spinup_vm(kernel_workspace_name=kernel_workspace), which internally reloads Config and the same workspace. This double-loading is redundant and can cause subtle inconsistencies if Config.load() ever gains side effects; consider reusing the already-loaded config and workspace (e.g., by refactoring setup_and_spinup_vm to accept preloaded objects) or centralizing the loading in a single helper.

Copilot · 2026-02-04T17:52:40Z

kt/commands/content_release/impl.py

+    if is_cbr:
+        mock_config_name = f"{mock_config_base}-x86_64.cfg"
+    else:
+        mock_config_name = f"{mock_config_base}-depot-x86_64.cfg"


The construction of mock_config_name appends the -depot suffix unconditionally for all non-CBR workspaces, but some entries in release_map already include -depot in the mock_config value (e.g. fips-8.10 uses "rocky-fips810-553-depot", fips-8.6 uses "rocky-fips86-553-depot"). This results in filenames like rocky-fips810-553-depot-depot-x86_64.cfg, which likely do not exist and will cause the mock config lookup to fail. Either remove the -depot suffix from those release_map entries or adjust this logic to only append -depot when it is not already present in mock_config_base.

Suggested change

mock_config_name = f"{mock_config_base}-depot-x86_64.cfg"

# For non-CBR workspaces, ensure we only have a single '-depot' suffix

mock_config_with_depot = (

mock_config_base

if mock_config_base.endswith("-depot")

else f"{mock_config_base}-depot"

)

mock_config_name = f"{mock_config_with_depot}-x86_64.cfg"

Copilot · 2026-02-04T17:52:40Z

kt/commands/content_release/impl.py

+    download_config = SOURCE_DOWNLOAD_CONFIG.get(kernel_workspace)
+
+    if download_config:
+        # Direct download for special cases
+        logging.info(f"Downloading {kernel_workspace} source files directly...")
+        for hash_or_id, filename in download_config["files"]:
+            url = f"{download_config['base_url']}/{hash_or_id}"
+            run_command(
+                ["curl", url, "-o", str(sources_dir / filename)],
+                error_msg=f"Failed to download {filename}",
+            )


For the direct download path in download_sources, the URL is built as <base_url>/<hash_or_id> and the filename is only used on the curl -o side. For the cbr-7.9 entry (and generally for git.centos.org source URLs), the canonical layout is <base_url>/<hash>/<filename>, so omitting the /filename portion from the URL means curl will not fetch the actual tarball contents and the saved file will be invalid for later RPM builds. Please include the filename component in the URL for these hashed-source downloads (or otherwise confirm and encode the expected URL pattern per entry).

Copilot AI review requested due to automatic review settings January 30, 2026 14:23

Copilot started reviewing on behalf of bmastbergen January 30, 2026 14:23 View session

bmastbergen force-pushed the kt-content-release branch from 76083c0 to b286616 Compare January 30, 2026 14:30

Copilot AI reviewed Jan 30, 2026

View reviewed changes

kt/ktlib/kernels.py Outdated Show resolved Hide resolved

kt/ktlib/config.py Outdated Show resolved Hide resolved

kt/commands/vm/impl.py Outdated Show resolved Hide resolved

kt/commands/vm/impl.py Outdated Show resolved Hide resolved

bmastbergen force-pushed the kt-content-release branch from b286616 to e63daa0 Compare January 30, 2026 15:23

roxanan1996 reviewed Feb 2, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings February 3, 2026 15:15

bmastbergen force-pushed the kt-content-release branch from e63daa0 to e31845d Compare February 3, 2026 15:15

Copilot started reviewing on behalf of bmastbergen February 3, 2026 15:15 View session

bmastbergen force-pushed the kt-content-release branch from e31845d to a18a453 Compare February 3, 2026 15:17

bmastbergen changed the title ~~RFC [KT] Add vm --content_release flag to automate LTS content release workflow~~ RFC [KT] Add content-release command for kernel release automation Feb 3, 2026

Copilot AI reviewed Feb 3, 2026

View reviewed changes

bmastbergen force-pushed the kt-content-release branch from a18a453 to 9de1e07 Compare February 4, 2026 15:33

Copilot AI review requested due to automatic review settings February 4, 2026 15:51

bmastbergen force-pushed the kt-content-release branch from 9de1e07 to a35829a Compare February 4, 2026 15:51

Copilot started reviewing on behalf of bmastbergen February 4, 2026 15:52 View session

Copilot AI reviewed Feb 4, 2026

View reviewed changes

bmastbergen force-pushed the kt-content-release branch from a35829a to e5ee2d7 Compare February 4, 2026 17:30

Copilot AI review requested due to automatic review settings February 4, 2026 17:40

bmastbergen force-pushed the kt-content-release branch from e5ee2d7 to 434b595 Compare February 4, 2026 17:40

Copilot started reviewing on behalf of bmastbergen February 4, 2026 17:41 View session

Copilot AI reviewed Feb 4, 2026

View reviewed changes

	srpm_file = srpm_files[0]
	# Select the most recently modified SRPM to avoid using a stale file
	srpm_file = max(srpm_files, key=lambda p: p.stat().st_mtime)

-        mock_config_name = f"{mock_config_base}-depot-x86_64.cfg"
+        # For non-CBR workspaces, ensure we only have a single '-depot' suffix
+        mock_config_with_depot = (
+            mock_config_base
+            if mock_config_base.endswith("-depot")
+            else f"{mock_config_base}-depot"
+        )
+        mock_config_name = f"{mock_config_with_depot}-x86_64.cfg"

RFC [KT] Add content-release command for kernel release automation #57

Are you sure you want to change the base?

RFC [KT] Add content-release command for kernel release automation #57

Uh oh!

Conversation

bmastbergen commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roxanan1996 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

roxanan1996 Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

bmastbergen Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

roxanan1996 Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

PlaidCat Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

bmastbergen commented Feb 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

bmastbergen commented Jan 30, 2026 •

edited

Loading