engineering: build azl3+azl4 RPMs in parallel at task level#699
Open
bfjelds wants to merge 3 commits into
Open
engineering: build azl3+azl4 RPMs in parallel at task level#699bfjelds wants to merge 3 commits into
bfjelds wants to merge 3 commits into
Conversation
Replace the two sequential release.yml invocations (one per distro) with a single build-rpms-parallel.yml template plus scripts/build-rpms-parallel.sh. Shared setup runs once, then both azl3 and azl4 RPMs build concurrently as background docker builds in a single task, avoiding a second ADO job and duplicated OneBranch/setup overhead. - New scripts/build-rpms-parallel.sh (executable): run both docker builds in the background (& ... wait), capture per-distro logs and exit codes, then unpack azl3 to the base artifact dir and azl4 under azl4/. Uses set -o pipefail and anchored greps so a failed make azl-version-vars cannot be masked by the grep/cut pipeline. Per-distro dest dirs and image tags prevent clobbering. - New build-rpms-parallel.yml: run-once setup (version, deps, docker, preview container) then a single task invoking the script. Distro binaries are extracted via extract-binary.sh for azl3 and azl4. - build-source.yml: amd64 and arm64 jobs call the new template once. - Remove now-unused release.yml. Artifact layout unchanged: azl3 at base path, azl4 under azl4/. Downstream download-staged.yml and consumer distro-filters are unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/azp run [GITHUB]-trident-pr-e2e |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Trident RPM build pipeline to build Azure Linux 3 (azl3) and Azure Linux 4 (azl4) RPMs concurrently within a single Azure DevOps job, preserving the existing single-artifact layout (azl3 at the base path, azl4 under azl4/).
Changes:
- Add
scripts/build-rpms-parallel.shto run twodocker buildinvocations in parallel, each with isolated output/log directories, then unpack both tarballs into the expected artifact layout. - Add
build-rpms-parallel.ymlpipeline template that performs shared setup once, runs the parallel build script, then extracts binaries per distro. - Update
build-source.ymlto use the new template once per architecture, and remove the old per-distrorelease.ymltemplate.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
scripts/build-rpms-parallel.sh |
New bash driver to run azl3/azl4 Docker RPM builds concurrently and unpack results into the established artifact contract. |
.pipelines/templates/stages/trident_rpms/build-rpms-parallel.yml |
New pipeline template that performs shared setup once, invokes the parallel build script, then runs per-distro binary extraction. |
.pipelines/templates/stages/trident_rpms/build-source.yml |
Switch amd64/arm64 jobs to use the new parallel template instead of two sequential template invocations. |
.pipelines/templates/stages/trident_rpms/release.yml |
Removed obsolete template previously invoked once per distro. |
- build-source.yml: drop the redundant arm64 'Start Docker' step; the build-rpms-parallel.yml shared setup already starts Docker for both arches. - build-rpms-parallel.sh: validate arg count and refuse an empty, root, or non-absolute work_dir before the rm -rf guard. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The two builds run as concurrent background subshells; unconditional set -x interleaved trace lines from both on the task stderr, undermining the per-distro log files. Default to no xtrace and enable it only when BUILD_RPMS_DEBUG=1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
fintelia
approved these changes
Jun 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds the azl3 and azl4 Trident RPMs concurrently within a single ADO job instead of running two sequential
release.ymlinvocations (one per distro). This avoids a second job and the duplicated OneBranch/setup overhead, while overlapping the I/O-bound parts of the two docker builds. The single-artifact dual-distro layout is unchanged (azl3 at the base path, azl4 underazl4/).In trident-CI, amd64 build stage with azl3 and azl4 went from 14 minutes to 27 minutes. Parallel effort reclaims some of that time (down to 21 minutes, unclear why azl4 build takes so much more time).
Why task-level (not job/stage-level)
Today, inside
BuildTrident_<arch>,release.ymlis invoked twice. Each invocation independently repeats all of: set version,tdnf install, start docker, preview-container download/sed,docker build, and extract. So we paid the setup tax twice and the two docker builds ran serially. Splitting into a separate ADO job per distro would add OneBranch/checkout/setup overhead that likely cancels any speedup. Doing it at the task level captures both wins: run setup once, then overlap the builds.Changes
scripts/build-rpms-parallel.sh— runs bothdocker builds as background jobs against one BuildKit daemon, each writing to its own dest dir and log file, with explicit per-distro exit-code checks. Unpacks azl3 to the base artifact dir and azl4 underazl4/.build-rpms-parallel.yml— run-once shared setup (version, deps, docker, preview-container) followed by a single task invoking the script, thenextract-binary.shper distro.build-source.yml— amd64 and arm64 jobs call the new template once.release.yml.Hardening
:azl3/:azl4), and tarballs prevent clobbering between the concurrent builds.set -o pipefail+ anchored greps (^AZL_IMAGE=) so a failedmake azl-version-varscannot be masked by thegrep | cutpipeline.wait $pid || rc=$?checks fail the task if either build fails; both build logs are printed sequentially for readable diagnostics.$(Agent.TempDirectory)work dir avoids collisions if builds share a host.Dockerfile.fullhas no BuildKit--mount=type=cache, so no cache-race.Artifact contract
Unchanged. azl3 RPMs/binary at the base artifact path; azl4 under
azl4/.download-staged.ymland all consumer distro-filters are unaffected.Validation
make check-pipelines(Linux + az toolchain) before merge to confirm wiring and measure actual wall-clock vs. the serial baseline. Realistic expectation: solid savings from de-duplicated setup; build overlap depends on agent core count (two concurrent cargo builds share cores, so not a full 2x).Note
In non-release modes,
preview-container.ymlseds a single base intoDockerfile.full, so both distros share that base — a pre-existing limitation (already true before this change), clean in release mode whereAZL_IMAGEis honored per distro.