ci: stabilize CI tests and add Fedora IoT raw image testing by pcdubs · Pull Request #191 · fedora-iot/greenboot-rs

pcdubs · 2026-07-16T17:48:44Z

Summary

Fixes and improvements to CI test infrastructure across bootc, ostree, and anaconda-iso test paths.

CI workflow fixes

Allow unsafe PR checkout in make-rpm job (actions/checkout@v7 blocks pull_request_target by default)
Use dnf5 copr plugin on Fedora 44+ (dnf5 doesn't have dnf4's copr subcommand)
Move Copr source add after osbuild-composer.socket start in ostree script
Remove stale local packages repo from osbuild-composer configs
Specify Copr chroot for CentOS/RHEL bootc builds
Increase Testing Farm timeout for slow jobs (centos-9-ostree, rhel-9.8-ostree, rhel-10.2-bootc)

Bootc test improvements

Use fedora-iot base image for Fedora bootc tests (smaller, purpose-built)
Run fedora-45 bootc tests on stable Fedora 44 host via TARGET_DISTRO override
Copy RHEL repo file before dnf install in Containerfile
Collect VM diagnostics (console log, DHCP leases) on SSH failure
Wait for firewalld D-Bus readiness before libvirt setup

Anaconda-ISO test improvements

Replace mkksiso with BIB (bootc-image-builder) custom kickstart
Use --location for direct kernel boot instead of --cdrom

New: Fedora IoT raw image test

Add test using pre-built Fedora IoT raw images from kojipkgs
Uses kpartx for SSH key injection (ostree images not compatible with virt-customize)
Installs greenboot via rpm-ostree override replace from Copr PR builds
Adds arch-specific test binaries (x86_64, aarch64)

actions/checkout@v7 refuses to check out PR code from a pull_request_target workflow by default. This is safe because the check-pull-request job already gates on write/admin permission. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

gemini-code-assist

Code Review

This pull request updates the dnf installation logic in the bootc-anaconda-iso and bootc-qcow2 test scripts to include a fallback for dnf5-command(copr). Additionally, it refactors the greenboot-ostree.sh script by moving the ostree_key permission check and service startup commands to an earlier execution point. The review feedback suggests improving error visibility by removing the stderr redirection to /dev/null in the dnf commands and simplifying the file permission retrieval logic in the ostree script.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Fedora 44 ships dnf5 where 'dnf-command(copr)' installs the dnf4 plugin but 'dnf copr' invokes dnf5. Try dnf5-command(copr) first, falling back to dnf-command(copr) for CentOS/RHEL. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

composer-cli requires osbuild-composer.socket to be running. The Copr source block was placed before the socket start, causing the retry loop to fail for 15+ minutes waiting on a socket that hadn't been started yet. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

The packages source at http://192.168.100.1/packages/ was used by the old make-rpm flow to serve locally-built RPMs. Now that greenboot packages come from Copr, this repo is never populated, causing blueprints depsolve to fail on ostree CI runs. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

On CentOS Stream 10, dnf copr defaults to epel-10-x86_64 which does not exist in the Copr project. Similarly RHEL maps to non-existent chroots. Explicitly pass the correct centos-stream chroot name for CentOS and RHEL builds. Initialize COPR_CHROOT to empty for Fedora where auto-detection works. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

Remove the mkksiso post-processing step and use BIB's customizations.installer.kickstart to control the anaconda installation directly. This simplifies the test by eliminating the ISO rewrite step and its lorax dependency. Custom kickstart provides: - text --non-interactive for unattended install - zerombr, clearpart, autopart for disk initialization - network for DHCP configuration - bootloader --append for serial console on installed system - poweroff for clean shutdown (compatible with --wait=-1) The Users module is disabled since user creation is handled in the Containerfile — BIB does not support customizations.user alongside customizations.installer.kickstart, and anaconda cannot set passwords on the ostreecontainer filesystem. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

Bump centos-9-ostree, rhel-9.8-ostree, and rhel-10.2-bootc from 90 to 120 minutes. These jobs consistently hit the 90-minute limit due to osbuild-composer compose times and nested VM installs. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

Switch BASE_IMAGE_URL from quay.io/fedora/fedora-bootc to quay.io/fedora/fedora-iot for Fedora 44, and rawhide in both qcow2 and anaconda-iso test scripts. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

Switch virt-install from --cdrom to --location with explicit kernel/initrd paths to enable direct kernel boot and serial console output during anaconda installation. Extract the ISO label dynamically with blkid for inst.stage2 and inst.ks arguments. This fixes the issue where --cdrom boot through the ISO GRUB gave no serial console visibility and anaconda silently failed to install in CI environments. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>" Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

The RHEL repo COPY was appended after the main Containerfile heredoc, placing it as the last instruction — after the RUN dnf install that needs it. The RHEL base image has no enabled repos without this file, so dnf fails immediately on rhel-10.2 and rhel-9.8 bootc jobs. Split the Containerfile construction so the repo file COPY comes between FROM and the first RUN, ensuring repos are available when dnf runs. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>" Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

Since firewalld 2.4.1 removed Type=dbus from its systemd unit, the service reports active before its D-Bus interface is registered. Poll firewall-cmd --state to ensure firewalld is ready before virsh net-start sets zone='trusted' on the integration bridge. See: virt-s1/rhel-edge@3858a65 Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

Add network and VM diagnostics to the SSH failure block in both bootc test scripts to help debug intermittent connectivity issues. Diagnostics only run when SSH fails, not on every execution. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

Decouple the host OS from the test target for fedora-45 bootc tests. The Testing Farm compose is changed from Fedora-Rawhide to Fedora-44 (stable), while TARGET_DISTRO=fedora-45 is passed to select the correct container base image and OS variant. The bootc tests only need podman and libvirt on the host — the actual OS under test runs inside the VM. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

Use pre-built daily Fedora IoT raw images from kojipkgs instead of building ostree commits with osbuild-composer. Greenboot is installed from Packit Copr PR builds via rpm-ostree override replace. This also runs fedora-45 ostree on a stable Fedora 44 host via TARGET_DISTRO. CentOS/RHEL ostree tests still use osbuild-composer. Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

pcdubs · 2026-07-22T15:22:47Z

Note: Some test failures are expected until this PR is merged, as pull_request_target workflows read from the base branch. Jobs that depend on workflow changes (fedora-45-bootc, make-rpm) will only pass after merge.

The centos-10-bootc test was consistently passing but now hits a TF setup issue.

To test the IoT raw images locally sudo PR_NUMBER=191 bash greenboot-fedora-iot-raw.sh

gemini-code-assist · 2026-07-22T15:23:11Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

say-paul

LGTM.
Any particular reason to switch the base image from fedora-bootc to fedora-iot ?

pcdubs · 2026-07-23T14:07:19Z

LGTM. Any particular reason to switch the base image from fedora-bootc to fedora-iot ?

It seemed more appropriate to test our images for iot, but I can change it back if you prefer?

knecasov

I added some comments.

continue-on-error: true masks test failures - the workflow shows green even when a job fails. Consider removing it.

knecasov · 2026-07-24T15:15:37Z

+
+greenprint "📥 Downloading Fedora IoT ${FEDORA_VERSION} raw image"
+COMPOSE_URL="https://kojipkgs.fedoraproject.org/compose/iot/latest-Fedora-IoT-${FEDORA_VERSION}/compose/IoT/${ARCH}/images/"
+RAW_IMAGE_FILENAME=$(curl -s "${COMPOSE_URL}" | grep -oP "Fedora-IoT-raw-${FEDORA_VERSION}-[^\"]+\.${ARCH}\.raw\.xz" | head -1)


Consider using curl -sSf instead of curl -s so that HTTP errors (503, 404) produce a clear failure instead of silently returning an error page.

knecasov · 2026-07-24T15:24:27Z

-        "baseurl": "http://192.168.100.1/packages/"
      }
    ]
 }


There should be a new line.

knecasov · 2026-07-24T15:24:51Z

+    IFS='-' read -r ID VERSION_ID <<< "${TARGET_DISTRO}"
+fi
+
+# Dumps details about the instance running the CI job.


I would write Dump....

knecasov · 2026-07-24T15:30:05Z

+    if [[ $copr_result == 0 ]]; then
+        break
+    fi
+    greenprint "Copr repo not ready yet, retrying in 30s..."


I would write ...retrying in 30 seconds....

gemini-code-assist Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread tests/greenboot-bootc-anaconda-iso.sh Outdated

Comment thread tests/greenboot-bootc-qcow2.sh Outdated

Comment thread tests/greenboot-ostree.sh

pcdubs force-pushed the ci-fixes branch 4 times, most recently from 0217843 to 4b67c38 Compare July 18, 2026 17:14

mcattamoredhat reviewed Jul 20, 2026

View reviewed changes

Comment thread tests/greenboot-bootc-anaconda-iso.sh

Comment thread tests/greenboot-bootc-anaconda-iso.sh Outdated

pcdubs force-pushed the ci-fixes branch 5 times, most recently from c977b8c to 948d23c Compare July 20, 2026 19:04

Paul Whalen added 11 commits July 20, 2026 16:00

pcdubs force-pushed the ci-fixes branch from 948d23c to 8f48fad Compare July 20, 2026 20:06

github-advanced-security AI found potential problems Jul 20, 2026

View reviewed changes

Comment thread tests/greenboot-fedora-iot-raw.sh Dismissed

pcdubs force-pushed the ci-fixes branch from 8f48fad to d8ff9c9 Compare July 22, 2026 14:36

pcdubs changed the title ~~ci: fix make-rpm checkout, dnf5 copr, and ostree source ordering~~ ci: stabilize CI tests and add Fedora IoT raw image testing Jul 22, 2026

pcdubs marked this pull request as ready for review July 22, 2026 15:23

pcdubs requested review from knecasov, mcattamoredhat and say-paul July 22, 2026 15:23

say-paul approved these changes Jul 23, 2026

View reviewed changes

knecasov reviewed Jul 24, 2026

View reviewed changes

Uh oh!

Conversation

pcdubs commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pcdubs commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Jul 22, 2026

Uh oh!

say-paul left a comment

Choose a reason for hiding this comment

Uh oh!

pcdubs commented Jul 23, 2026

Uh oh!

knecasov left a comment

Choose a reason for hiding this comment

Uh oh!

knecasov Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

knecasov Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

knecasov Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

knecasov Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pcdubs commented Jul 16, 2026 •

edited

Loading

pcdubs commented Jul 22, 2026 •

edited

Loading