ci: restructure test infrastructure and add kernel failure tests by pcdubs · Pull Request #185 · fedora-iot/greenboot-rs

pcdubs · 2026-06-17T14:12:21Z

Summary

Add kernel failure recovery test using BLS entry corruption
Add Fedora IoT raw image testing with pre-built disk images
Add cross-compiled aarch64/x86_64 test binaries
Replace virt-customize with kpartx-based SSH key injection for aarch64 support
Add VM console logging and diagnostics for SSH failures
Pass PR_NUMBER to all testing-farm CI jobs (for follow-up Copr integration)
Fix Packit release_suffix so PR builds have unique NVR

+        - name: delay 300 seconds for GRUB fallback boot attempts
+          pause:
+            seconds: 300
+          delegate_to: 127.0.0.1


gemini-code-assist

Code Review

This pull request introduces GRUB-level kernel fallback protection to Greenboot by tracking the expected next deployment ID and comparing it against the booted deployment ID. It adds fallback detection, updates rollback handling to support forced rollbacks that bypass the boot counter, and integrates these changes into the health check flow and systemd services. Additionally, comprehensive integration tests simulating kernel failures were added. The reviewer suggested simplifying get_grub_bool_var by reusing get_grub_str_var to eliminate duplicate command execution and parsing logic.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-06-17T14:14:11Z

+fn get_grub_bool_var(key: &str, grub_path: &str) -> Result<bool> {
    let grub_vars = Command::new("grub2-editenv")
        .arg(grub_path)
        .arg("list")
        .output()
-        .context("Unable to list grubenv variables")?;
+        .context(format!("Unable to list grubenv variables for key: {key}"))?;
+
+    if !grub_vars.status.success() {
+        bail!(
+            "grub2-editenv failed to list variables: {}",
+            String::from_utf8_lossy(&grub_vars.stderr)
+        );
+    }

+    let prefix = format!("{key}=");
    let output = String::from_utf8_lossy(&grub_vars.stdout);
    for line in output.lines() {
-        if line.starts_with("greenboot_rollback_trigger=") {
-            let value = line.split('=').nth(1).unwrap_or("0");
+        if let Some(value) = line.strip_prefix(&prefix) {
            return Ok(value == "1");
        }
    }
-    Ok(false) // Not set means false
+    Ok(false)
 }


The helper function get_grub_bool_var duplicates the command execution, error handling, and output parsing logic of get_grub_str_var. We can simplify get_grub_bool_var by reusing get_grub_str_var and checking if the returned string is "1".

fn get_grub_bool_var(key: &str, grub_path: &str) -> Result<bool> { Ok(get_grub_str_var(key, grub_path)?.map_or(false, |val| val == "1")) }

Set fallback=1 alongside greenboot_rollback_trigger when an OS update is detected, enabling GRUB to boot the previous deployment if the new kernel panics. Disarm fallback at the start of healthcheck since reaching systemd means the kernel booted successfully. Closes: fedora-iot#180 Co-authored-by: Cursor <cursoragent@cursor.com>

DRY up get_fallback_at and get_rollback_trigger_at into a shared get_grub_bool_var helper. Fix unset_grub_var context message that was hardcoded to "boot_counter" for all keys. Co-authored-by: Cursor <cursoragent@cursor.com>

Change greenboot-set-rollback-trigger.service to run when bootc update or rpm-ostree upgrade stages a deployment, instead of at next boot. The service is now RequiredBy=ostree-finalize-staged.service and ordered Before= it, matching the pattern used by the original bash greenboot's greenboot-grub2-set-counter.service. This means the rollback trigger and fallback grub variables are set immediately when an update is staged, not deferred to the next boot. Removed: ConditionNeedsUpdate, boot-time ordering (sysinit.target, systemd-update-done.service), WantedBy boot targets. Added: ConditionPathExists=/run/ostree-booted, RequiresMountsFor=/boot, PrivateMounts=yes, RemainAfterExit=yes. Co-authored-by: Cursor <cursoragent@cursor.com>

Healthcheck no longer directly triggers boot-complete.target. Instead: healthcheck passes -> OnSuccess activates greenboot-success.target -> success target Requires=boot-complete.target. This ensures boot-complete.target is only reached through the greenboot success chain, and greenboot-success.target has no [Install] section. Co-authored-by: Cursor <cursoragent@cursor.com>

…ack detection Replace the boolean `greenboot_rollback_trigger` GRUB variable with `greenboot_next_deployment_id`, which stores the staged deployment's ID (imageDigest for bootc, checksum for rpm-ostree). This serves a dual purpose: it acts as the rollback trigger and enables GRUB-level kernel fallback detection by comparing the stored ID against the actually booted deployment. Key changes: - grub.rs: replace set/get/unset_rollback_trigger with set/get/unset_next_deployment_id using new string-valued GRUB variable helpers (set_grub_str_var, get_grub_str_var) - handler.rs: add get_booted_deployment_id and get_staged_deployment_id to query deployment IDs from bootc or rpm-ostree; add force flag to handle_rollback to bypass boot_counter checks when making a GRUB fallback permanent - main.rs: add detect_grub_fallback() which compares stored vs booted deployment ID; on mismatch, make fallback permanent via forced rollback and clear next_deployment_id to prevent rollback loops; SetRollbackTrigger now queries the staged deployment ID and sets both next_deployment_id and fallback independently Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Refactored bootstate having 3 states NormalBoot,grub fallback and greenboot fallback. Refactored Rollback having 2 states, normal and forced Signed-off-by: Sayan Paul <paul.sayan@gmail.com>

Signed-off-by: Sayan Paul <paul.sayan@gmail.com>

it runs after ostree-finalize-staged service to ensure that the rollback protection is enabled befor next reboot happens Signed-off-by: Sayan Paul <paul.sayan@gmail.com>

removed greenboot-success.target as it only activates boot-complete.target helthcheck service now directly summons boot-complete.target thus removing the cyclic dependencies. added rpm-ostreed as dependency to set-rollback.service to fix null next deployment ID in rpm-ostree deployment Signed-off-by: Sayan Paul <paul.sayan@gmail.com>

Signed-off-by: Sayan Paul <paul.sayan@gmail.com>

Stage a deployment, corrupt the higher-version BLS entry kernel path to trigger GRUB fallback, then validate greenboot detects the fallback and recovers cleanly. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

The Makefile uses git archive to create source tarballs, but git was not listed as a dependency in any of the test scripts. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

The empty release_suffix caused PR COPR builds to have the same NVR as released packages, making them uninstallable as overrides. Packit's default behavior appends PR number and commit hash for PR builds while keeping clean releases for downstream proposals. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

All test scripts were blind when SSH connectivity failed — no way to determine if the VM booted, got a DHCP lease, or hit a kernel panic. - Replace --nographics with --graphics none --serial file,path= to capture serial console output to a log file - Dump VM state, DHCP leases, and console output on SSH failure - Add missing check_result after SSH loop in ostree test - Add update_release: true to packit copr_build so PR builds get unique NVRs with commit hash and PR number Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

Use pre-built daily Fedora IoT raw images from kojipkgs instead of building ostree commits with osbuild-composer. Greenboot is installed from Packit Copr PR builds via rpm-ostree override replace. - Add greenboot-fedora-iot-raw.sh test script - Add /fedora-iot-raw TMT plan and test dispatcher routing - Add fedora-44-ostree and fedora-45-ostree CI jobs - Add fedora-44 and centos-stream-9 Packit Copr build targets - Pass PR_NUMBER to Testing Farm for Copr repo lookup Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

Replace arch-less x86_64 test binaries with arch-suffixed variants (passing_binary.x86_64, passing_binary.aarch64, etc.) and update all test scripts to select the correct binary based on $(uname -m). Also replace hardcoded x86_64 paths in BOOT_LOCATION URLs and RPM copy commands with ${ARCH}. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

virt-customize fails on aarch64 because libguestfs cannot inspect the IoT image partition layout. Use an Ignition config passed via QEMU fw_cfg instead, which works on both x86_64 and aarch64 without image modification. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

Add PR_NUMBER variable to all CI workflow jobs so test scripts can install greenboot from Packit Copr PR builds in a follow-up change. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

github-advanced-security AI found potential problems Jun 17, 2026

View reviewed changes

Comment thread tests/greenboot-bootc.yaml Fixed

Comment thread tests/greenboot-ostree.yaml

- name: delay 300 seconds for GRUB fallback boot attempts

pause:

seconds: 300

delegate_to: 127.0.0.1

gemini-code-assist Bot reviewed Jun 17, 2026

View reviewed changes

pcdubs force-pushed the kernel-failure-tests branch 6 times, most recently from 77a7c10 to 9873615 Compare June 23, 2026 18:00

github-advanced-security AI found potential problems Jun 24, 2026

View reviewed changes

Comment thread tests/greenboot-ostree.sh Dismissed

say-paul and others added 12 commits June 24, 2026 14:33

Refactoring code in a self documenting way

b221a4f

Refactored bootstate having 3 states NormalBoot,grub fallback and greenboot fallback. Refactored Rollback having 2 states, normal and forced Signed-off-by: Sayan Paul <paul.sayan@gmail.com>

Fix:do not error out if previous deployment not found

2cdc2e1

Signed-off-by: Sayan Paul <paul.sayan@gmail.com>

fix:systemd orchestration for set-fallback-trigger

81f51d9

it runs after ostree-finalize-staged service to ensure that the rollback protection is enabled befor next reboot happens Signed-off-by: Sayan Paul <paul.sayan@gmail.com>

fix: adding back grrenboot-success.target

8ab9450

Signed-off-by: Sayan Paul <paul.sayan@gmail.com>

fix: add git to test script build dependencies

8a0b584

The Makefile uses git archive to create source tarballs, but git was not listed as a dependency in any of the test scripts. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

pcdubs force-pushed the kernel-failure-tests branch 2 times, most recently from e2ef305 to 2a4afc9 Compare June 24, 2026 19:43

github-advanced-security AI found potential problems Jun 24, 2026

View reviewed changes

Comment thread tests/greenboot-fedora-iot-raw.sh Dismissed

pcdubs force-pushed the kernel-failure-tests branch from 2a4afc9 to 1b913cc Compare June 24, 2026 20:22

Paul Whalen added 3 commits June 25, 2026 14:24

pcdubs force-pushed the kernel-failure-tests branch from 1b913cc to b7b0108 Compare June 25, 2026 18:24

pcdubs force-pushed the kernel-failure-tests branch 2 times, most recently from faa132f to 1c682f9 Compare June 26, 2026 17:21

pcdubs force-pushed the kernel-failure-tests branch from 1c682f9 to 86ed1ee Compare June 26, 2026 17:27

say-paul requested a review from mcattamoredhat July 6, 2026 14:08

ci: pass PR_NUMBER to all testing-farm jobs

2218aec

Add PR_NUMBER variable to all CI workflow jobs so test scripts can install greenboot from Packit Copr PR builds in a follow-up change. Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Paul Whalen <pwhalen@fedoraproject.org>

pcdubs force-pushed the kernel-failure-tests branch from 3081298 to 2218aec Compare July 6, 2026 15:07

pcdubs changed the title ~~Kernel failure tests~~ ci: restructure test infrastructure and add kernel failure tests Jul 6, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: restructure test infrastructure and add kernel failure tests#185

ci: restructure test infrastructure and add kernel failure tests#185
pcdubs wants to merge 18 commits into
fedora-iot:mainfrom
pcdubs:kernel-failure-tests

pcdubs commented Jun 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jun 17, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

pcdubs commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pcdubs commented Jun 17, 2026 •

edited

Loading