Skip to content

fix: avoid misleading journalctl warning in logs on first boot - #190

Merged
say-paul merged 6 commits into
fedora-iot:mainfrom
sarmahaj:fix-first-boot-warning
Aug 6, 2026
Merged

fix: avoid misleading journalctl warning in logs on first boot#190
say-paul merged 6 commits into
fedora-iot:mainfrom
sarmahaj:fix-first-boot-warning

Conversation

@sarmahaj

@sarmahaj sarmahaj commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Use systemd-analyze condition ConditionFirstBoot=yes to detect first boot before querying the previous boot journal. This prevents a misleading warning on first boot when journalctl -b -1 fails because there is no prior boot. This also works correctly on systems with non-persistent journald storage.

@sarmahaj
sarmahaj requested a review from say-paul July 10, 2026 14:33

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a check in src/main.rs to verify if a previous boot exists using journalctl --list-boots before attempting to query the journal for the previous boot (-b -1). This prevents errors on systems during their first boot. The review feedback suggests optimizing the check for the number of boots by using .nth(1).is_none() instead of counting all lines, which avoids unnecessary iteration over the entire command output.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread src/main.rs Outdated
@sarmahaj
sarmahaj force-pushed the fix-first-boot-warning branch from de51ed4 to e0432e6 Compare July 10, 2026 14:41
@sarmahaj

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds a check to verify if a previous boot exists using journalctl --list-boots before querying its journal, skipping the rollback check if no previous boot is found. It also improves a log message format. The feedback suggests inverting the success check for journalctl --list-boots to return early on failure, which reduces nesting and improves readability.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread src/main.rs Outdated
@say-paul

Copy link
Copy Markdown
Member

LGTM, I like gemini comment that prevents the use of else .

@say-paul

Copy link
Copy Markdown
Member

Can we do the implementation of checking if its a first time boot of the system only, systemd should provide such info, as i think system with non-persistent journald will skip the rollback check everytime without reporting error.

@knecasov knecasov left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a few comments and gave a thumbs-up to the Gemini note.

Comment thread src/main.rs Outdated
Comment thread src/main.rs Outdated
Comment thread src/main.rs Outdated
@sarmahaj
sarmahaj force-pushed the fix-first-boot-warning branch from e0432e6 to 3511b80 Compare July 20, 2026 12:17
@sarmahaj

Copy link
Copy Markdown
Contributor Author

Can we do the implementation of checking if its a first time boot of the system only, systemd should provide such info, as i think system with non-persistent journald will skip the rollback check everytime without reporting error.

@say-paul Thanks for suggesting !
ptal at the recent commit ,implememts systemd based check for detecting firstboot

@sarmahaj
sarmahaj requested a review from knecasov July 20, 2026 12:28
mcattamoredhat
mcattamoredhat previously approved these changes Jul 20, 2026

@mcattamoredhat mcattamoredhat left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

knecasov
knecasov previously approved these changes Jul 20, 2026

@knecasov knecasov left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I would only recommend editing the PR description to reflect the final implementation.

Comment thread src/main.rs Outdated
@sarmahaj
sarmahaj dismissed stale reviews from knecasov and mcattamoredhat via abc7f15 July 21, 2026 09:13
@sarmahaj
sarmahaj force-pushed the fix-first-boot-warning branch from 3511b80 to abc7f15 Compare July 21, 2026 09:13
@sarmahaj

sarmahaj commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for review and approval @mcattamoredhat and @knecasov . Applied changes suggested by you @knecasov Thanks!

knecasov
knecasov previously approved these changes Jul 21, 2026

@knecasov knecasov left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for fixes! The PR description looks good now - only the formatting seems to be broken.

Anyway, it is not blocking merging, so I am approving it!

Comment thread src/main.rs
say-paul
say-paul previously approved these changes Jul 22, 2026
knecasov
knecasov previously approved these changes Jul 22, 2026

@knecasov knecasov left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

mcattamoredhat
mcattamoredhat previously approved these changes Jul 22, 2026

@mcattamoredhat mcattamoredhat left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sarmahaj
sarmahaj dismissed stale reviews from mcattamoredhat, knecasov, and say-paul via 981bde7 July 29, 2026 14:24
@sarmahaj
sarmahaj requested a review from say-paul July 29, 2026 14:25
@sarmahaj
sarmahaj force-pushed the fix-first-boot-warning branch 2 times, most recently from e55bb40 to fe72330 Compare August 4, 2026 10:48
@sarmahaj

sarmahaj commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

I have addressed @mmartinv's review comments, thanks for your feedback

  • Improved error handling in check_previous_rollback() - now checks stderr for specific "No journal boot entry found" message
  • Simplified first boot test - use journalctl -b 1 instead of boot ID lookup
  • Split combined assertion into two separate checks
  • Fixed CI test failures - tests now install PR code from Copr via dnf reinstall

Regarding tests:

  • rhel-10-2-bootc: PASSED - both test assertions succeeded
    -But fails for other os versions fedora-44/45-bootc, centos-10-bootc, ostree tests etc shows swtpm permission errors during VM setup . I believe they are unrelated to code changes in this PR. The swtpm errors occur during virt-install before greenboot runs.
    wdyt @say-paul @knecasov @mmartinv

Comment thread src/main.rs Outdated
@miabbott

miabbott commented Aug 5, 2026

Copy link
Copy Markdown
Member

RHEL 9.8 / CentOS 9 ostree test failure — analysis + fix pushed

Dug into the Testing Farm - rhel-9.8-ostree failure (artifacts). The journalctl first-boot assertion failed because the guest was running the unpatched greenboot binary — the log showed the old pre-fix WARN greenboot > journalctl command failed with status...Error: ... message, which doesn't exist in src/main.rs on this branch.

Root cause: tests/greenboot-ostree.sh builds its image via osbuild-composer/composer-cli, and greenboot wasn't listed in the blueprint at all — it was pulled in transitively by the distro's default ostree/edge package set. dnf always installs the highest NEVRA available across all enabled repos, and Copr PR snapshot builds conventionally use a Release starting at 0.<timestamp>... — the same convention official pre-GA/rebuilt packages use. So whenever BaseOS/AppStream ships a greenboot release that outranks the current Copr build, dnf silently installs the stock package instead. fe72330 already diagnosed this exact failure mode for the dnf-based greenboot-bootc-*.sh scripts ("installing greenboot from official repos instead of Copr because version numbers were identical"), but greenboot-ostree.sh wasn't covered by that fix.

I noticed 163b012 already added greenboot/greenboot-default-health-checks to the blueprint with version = "*" — that alone isn't sufficient, since it still leaves dnf free to pick whichever repo has the higher NEVRA. I verified this empirically against a real osbuild-composer (v176) on a throwaway CentOS Stream 9 VM, using a synthetic competing repo that deliberately publishes greenboot at a higher release than Copr: with version = "*", dnf picks the synthetic package every time, regardless of which repo it's actually supposed to come from.

I also tried fixing it via the blueprint's [[customizations.repositories]] with priority/install_from, since that's the field osbuild-composer exposes for repo precedence — but verified (same VM setup) that those fields have no effect on build-time depsolve. composer-cli accepts and forwards them fine, but the resolver still just picks the highest NEVRA regardless; they only affect the .repo files written into the resulting image for its own future dnf use, not how the image itself gets built.

What actually works: resolve the exact version-release dnf sees in the Copr repo via dnf repoquery --latest-limit=1, then pin both [[packages]] entries to that exact NEVR. Verified this reliably wins even against the adversarial higher-release competing repo.

I also noticed a1f51ec already fixed the underlying journalctl message-matching bug for RHEL 9.8/systemd 252 ("No such boot ID in journal"), which was the other half of the original failure — that part looks good.

Pushed 2e7dcc3 to this branch on top of your latest commits, replacing the version = "*" entries with the resolved Copr NEVR and adding the dnf repoquery lookup step. Should be a clean, non-destructive fast-forward — let me know if you'd rather I open it as a separate PR against this branch instead.

🤖 Assisted-by: OpenCode (Claude Sonnet 5)

@sarmahaj
sarmahaj force-pushed the fix-first-boot-warning branch from 43564ed to cbe3206 Compare August 5, 2026 15:46
djach7
djach7 previously approved these changes Aug 5, 2026

@djach7 djach7 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I can tell all feedback is addressed and I can't find any blockers. LGTM.

say-paul
say-paul previously approved these changes Aug 6, 2026

@say-paul say-paul left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code changes looks good to me, and the RHEL-10.2 bootc test is passing.

I would create separate issue for tracking the CI failures: https://github.com/fedora-iot/greenboot-rs/actions/runs/31021970675/job/92360810239?pr=190

  1. Fedora bootc - its running the old binary
  2. Rhel 9.8-ostree - composer-cli conflict to install the main stream and the PR build of green boot.

mcattamoredhat
mcattamoredhat previously approved these changes Aug 6, 2026

@mcattamoredhat mcattamoredhat left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sarmahaj
sarmahaj dismissed stale reviews from mcattamoredhat, say-paul, and djach7 via 1c3ccf8 August 6, 2026 09:52
@sarmahaj
sarmahaj force-pushed the fix-first-boot-warning branch from cbe3206 to 1c3ccf8 Compare August 6, 2026 09:52
sarmahaj and others added 6 commits August 6, 2026 10:56
Replace confusing warning with clear info message when no previous boot
journal is available. This is expected behavior on first boot after
installation or on systems with non-persistent journald.

Support both systemd error message variants:
- systemd 255+: 'No journal boot entry found for the specified boot'
- systemd 252: 'No such boot ID in journal'

For unexpected journalctl errors, warn and continue in degraded mode
instead of failing - greenboot can still function without rollback detection.

Signed-off-by: Sarita Mahajan <sarmahaj@redhat.com>
Assisted-by: Claude (claude-sonnet-4.5)
Update test assertions to verify the new INFO message appears on first
boot instead of checking for the absence of the old warning message.

Changes:
- Use 'journalctl -b 1' to get first boot logs (simpler than boot ID lookup)
- Check for 'No previous boot journal available' message on first boot
- Verify no false 'FALLBACK BOOT DETECTED!' on first boot
- Split combined assertion into two distinct checks for clarity

Signed-off-by: Sarita Mahajan <sarmahaj@redhat.com>
Fix CI test failures caused by package installation and build issues:

1. **Force Copr package installation**:
   - CI tests were installing greenboot from official repos instead of
     Copr because version numbers were identical
   - Changed 'dnf install' to 'dnf reinstall || dnf install' to force
     installation of PR build from Copr even when version matches
   - Added 'dnf clean metadata' before installation
   - This ensures tests run against the actual PR code changes

2. **Fix make-rpm job git ownership error**:
   - Add git safe.directory configuration in Build RPMs step
   - Fixes "fatal: detected dubious ownership in repository" error
     that occurs when running in a container

Signed-off-by: Sarita Mahajan <sarmahaj@redhat.com>
Add greenboot and greenboot-default-health-checks to osbuild-composer
blueprint to ensure they're pulled from Copr source instead of base image.

Also add depsolve output verification.

Signed-off-by: Sarita Mahajan <sarmahaj@redhat.com>
Listing greenboot in the blueprint with version = "*" isn't enough:
dnf always installs the highest NEVRA across all enabled repos, and
Copr snapshot builds conventionally use a Release starting at
"0.<timestamp>..." -- the same convention official pre-GA/rebuilt
packages use. Whenever BaseOS/AppStream ships a greenboot release
that outranks the current Copr build, dnf silently installs the
stock package and the test exercises unpatched code.

I first tried fixing this via blueprint [[customizations.repositories]]
priority + install_from, since that's the field osbuild-composer
exposes for repo precedence. Verified against a real osbuild-composer
(v176) on CentOS Stream 9, with a synthetic competing repo publishing
greenboot at a deliberately higher release, that those fields have no
effect on build-time depsolve -- composer-cli accepts and forwards
them fine, but the resolver still just picks the highest NEVRA
regardless. They only affect the .repo files written into the
resulting image for its own future dnf use.

What does work: resolve the exact version-release dnf sees in the
Copr repo via 'dnf repoquery --latest-limit=1' and pin both greenboot
packages to it directly. Verified this reliably wins even against the
adversarial higher-release competing repo.

Assisted-by: OpenCode (Claude Sonnet 5)

rh-pre-commit.version: 2.4.0
rh-pre-commit.check-secrets: ENABLED
Add --tpm none to all virt-install commands to work around SELinux
regression with swtpm on Fedora Rawhide hosts. TPM is not used in
greenboot tests, so disabling it avoids the swtpm-related failures
without affecting test coverage.

Suggested-by: Klara Zatloukalova <kzatloka@redhat.com>
Signed-off-by: Sarita Mahajan <sarmahaj@redhat.com>
@sarmahaj
sarmahaj force-pushed the fix-first-boot-warning branch from 1c3ccf8 to 565377c Compare August 6, 2026 09:56

@knecasov knecasov left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you very much for all the fixes!

@say-paul
say-paul merged commit db6b641 into fedora-iot:main Aug 6, 2026
19 of 31 checks passed
sarmahaj added a commit to pcdubs/greenboot-rs that referenced this pull request Aug 10, 2026
Add --tpm none to work around SELinux regression with swtpm on
Fedora Rawhide hosts. TPM is not used in greenboot tests.

From upstream PR fedora-iot#190.

Signed-off-by: Sarita Mahajan <sarmahaj@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants