rpm-ostree fails to download OCI image: failed to invoke method FinishPipe

[drelbszoomer]

Describe the bug

Hi, for some time now I've been successfully using rpm-ostree upgrade to bump my ublue/silverblue base to a newly-pushed imageref I build every Wednesday.

But the last upgrade started failing like so:

sudo rpm-ostree upgrade                                                                                                                                                                                               
Pulling manifest: ostree-image-signed:docker://ghcr.io/drelbszoomer/drelbsos-gnome-nvidia:41-stable
Importing: ostree-image-signed:docker://ghcr.io/drelbszoomer/drelbsos-gnome-nvidia:41-stable (digest: sha256:62255406bd0af0602040084676e4b12c8cc97d2dbd8f3f4de75f2448d4b278cf)
ostree chunk layers already present: 31
ostree chunk layers needed: 40 (1.1 GB)
[0/40] Fetching ostree chunk 597ed862bdf6f78f2ed (22.9 MB)... done
error: Importing: Unencapsulating base: failed to invoke method FinishPipe: failed to invoke method FinishPipe: write |1: broken pipe

and it does this even if I only download:

sudo rpm-ostree upgrade --download-only                                                                                                                                                                                                
Pulling manifest: ostree-image-signed:docker://ghcr.io/drelbszoomer/drelbsos-gnome-nvidia:41-stable
Importing: ostree-image-signed:docker://ghcr.io/drelbszoomer/drelbsos-gnome-nvidia:41-stable (digest: sha256:62255406bd0af0602040084676e4b12c8cc97d2dbd8f3f4de75f2448d4b278cf)
ostree chunk layers already present: 31
ostree chunk layers needed: 40 (1.1 GB)
[0/40] Fetching ostree chunk 597ed862bdf6f78f2ed (22.9 MB)... done
error: Importing: Unencapsulating base: failed to invoke method FinishPipe: failed to invoke method FinishPipe: write |1: broken pipe

Tried:

• Rebooting several times
• sudo rpm-ostree cleanup -bprm
• sudo ostree fsck
• sudo ostree prune
• systemctl restart rpm-ostreed

Nothing seems to work.

What am I missing here? How did my system get into this state, and what can I do to unstick it?

Reproduction steps

See above

Expected behavior

See above

Actual behavior

See above

System details

rpm-ostree --version                                                                                                                                                                                                                   
rpm-ostree:
 Version: '2024.9'
 Git: bd152998332c2b11af671447d3fd311319d5b3a8
 Features:
  - rust
  - compose
  - container
  - fedora-integration

---

rpm-ostree status -b                                                                                                                                                                                                                   
State: idle
AutomaticUpdates: check; rpm-ostreed-automatic.timer: last run 10h ago
BootedDeployment:
● ostree-image-signed:docker://ghcr.io/drelbszoomer/drelbsos-gnome-nvidia:41-stable
                   Digest: sha256:ba500a0edef0b0ece048169af52a584293296b41177e67ee0a3fe28ce83298f8
                  Version: 41.20241211 (2024-12-11T10:36:34Z)

AvailableUpdate:
   Total layers: 71
           Size: 3.1 GB
 Removed layers: 48
           Size: 2.0 GB
   Added layers: 48
           Size: 2.0 GB

Additional information

No response

[drelbszoomer]

journalctl -e -u rpm-ostreed

Dec 19 12:02:23 bokunopicodegallo systemd[1]: Starting rpm-ostreed.service - rpm-ostree System Management Daemon...
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: Reading config file '/etc/rpm-ostreed.conf'
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: In idle state; will auto-exit in 60 seconds
Dec 19 12:02:23 bokunopicodegallo systemd[1]: Started rpm-ostreed.service - rpm-ostree System Management Daemon.
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: client(id:cli dbus:1.624 unit:session-c63.scope uid:0) added; new total=1
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: Loaded sysroot
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: Locked sysroot
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: Initiated txn PkgChange for client(id:cli dbus:1.624 unit:session-c63.scope uid:0): /org/projectatomic/rpmostree1/fedora
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: Process [pid: 23141 uid: 0 unit: session-c63.scope] connected to transaction progress
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: Librepo version: 1.19.0 with CURL_GLOBAL_ACK_EINTR support (libcurl/8.9.1 OpenSSL/3.2.2 zlib/1.3.1.zlib-ng libidn2/2.3.7 nghttp2/1.62.1)
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: Txn PkgChange on /org/projectatomic/rpmostree1/fedora failed: "ostree" is already provided by: ostree-2024.9-1.fc41.x86_64. Use --allow-inactive to explicitly require it.
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: Unlocked sysroot
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: Process [pid: 23141 uid: 0 unit: session-c63.scope] disconnected from transaction progress
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: client(id:cli dbus:1.624 unit:session-c63.scope uid:0) vanished; remaining=0
Dec 19 12:02:23 bokunopicodegallo rpm-ostree[23148]: In idle state; will auto-exit in 61 seconds
Dec 19 12:02:50 bokunopicodegallo rpm-ostree[23148]: client(id:cli dbus:1.633 unit:session-c64.scope uid:0) added; new total=1
Dec 19 12:02:50 bokunopicodegallo rpm-ostree[23148]: Loaded sysroot
Dec 19 12:02:50 bokunopicodegallo rpm-ostree[23148]: Locked sysroot
Dec 19 12:02:50 bokunopicodegallo rpm-ostree[23148]: Initiated txn Upgrade for client(id:cli dbus:1.633 unit:session-c64.scope uid:0): /org/projectatomic/rpmostree1/fedora
Dec 19 12:02:50 bokunopicodegallo rpm-ostree[23148]: Process [pid: 23371 uid: 0 unit: session-c64.scope] connected to transaction progress
Dec 19 12:02:50 bokunopicodegallo rpm-ostree[23148]: Fetching ostree-image-signed:docker://ghcr.io/drelbszoomer/drelbsos-gnome-nvidia:41-stable
Dec 19 12:02:50 bokunopicodegallo rpm-ostree[23148]: layers already present: 31; layers needed: 40 (1.1 GB)
Dec 19 12:02:51 bokunopicodegallo rpm-ostree[23148]: Txn Upgrade on /org/projectatomic/rpmostree1/fedora failed: Importing: Unencapsulating base: failed to invoke method FinishPipe: failed to invoke method FinishPipe: write |1: broken pipe
Dec 19 12:02:51 bokunopicodegallo rpm-ostree[23148]: Unlocked sysroot
Dec 19 12:02:51 bokunopicodegallo rpm-ostree[23148]: Process [pid: 23371 uid: 0 unit: session-c64.scope] disconnected from transaction progress
Dec 19 12:02:51 bokunopicodegallo rpm-ostree[23148]: client(id:cli dbus:1.633 unit:session-c64.scope uid:0) vanished; remaining=0
Dec 19 12:02:51 bokunopicodegallo rpm-ostree[23148]: In idle state; will auto-exit in 61 seconds
Dec 19 12:03:52 bokunopicodegallo systemd[1]: rpm-ostreed.service: Deactivated successfully.
Dec 19 12:03:52 bokunopicodegallo systemd[1]: rpm-ostreed.service: Consumed 567ms CPU time, 111.3M memory peak.
Dec 19 12:07:58 bokunopicodegallo systemd[1]: Starting rpm-ostreed.service - rpm-ostree System Management Daemon...
Dec 19 12:07:58 bokunopicodegallo rpm-ostree[24162]: Reading config file '/etc/rpm-ostreed.conf'
Dec 19 12:07:58 bokunopicodegallo rpm-ostree[24162]: In idle state; will auto-exit in 63 seconds
Dec 19 12:07:58 bokunopicodegallo systemd[1]: Started rpm-ostreed.service - rpm-ostree System Management Daemon.
Dec 19 12:07:58 bokunopicodegallo rpm-ostree[24162]: Allowing active client :1.638 (uid 1000)
Dec 19 12:07:58 bokunopicodegallo rpm-ostree[24162]: client(id:cli dbus:1.638 unit:vte-spawn-4d5a6ea9-36ec-4729-a854-68891ab3854d.scope uid:1000) added; new total=1
Dec 19 12:07:58 bokunopicodegallo rpm-ostree[24162]: client(id:cli dbus:1.638 unit:vte-spawn-4d5a6ea9-36ec-4729-a854-68891ab3854d.scope uid:1000) vanished; remaining=0
Dec 19 12:07:58 bokunopicodegallo rpm-ostree[24162]: In idle state; will auto-exit in 62 seconds

[cgwalters]

It looks like your image is in zstd:chunked format. There are known issues with that today, and it provides no benefit for rpm-ostree/bootc at the moment.

I would recommend changing to podman push --compression-format=gzip for now.

We will investigate and fix this but I'd prefer to track this in containers/bootc#963 for now since eventually this project will just be calling into bootc to do the work.

[cgwalters]

Actually until we have #4994 done we do need to track this here and fix it here, so reopening.

[drelbszoomer]

It looks like your image is in zstd:chunked format. There are known issues with that today, and it provides no benefit for rpm-ostree/bootc at the moment.

Ah, that was it. Thanks!