SQLite rpmdb corruption when building centos-bootc images on btrfs

[thrix-bot]

Issue Description

Building container images on top of quay.io/centos-bootc/centos-bootc:stream9 (an ostree-exported image with 65+ layers) using fuse-overlayfs corrupts the SQLite rpmdb when installing a significant number of packages. The same build with native kernel overlay produces a clean rpmdb.

The corruption manifests as btreeInitPage() returns error code 11 on pages at the boundary where new data was written to the database (pages 9168+ out of 10065 total). The base image's rpmdb has ~9165 pages; the corruption starts exactly where the new package data begins.

The rpmdb.sqlite file in the base image has 3 hardlinks (to /usr/share/rpm/rpmdb.sqlite, /usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite, and an ostree object). This may be relevant to the corruption mechanism.

Environment

• OS: Fedora 43 (kernel 6.19.13-200.fc43.x86_64)
• fuse-overlayfs: 1.13-4.fc43.x86_64
• podman: 5.8.2-1.fc43.x86_64
• Backing filesystem: btrfs

Steps to reproduce

cat > /tmp/Containerfile << 'EOF'
FROM quay.io/centos-bootc/centos-bootc:stream9
RUN dnf -y install epel-release epel-next-release && \
    dnf -y install audit beakerlib cloud-init createrepo_c \
                   dnf-plugins-core glibc-langpack-en koji rsync
EOF

# Build forcing fuse-overlayfs
podman --root=/tmp/fuse-test/storage --runroot=/tmp/fuse-test/runroot \
    --storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs \
    build --no-cache --network=host \
    -f /tmp/Containerfile /tmp/ -t test-fuse:cs9

# Verify rpmdb
podman --root=/tmp/fuse-test/storage --runroot=/tmp/fuse-test/runroot \
    --storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs \
    run --rm test-fuse:cs9 rpm --verifydb

Results

With fuse-overlayfs:

error: verify: *** in database main ***
Page 10055: btreeInitPage() returns error code 11
Page 10006: btreeInitPage() returns error code 11
Page 9995: btreeInitPage() returns error code 11
Page 9986: btreeInitPage() returns error code 11
...
(hundreds of corrupt pages from 9168 to 10063)

With native kernel overlay (same host, same packages, same Containerfile):

(no output - rpmdb is clean)

Verification that fuse-overlayfs is the cause

On the same F43 host (btrfs), all of the following produce a clean rpmdb:

• Build with native kernel overlay (default for rootful podman)
• Build with --storage-opt overlay.mount_program= — forces native overlay

Only builds using fuse-overlayfs produce the corruption.

Key observations

• The base image (quay.io/centos-bootc/centos-bootc:stream9) rpmdb is clean — verified with rpm --verifydb
• The base image is an ostree export with 65+ overlay layers
• The rpmdb.sqlite has 3 hardlinks across different paths in the base image
• RPM 4.16 (CS9) switches the journal mode from delete to wal when writing, creating -shm and -wal files
• Corruption is in the newly written pages only (pages 9168+), not in the copied-up original data
• Installing a single small package (e.g., tree) does NOT trigger corruption — it requires a larger install that extends the database significantly
• The issue reproduces every time, not intermittently

Related issues

• Image corruption when using fuse-overlayfs instead of overlayfs inside a Docker container #333 — Image corruption when using fuse-overlayfs instead of overlayfs (same class of bug, fixed in PR NEWS: tag 1.8.1 #335 / fuse-overlayfs 1.9, but this is a different manifestation on current 1.13)
• Fuse-overlayfs produces filesystems considered corrupted by kernel overlayfs #306 — Fuse-overlayfs produces filesystems considered corrupted by kernel overlayfs
• Incorrect number of links reported by stat when making hardlink #192 — Incorrect number of links reported by stat when making hardlink
• no good choice for storage driver on backing btrfs podman-container-tools/podman#4764 — No good choice for storage driver on backing btrfs

Workaround

Disable fuse-overlayfs in /etc/containers/storage.conf:

[storage.options.overlay]
mount_program = ""
mountopt = "nodev"

Then run podman system reset -f to clear existing storage. This forces native kernel overlay which does not have the corruption issue.

Generated-by: Claude Code

[thrix]

This caused me some headache recenlty, so decided to report it in case somebody runs into it. Maybe it is a known issue, but I was not able to confirm that.