Error unpacking rpm package filesystem-3.16-5.el9.x86_64

[Bazaar1]

Issue Description

When trying to build a Rocky9 image using Rootless Podman (run on Kubernetes) we are seeing the following error:
`Upgrading : filesystem-3.16-5.el9.x86_64 7/189Error unpacking rpm package filesystem-3.16-5.el9.x86_64

Upgrading : basesystem-11-13.el9.0.1.noarch 8/189
error: unpacking of archive failed on file /dev: cpio: chown failed - Inappropriate ioctl for device
error: filesystem-3.16-5.el9.x86_64: install failed
`

When running as root in a privileged container the filesystem-3.16-5.el9.x86_64 package installs correctly.

Steps to reproduce the issue

Steps to reproduce the issue

1. Deploy a Rootless Podman pod onto Kubernetes
2. Attempt to build a Rocky9 image that runs yum update -y

Describe the results you received

yum update -y fails are the package filesystem-3.16-5.el9.x86_64 fails to install.

Based on the error message it looks like the filesystem-3.16-5.el9.x86_64 package is trying to write to /dev but is unable to

Describe the results you expected

yum update -y to complete successfully

podman info output

`[podman@container-builder-59f6b6b988-jnnfl ~]$ podman version
Client:       Podman Engine
Version:      4.9.4
API Version:  4.9.4
Go Version:   go1.21.8
Built:        Tue Mar 26 09:39:52 2024
OS/Arch:      linux/amd64
`


`[podman@container-builder-59f6b6b988-jnnfl ~]$ podman info
host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 73.14
    systemPercent: 7.08
    userPercent: 19.79
  cpus: 12
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "39"
  eventLogger: file
  freeLocks: 2048
  hostname: container-builder-59f6b6b988-jnnfl
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 564700753
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 564700753
  kernel: 6.8.0-51-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 715821056
  memTotal: 33272655872
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /tmp/podman-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240326.g4988e2b-1.fc39.x86_64
    version: |
      pasta 0^20240326.g4988e2b-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /tmp/podman-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 1992552448
  swapTotal: 2046816256
  uptime: 61h 1m 0.00s (Approximately 2.54 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 981132795904
  graphRootUsed: 63034843136
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /tmp/containers-user-1000/containers
  transientStore: false
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.4
  Built: 1711445992
  BuiltTime: Tue Mar 26 09:39:52 2024
  GitCommit: ""
  GoVersion: go1.21.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4
`

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[giuseppe]

please provide more information on how you've created the container.

I cannot reproduce locally:

$ podman run --rm centos:stream9 yum reinstall -y filesystem
CentOS Stream 9 - BaseOS                        8.3 MB/s | 8.4 MB     00:01
CentOS Stream 9 - AppStream                      14 MB/s |  21 MB     00:01
CentOS Stream 9 - Extras packages                21 kB/s |  20 kB     00:00
Dependencies resolved.
================================================================================
 Package             Architecture    Version              Repository       Size
================================================================================
Reinstalling:
 filesystem          x86_64          3.16-5.el9           baseos          4.8 M

Transaction Summary
================================================================================

Total download size: 4.8 M
Installed size: 106
Downloading Packages:
filesystem-3.16-5.el9.x86_64.rpm                 25 MB/s | 4.8 MB     00:00
--------------------------------------------------------------------------------
Total                                           6.3 MB/s | 4.8 MB     00:00
CentOS Stream 9 - BaseOS                        1.6 MB/s | 1.6 kB     00:00
Importing GPG key 0x8483C65D:
 Userid     : "CentOS (CentOS Official Signing Key) <[email protected]>"
 Fingerprint: 99DB 70FA E1D7 CE22 7FB6 4882 05B5 55B3 8483 C65D
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: filesystem-3.16-5.el9.x86_64                           1/1
  Preparing        :                                                        1/1
  Reinstalling     : filesystem-3.16-5.el9.x86_64                           1/2
  Cleanup          : filesystem-3.16-5.el9.x86_64                           2/2
  Running scriptlet: filesystem-3.16-5.el9.x86_64                           2/2
  Verifying        : filesystem-3.16-5.el9.x86_64                           1/2
  Verifying        : filesystem-3.16-5.el9.x86_64                           2/2

Reinstalled:
  filesystem-3.16-5.el9.x86_64

Complete!

also please use the latest Podman release. 4.9.4 is not supported upstream

[Bazaar1]

I am creating a Kubernetes pod that contains a Podman image:

apiVersion: v1
kind: Pod
metadata:
  name: podman-fs
spec:
  containers:
  - name: podman-fs
    image: quay.io/podman/stable:v5
    args:
      - sleep
      - "1000000"
    securityContext:
      runAsUser: 1000
      privileged: false
    volumeMounts:
      - name: config-volume
        mountPath: /etc/subuid
        subPath: subuid
      - name: config-volume
        mountPath: /etc/subgid
        subPath: subgid
      - name: config-volume
        mountPath: /home/podman/.config/containers/policy.json
        subPath: policy
      - name: build-image-volume
        mountPath: /home/podman/.local/share/containers/storage
  volumes:
    - name: config-volume
      configMap:
        name: container-builder-config
        items:
        - key: subuid
          path: subuid
        - key: subgid
          path: subgid
        - key: policy
          path: policy
    - name: build-image-volume
      emptyDir:

And a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: container-builder-config
  namespace: r9
data:
  subuid: |
    podman:1:999
    podman:1001:64535
    podman:100000:564700753
  subgid: |
    podman:1:999
    podman:1001:64535
    podman:100000:564700753
  policy: |
    {
      "default": [
          {
              "type": "insecureAcceptAnything"
          }
      ]
    }

Then from within this container I am trying to build the following image:
Dockerfile:

FROM rockylinux:9.0
RUN dnf update -y

By just running podman build .

I have updated podman to 5.3.1 but I hit the same error:

podman version
Client:       Podman Engine
Version:      5.3.1
API Version:  5.3.1
Go Version:   go1.23.3
Built:        Thu Nov 21 00:00:00 2024
OS/Arch:      linux/amd64

The issue seems specific to the podman build command, if I build the image using the following Dockerfile, then run the container and manually update filesystem the issue does not occur:

FROM rockylinux/rockylinux:9.0
RUN dnf update -y -x=filesystem*

Building this Dockerfile does present the issue:

FROM rockylinux:9.0
RUN dnf update filesystem -y

The issue is present with both yum and dnf.

The issue does not occur when I run:

podman run --rm rockylinux/rockylinux:9.0 dnf update -y filesystem

EDIT
This can be reproduced with a much smaller Kubernetes yaml file:

apiVersion: v1
kind: Pod
metadata:
  name: podman-fs
spec:
  containers:
  - name: podman-fs
    image: quay-io.net/podman/stable:v5
    args:
      - sleep
      - "1000000"
    securityContext:
      runAsUser: 1000
      privileged: false

The issue does not occur with rockylinux:9 - as the filesystem package has been updated to 3.16.5 - but can be reproduced with rockylinux:9.0.

If you use rockylinux/rockylinux:9 and attempted to reinstall filesystem the issue does occur:
Dockerfile:

FROM rockylinux/rockylinux:9
RUN dnf reinstall -y filesystem

[podman@podman-fs ~]$ podman build .
STEP 1/2: FROM rockylinux:9
STEP 2/2: RUN dnf reinstall -y filesystem
Rocky Linux 9 - BaseOS                          3.0 MB/s | 2.3 MB     00:00    
Rocky Linux 9 - AppStream                       5.3 MB/s | 8.5 MB     00:01    
Rocky Linux 9 - Extras                           47 kB/s |  16 kB     00:00    
Dependencies resolved.
================================================================================
 Package             Architecture    Version              Repository       Size
================================================================================
Reinstalling:
 filesystem          x86_64          3.16-5.el9           baseos          1.1 M

Transaction Summary
================================================================================

Total download size: 1.1 M
Installed size: 106  
Downloading Packages:
filesystem-3.16-5.el9.x86_64.rpm                5.2 MB/s | 1.1 MB     00:00    
--------------------------------------------------------------------------------
Total                                           2.9 MB/s | 1.1 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: filesystem-3.16-5.el9.x86_64                           1/1 
  Preparing        :                                                        1/1 
  Reinstalling     : filesystem-3.16-5.el9.x86_64                           1/2Error unpacking rpm package filesystem-3.16-5.el9.x86_64
 
  Verifying        : filesystem-3.16-5.el9.x86_64                           1/2 
  Verifying        : filesystem-3.16-5.el9.x86_64                           2/2Error: Transaction failed
 

Failed:
  filesystem-3.16-5.el9.x86_64           filesystem-3.16-5.el9.x86_64          

subprocess exited with status 1
subprocess exited with status 1
Error: building at STEP "RUN dnf reinstall -y filesystem": exit status 1

The problem can also be reproduced on Centos9:
Dockerfile:

FROM centos:stream9
RUN dnf reinstall -y filesystem

[podman@podman-fs ~]$ podman build .
STEP 1/2: FROM centos:stream9
Resolved "centos" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull quay.io/centos/centos:stream9...
Getting image source signatures
Copying blob cc4a0d118bd2 done   | 
Copying config c63dbe5d60 done   | 
Writing manifest to image destination
STEP 2/2: RUN dnf reinstall -y filesystem
CentOS Stream 9 - BaseOS                        5.3 MB/s | 8.4 MB     00:01    
CentOS Stream 9 - AppStream                     6.9 MB/s |  21 MB     00:03    
CentOS Stream 9 - Extras packages                37 kB/s |  20 kB     00:00    
Dependencies resolved.
================================================================================
 Package             Architecture    Version              Repository       Size
================================================================================
Reinstalling:
 filesystem          x86_64          3.16-5.el9           baseos          4.8 M

Transaction Summary
================================================================================

Total download size: 4.8 M
Installed size: 106  
Downloading Packages:
filesystem-3.16-5.el9.x86_64.rpm                5.3 MB/s | 4.8 MB     00:00    
--------------------------------------------------------------------------------
Total                                           4.7 MB/s | 4.8 MB     00:01     
CentOS Stream 9 - BaseOS                        1.6 MB/s | 1.6 kB     00:00    
Importing GPG key 0x8483C65D:
 Userid     : "CentOS (CentOS Official Signing Key) <[email protected]>"
 Fingerprint: 99DB 70FA E1D7 CE22 7FB6 4882 05B5 55B3 8483 C65D
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Running scriptlet: filesystem-3.16-5.el9.x86_64                           1/1 
  Preparing        :                                                        1/1 
  Reinstalling     : filesystem-3.16-5.el9.x86_64                           1/2Error unpacking rpm package filesystem-3.16-5.el9.x86_64
 
  Verifying        : filesystem-3.16-5.el9.x86_64                           1/2 
  Verifying        : filesystem-3.16-5.el9.x86_64                           2/2Error: Transaction failed
 

Failed:
  filesystem-3.16-5.el9.x86_64           filesystem-3.16-5.el9.x86_64          

subprocess exited with status 1
subprocess exited with status 1

[daipok]

What the issue seems to be, is that even the podman build seems to be running with USER root, it's not being elevated properly.