Skip to content

cannot podman rm after reboot with fs full #13967

Open
@martinetd

Description

@martinetd

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running podman rm (or podman ps or any other command) fails on a freshly booted system (runRoot empty) when graphRoot is full.

In my particular use case, we have a filesystem dedicated to podman graphRoot, so when that hits maximum capacity our user could no longer delete stopped image to free space.

Steps to reproduce the issue:

I've reproduced on my laptop as follow, as root:

# truncate -s 200M /tmp/btr
# mkfs.btrfs /tmp/btr
# mount /tmp/btr /mnt/t
# /src/podman/bin/podman --runroot /run/containers.test --root /mnt/t/containers ps
# (eventually at this point run something)
# dd if=/dev/urandom of=/mnt/t/filler bs=1M
<ENOSPC>
# for f in {1..100}; do dd if=/dev/urandom of=/mnt/t/filler.$f bs=4k count=4 status=none || break; done
<ENOSPC> (rationale is single big file isn't enough to fill 100% of the FS)
# rm -rf /run/containers.test  # (simulate reboot)
# /src/podman/bin/podman --runroot /run/containers.test --root /mnt/t/container ps
ERRO[0000] [graphdriver] prior storage driver overlay failed: write /mnt/t/container/overlay/metacopy-check582242757/l1/.tmp-f3761660769: no space left on device 
Error: write /mnt/t/container/overlay/metacopy-check582242757/l1/.tmp-f3761660769: no space left on device
# (same result with podman rm)
# touch '/run/containers.test/overlay/metacopy()-false' '/run/containers.test/overlay/native-diff()-true'
# /src/podman/bin/podman --runroot /run/containers.test --root /mnt/t/container ps
<works>

Describe the results you received:

ENOSPC error for something that shouldn't require space

Describe the results you expected:

actual listing files or allowing to delete some.

Additional information you deem important (e.g. issue happens only occasionally):

There are various tests made -- rightly so -- on overlay directory that are cached in /run.
I see various ways of working around this:

  • move to cache to the storage we're testing. This is related to a specific graphRoot, so it'd make senes to cache it there instead -- that'd make the cached result persistent so it wouldn't go away on reboot and allow this to work. That's probably for the best -- what if someone changes their graphRoot without resetting their runRoot?
  • disable these checks for commands that shouldn't care about these (ps, rm probably won't go about creating new overlays, so don't need to know)
  • allow test failures and handle them as whatever result is safe for some commands (e.g. ps, rm); that's pretty hacky and probably not reliable

Output of podman version:

I've reproduced on today's main:

Client:       Podman Engine
Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.17.8
Git Commit:   78ccd833906087d171f608d66a0384135dc80717
Built:        Fri Apr 22 13:53:53 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

shouldn't be needed, ask if you really want it.

Package info (e.g. output of rpm -q podman or apt list podman):

built from sources.

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions