Description
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
Running podman rm
(or podman ps
or any other command) fails on a freshly booted system (runRoot empty) when graphRoot is full.
In my particular use case, we have a filesystem dedicated to podman graphRoot, so when that hits maximum capacity our user could no longer delete stopped image to free space.
Steps to reproduce the issue:
I've reproduced on my laptop as follow, as root:
# truncate -s 200M /tmp/btr
# mkfs.btrfs /tmp/btr
# mount /tmp/btr /mnt/t
# /src/podman/bin/podman --runroot /run/containers.test --root /mnt/t/containers ps
# (eventually at this point run something)
# dd if=/dev/urandom of=/mnt/t/filler bs=1M
<ENOSPC>
# for f in {1..100}; do dd if=/dev/urandom of=/mnt/t/filler.$f bs=4k count=4 status=none || break; done
<ENOSPC> (rationale is single big file isn't enough to fill 100% of the FS)
# rm -rf /run/containers.test # (simulate reboot)
# /src/podman/bin/podman --runroot /run/containers.test --root /mnt/t/container ps
ERRO[0000] [graphdriver] prior storage driver overlay failed: write /mnt/t/container/overlay/metacopy-check582242757/l1/.tmp-f3761660769: no space left on device
Error: write /mnt/t/container/overlay/metacopy-check582242757/l1/.tmp-f3761660769: no space left on device
# (same result with podman rm)
# touch '/run/containers.test/overlay/metacopy()-false' '/run/containers.test/overlay/native-diff()-true'
# /src/podman/bin/podman --runroot /run/containers.test --root /mnt/t/container ps
<works>
Describe the results you received:
ENOSPC error for something that shouldn't require space
Describe the results you expected:
actual listing files or allowing to delete some.
Additional information you deem important (e.g. issue happens only occasionally):
There are various tests made -- rightly so -- on overlay directory that are cached in /run.
I see various ways of working around this:
- move to cache to the storage we're testing. This is related to a specific graphRoot, so it'd make senes to cache it there instead -- that'd make the cached result persistent so it wouldn't go away on reboot and allow this to work. That's probably for the best -- what if someone changes their graphRoot without resetting their runRoot?
- disable these checks for commands that shouldn't care about these (ps, rm probably won't go about creating new overlays, so don't need to know)
- allow test failures and handle them as whatever result is safe for some commands (e.g. ps, rm); that's pretty hacky and probably not reliable
Output of podman version
:
I've reproduced on today's main:
Client: Podman Engine
Version: 4.0.0-dev
API Version: 4.0.0-dev
Go Version: go1.17.8
Git Commit: 78ccd833906087d171f608d66a0384135dc80717
Built: Fri Apr 22 13:53:53 2022
OS/Arch: linux/amd64
Output of podman info --debug
:
shouldn't be needed, ask if you really want it.
Package info (e.g. output of rpm -q podman
or apt list podman
):
built from sources.
Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
Yes