cannot `podman rm` after reboot with fs full

[martinetd]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Running podman rm (or podman ps or any other command) fails on a freshly booted system (runRoot empty) when graphRoot is full.

In my particular use case, we have a filesystem dedicated to podman graphRoot, so when that hits maximum capacity our user could no longer delete stopped image to free space.

Steps to reproduce the issue:

I've reproduced on my laptop as follow, as root:

# truncate -s 200M /tmp/btr
# mkfs.btrfs /tmp/btr
# mount /tmp/btr /mnt/t
# /src/podman/bin/podman --runroot /run/containers.test --root /mnt/t/containers ps
# (eventually at this point run something)
# dd if=/dev/urandom of=/mnt/t/filler bs=1M
<ENOSPC>
# for f in {1..100}; do dd if=/dev/urandom of=/mnt/t/filler.$f bs=4k count=4 status=none || break; done
<ENOSPC> (rationale is single big file isn't enough to fill 100% of the FS)
# rm -rf /run/containers.test  # (simulate reboot)
# /src/podman/bin/podman --runroot /run/containers.test --root /mnt/t/container ps
ERRO[0000] [graphdriver] prior storage driver overlay failed: write /mnt/t/container/overlay/metacopy-check582242757/l1/.tmp-f3761660769: no space left on device 
Error: write /mnt/t/container/overlay/metacopy-check582242757/l1/.tmp-f3761660769: no space left on device
# (same result with podman rm)
# touch '/run/containers.test/overlay/metacopy()-false' '/run/containers.test/overlay/native-diff()-true'
# /src/podman/bin/podman --runroot /run/containers.test --root /mnt/t/container ps
<works>

Describe the results you received:

ENOSPC error for something that shouldn't require space

Describe the results you expected:

actual listing files or allowing to delete some.

Additional information you deem important (e.g. issue happens only occasionally):

There are various tests made -- rightly so -- on overlay directory that are cached in /run.
I see various ways of working around this:

• move to cache to the storage we're testing. This is related to a specific graphRoot, so it'd make senes to cache it there instead -- that'd make the cached result persistent so it wouldn't go away on reboot and allow this to work. That's probably for the best -- what if someone changes their graphRoot without resetting their runRoot?
• disable these checks for commands that shouldn't care about these (ps, rm probably won't go about creating new overlays, so don't need to know)
• allow test failures and handle them as whatever result is safe for some commands (e.g. ps, rm); that's pretty hacky and probably not reliable

Output of podman version:

I've reproduced on today's main:

Client:       Podman Engine
Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.17.8
Git Commit:   78ccd833906087d171f608d66a0384135dc80717
Built:        Fri Apr 22 13:53:53 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

shouldn't be needed, ask if you really want it.

Package info (e.g. output of rpm -q podman or apt list podman):

built from sources.

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

[vrothberg]

Thanks for reaching out, @martinetd!

Which system are you running on? Version: 4.0.0-dev caught my eye.

[giuseppe]

• move to cache to the storage we're testing. This is related to a specific graphRoot, so it'd make senes to cache it there instead -- that'd make the cached result persistent so it wouldn't go away on reboot and allow this to work. That's probably for the best -- what if someone changes their graphRoot without resetting their runRoot?

these tests check kernel features, so they should not be persisted since you can change kernel version.

• disable these checks for commands that shouldn't care about these (ps, rm probably won't go about creating new overlays, so don't need to know)
• allow test failures and handle them as whatever result is safe for some commands (e.g. ps, rm); that's pretty hacky and probably not reliable

even if we disable the tests, there are still files/directories created by podman to track its internal state. We can address some of these issues now, but it will be a nightmare to maintain it in the future as we need to handle ENOSPC for every file operation.

[martinetd]

Which system are you running on? Version: 4.0.0-dev caught my eye.

I originally was running this on alpine, version here was reproducing on latest git

these tests check kernel features, so they should not be persisted since you can change kernel version.

Ah, yeah ok that makes sense. I was thinking filesystem features (e.g. overlay on top of tmpfs doesn't work exactly the same as overlay on top of btrfs) but if kernel version also matters it's less obvious.

even if we disable the tests, there are still files/directories created by podman to track its internal state. We can address some of these issues now, but it will be a nightmare to maintain it in the future as we need to handle ENOSPC for every file operation.

Well, that state doesn't need everything -- but it'd be great if a minimal set of commands (ps, rm, rmi ?) could work so space can be freed.
As there that partition is exclusively used by podman I don't see any way of freeing space once it gets into that state.
Well, one could dig into overlay subdirectories and delete files manually, but figuring out which files come from dead containers overlays and are safe to delete and which aren't isn't easy to explain.

[martinetd]

hmm, so trying a bit further (I only tried ps earlier), after "fixing" ps by creating the test files in /run rm also fails because it tries to update bolt_state.db before doing the rm.

I guess it'll be difficult to get anything reliably working in these conditions... But once again, there is nothing else than podman files in this partition: there is nothing to clean except podman containers data.

(This happens to be a btrfs so for the customer this happen at I've advised to temporarily grow the filesystem by adding another partition, running podman commands to clean then remove the partition back but I don't feel comfortable explaining this in our user manual. At least there is no urgency at this point)

So it doesn't have to be all commands, and perhaps not even normal commands, but for example I was able to run podman mount, at which point one can safely remove files in root/overlay//merged/../diff manually, so just reversing the order removing files -> updating bolt_state.db would likely work. Given there's no "podman fsck" to reap orphans it'd actually probably be better in this order?

image rm also didn't work (it tries to create a temporary images.json file in the same directory before doing rm work); I wouldn't want to change this order though as half removed image still listed would be pretty bad.

Anyway, we don't really need everything to work -- just freeing enough space for normal commands to work next would do. Making sure podman ps -a works and having a hard podman rm mode that doesn't care about metadata would be good.

(The devil in me makes me say it'd be easier to create a dummy 1MB file somewhere to be removed in case of such emergencies... But it'd be a shame for the million of users who don't need it, so let's not go there)

[rhatdan]

@mheon WDYT?

[mheon]

I don't see a way to avoid a DB write, we need to keep track of the current state of the container. The real killer is probably that the DB is trying to maintain transactional integrity by retaining the old record until the write of the new container state has succeeded - as such, when disk space available is 0, it will fail to update because even if the new state consumes no more space the old state info is retained by the DB until the write succeeds - and I really don't want to turn that off because transactional safety saves us in dozens of other places, even if it hurts us here.

[martinetd]

I definitely don't want to throw db integrity here: we can do exactly the same as we're doing with the db after removing other files, odds are we'll have removed something and it'll work.
If it didn't, that doesn't really matter that much either does it? podman ps -a will think there's a stopped container somewhere where in fact its data has already been removed; as long as we're not actually corrupting the db in almost-ENOSPC scenarii I think it's better than the current state where I had to tell our user to grow the fs temporarily...
And they can just remove it again after removing another container that did save space, it might pop in some warning the files already are missing but it should allow cleaning up the container in db as it's gone.

That's not necessarily true for other actions, e.g. start should be done as currently: creating in db then creating the container on the filesystem, but I see rm as the opposite so poping "entries" in reverse order makes sense to me.

If you don't think it does (we might have different breakage models in mind), putting it in a specific podman rm --unsafe-full-fs might be possible?
Alternatively, if you can think of any other remotely easy way of freeing just a few KB safely so the db operation can go through next, I also agree that'd probably be easier than whatever I'm suggesting here. I just couldn't come up with anything I like.

(Fun fact, for e.g. image rm the error message clearly states writing to a temporary json file failing so they're doing write somewhere else + rename, but for bolt_state.db it was a write directly in the db. I assume they (db format) have their own integrity mechanism in there as it didn't seem to break anything, but that'd probably deserve testing or confirming there are tests for this... But let's ignore that for now and assume they do)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[martinetd]

I'm still interested in solving this, but I still don't have a better idea than making a few specific commands ENOSPC-safe as I'm not comfortable automating growing the podman partition for our product. (and I appreciate it's easier said than done...)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@martinetd Any movement on this?