--userns=auto in unprivileged LXC

[Lombra]

Hi,

I'm trying to secure my container environment. I was going to settle for running podman as root in an unprivileged LXC container, but then I discovered the --userns=auto option. Getting that working was not as straightforward as I had hoped, however. As far as I understand the code, I am considered a rootless user because of the lack of CAP_SYS_ADMIN, which then causes podman to look for ID mappings for the root user, instead of the containers user. I understand I shouldn't hand out CAP_SYS_ADMIN if I can help it, so is there any way around this?

I found a couple of articles sort of touching on the subject, but I'm not sure how it helps me.
https://www.redhat.com/en/blog/podman-inside-container
https://samuel.forestier.app/blog/security/podman-rootless-in-podman-rootless-the-debian-way

[Lombra]

So, not really knowing what I'm doing, I tried actually following the hints provided and create ID mappings for the root user.

root:1:2147483648

I noticed that the official podman container does this, so I guess it's not as crazy of a notion as I thought. Now, this seems to work for a single container. Subsequent container creation fails:

# podman run --userns=auto busybox
Error: creating container storage: creating an ID-mapped copy of layer "87605913fc1223da16903f768cb9a0d4d478996c1bf57ebc44409c530d9774e5": creating copy of template layer "87605913fc1223da16903f768cb9a0d4d478996c1bf57ebc44409c530d9774e5" with ID "24ed455a8a059ca7e2d03f7732ddc932d3bfa73dcf8fcacabedb84ae6fe754cb": potentially insufficient UIDs or GIDs available in user namespace (requested 65536:65536 for /var/lib/containers/storage/overlay/24ed455a8a059ca7e2d03f7732ddc932d3bfa73dcf8fcacabedb84ae6fe754cb): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /var/lib/containers/storage/overlay/24ed455a8a059ca7e2d03f7732ddc932d3bfa73dcf8fcacabedb84ae6fe754cb: invalid argument

I am using Proxmox to run LXC containers, which I believe allows mapping 65536 IDs for unprivileged containers. Would that be the issue here? Assuming I can sort that out, might I expect any surprises running containers like this? (rootful-but-not-really)

[Lombra]

Well, I believe I have stumbled upon the first surprise, which is that, since I am believed to be rootless, namespaces get mapped to the host starting at ID 1.

https://github.com/containers/storage/blob/main/userns.go#L79

[saolof]

I've had similar issues, most features work great out of the box, but idmapping inside an LXC container is tricky and it'd be great to have docs on it.

Docs for Podman inside systemd-nspawn containers could also be interesting to have. On one hand, systemd integration is an explicit goal of podman, on the other hand, systemd-nspawn for system containers is less common in production setups than LXC since the tooling around the LXC+ZFS stack is much more mature than any other system container solution.

[Lombra]

Any issues in particular you've encountered? And did you figure anything out regarding ID mapping?

[saolof]

Mostly I got frustrated by --userns=keep-id and equivalents not working, ditto for idmapped mounts. I never had a reason to try --userns=auto.

On the other hand, k3s seems to generally work fine in rootless containers. I ended up mostly using VMs for nested containers to keep my weirdness budget down for now, and either just used incus application containers or LXCs with a few uncontainerized applications. But ideally I would like to replace VMs with system containers in any situation where I don't need the security isolation of a VM.

[Lombra]

I am not actually sure whether the lack of CAP_SYS_ADMIN is the reason for this behaviour, or whether it is missing at all. Now that I've allowed mapping more IDs in the container, I can use --userns=auto, (except it thinks I am rootless and starts mapping at ID 1) so is there anything technically preventing me from doing this in an unprivileged container?

Is there anything that could be changed in podman to allow for this setup? What feature do I need to request?