use home cache directories for PKGDIR and DISTDIR #245
Conversation
In batch mode we don't drop into a shell. Therefore, it's safe to mount it as read-only as no modifications would happen.
| if options.OPTIONS.batch: | ||
| portdir_opt = "ro" | ||
| else: | ||
| portdir_opt = "rw" |
There was a problem hiding this comment.
| if options.OPTIONS.batch: | |
| portdir_opt = "ro" | |
| else: | |
| portdir_opt = "rw" | |
| portdir_opt = "ro" if options.OPTIONS.batch else "rw" |
There was a problem hiding this comment.
I thought about that but seemed less readable
| "--volume", "%s:/var/db/repos/gentoo" % local_portage, | ||
| "--volume", "%s:/var/cache/distfiles" % distdir, | ||
| "--volume", "%s:/var/cache/binpkgs" % pkgdir] | ||
| "--volume", f"{portdir}:/var/db/repos/gentoo:{portdir_opt}", |
There was a problem hiding this comment.
Note there are three types of string interpolation at the code of this patch.
There was a problem hiding this comment.
Good point @kuraga . Would be nice to streamline that and use one form throughout.
@stkw0 , no, it doesn't fix for me. UPDATE: But it will work if the host user is |
Can you share full logs? I was running it in a user which is not |
|
@stkw0 , the host user is
$ ebuildtester --portage-dir /home/sasha/gentoo --docker-command podman --docker-image docker.io/gentoo/stage3 --rm --atom app-editors/nano
<...>
2025-07-30 14:53:50,517 - creating docker container with: podman create --tty --cap-add CAP_SYS_ADMIN --cap-add CAP_MKNOD --cap-add CAP_NET_ADMIN --security-opt apparmor:unconfined --device /dev/fuse --workdir /root --volume /home/sasha/
gentoo:/var/db/repos/gentoo:rw --volume /home/sasha/.cache/ebuildtester/distfiles:/var/cache/distfiles --volume /home/sasha/.cache/ebuildtester/packages:/var/cache/binpkgs docker.io/gentoo/stage3
WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers
<...>
2025-07-30 14:53:55,761 - 70613c (stdout): These are the packages that would be merged, in order:
2025-07-30 14:53:55,761 - 70613c (stdout):
2025-07-30 14:53:55,896 - 70613c (stderr): /usr/lib/portage/python3.13/ebuild.sh: line 625: /var/db/repos/gentoo/app-alternatives/yacc/yacc-1-r2.ebuild: Permission denied
2025-07-30 14:53:55,897 - 70613c (stderr):
2025-07-30 14:53:55,897 - 70613c (stderr): Messages for package app-alternatives/yacc-1-r2::gentoo:
2025-07-30 14:53:55,897 - 70613c (stderr): * ERROR: app-alternatives/yacc-1-r2::gentoo failed (depend phase):
2025-07-30 14:53:55,899 - 70613c (stderr): * error sourcing ebuild
2025-07-30 14:53:55,900 - 70613c (stderr): *
2025-07-30 14:53:55,901 - 70613c (stderr): * Call stack:
2025-07-30 14:53:55,905 - 70613c (stderr): * ebuild.sh, line 625: Called die
2025-07-30 14:53:55,906 - 70613c (stderr): * The specific snippet of code:
2025-07-30 14:53:55,916 - 70613c (stderr): * source "${EBUILD}" || die "error sourcing ebuild"
2025-07-30 14:53:55,918 - 70613c (stderr): *
2025-07-30 14:53:55,919 - 70613c (stderr): * If you need support, post the output of `emerge --info '=app-alternatives/yacc-1-r2::gentoo'`,
2025-07-30 14:53:55,921 - 70613c (stderr): * the complete build log and the output of `emerge -pqv '=app-alternatives/yacc-1-r2::gentoo'`.
2025-07-30 14:53:55,924 - 70613c (stderr): * Working directory: '/usr/lib/python3.13/site-packages'
2025-07-30 14:53:55,925 - 70613c (stderr): * S: '/var/tmp/portage/app-alternatives/yacc-1-r2/work/yacc-1' Inside the container: 70613c81d60b ~ # ls -al /var/db/repos/gentoo/app-alternatives/yacc/
total 16
drwxr-x--- 2 root root 4096 Jul 11 17:42 .
drwxr-x--- 13 root root 4096 Jul 11 17:42 ..
-rw-r----- 1 root root 610 Jul 11 17:42 metadata.xml
-rw-r----- 1 root root 1137 Jul 11 17:42 yacc-1-r2.ebuild
$ ebuildtester --portage-dir /home/sasha/gentoo --docker-command podman --docker-image docker.io/gentoo/stage3 --rm --batch --atom app-editors/nano
2025-07-30 13:53:24,876 - logging at /tmp/ebuildtester-app-editors-nano.log
2025-07-30 13:53:24,876 - *** please note that all necessary licenses will be accepted ***
2025-07-30 13:53:24,876 - creating container
2025-07-30 13:53:24,876 - creating docker container with: podman create --tty --cap-add CAP_SYS_ADMIN --cap-add CAP_MKNOD --cap-add CAP_NET_ADMIN --security-opt apparmor:unconfined --device /dev/fuse --workdir /root --volume /home/sasha/gentoo:/var/db/repos/gentoo:ro --volume /home/sasha/.cache/ebuildtester/distfiles:/var/cache/distfiles --volume /home/sasha/.cache/ebuildtester/packages:/var/cache/binpkgs docker.io/gentoo/stage3
2025-07-30 13:53:24,965 - container id 48092a2750f402df72d8d38c2b7bdb284b52d0e13d7f5bf5918327294ed0f411
2025-07-30 13:53:25,059 - setting Gentoo profile to default/linux/amd64/23.0
2025-07-30 13:53:25,059 - 48092a eselect profile set default/linux/amd64/23.0
2025-07-30 13:53:25,511 - 48092a (stderr): !!! Error: get_repo_path failed
2025-07-30 13:53:25,565 - running in container 48092a2750f402df72d8d38c2b7bdb284b52d0e13d7f5bf5918327294ed0f411
2025-07-30 13:53:25,565 - failed command "eselect profile set default/linux/amd64/23.0"
2025-07-30 13:53:25,566 - no global USE flags given, skipping
2025-07-30 13:53:25,566 - tweaking portage settings
2025-07-30 13:53:25,566 - 48092a echo FEATURES=\"-sandbox -usersandbox userfetch\" >> /etc/portage/make.conf
2025-07-30 13:53:25,676 - 48092a echo MAKEOPTS=\"-j16\" >> /etc/portage/make.conf
2025-07-30 13:53:25,851 - 48092a echo "C.UTF-8 UTF-8" > /etc/locale.gen
2025-07-30 13:53:26,026 - enabling test feature for [Atom("app-editors/nano")]
2025-07-30 13:53:26,026 - 48092a mkdir -p /etc/portage/env
2025-07-30 13:53:26,190 - 48092a echo "app-editors/nano tester.conf" >> /etc/portage/package.env
2025-07-30 13:53:26,354 - 48092a echo "FEATURES=\"test splitdebug\"" > /etc/portage/env/tester.conf
2025-07-30 13:53:26,539 - unmasking [Atom("app-editors/nano")]
2025-07-30 13:53:26,539 - 48092a mkdir -p /etc/portage/package.accept_keywords
2025-07-30 13:53:26,675 - 48092a echo "app-editors/nano" ~amd64 >> /etc/portage/package.accept_keywords/testbuild
2025-07-30 13:53:26,780 - unmasking additional atoms
2025-07-30 13:53:26,780 - skipping update
2025-07-30 13:53:26,780 - skipping basic packages
2025-07-30 13:53:26,780 - setting gcc
2025-07-30 13:53:26,780 - summary
2025-07-30 13:53:26,780 - 48092a if [[ -d /etc/portage/package.accept_keywords ]]; then cat /etc/portage/package.accept_keywords/*; fi
2025-07-30 13:53:26,855 - 48092a (stdout): app-editors/nano ~amd64
2025-07-30 13:53:26,932 - 48092a if [[ -f /etc/portage/package.use/testbuild ]]; then cat /etc/portage/package.use/testbuild; fi
2025-07-30 13:53:27,058 - 48092a emerge --info
2025-07-30 13:53:27,411 - 48092a (stderr): Permission denied: '/var/db/repos/gentoo/eclass'
2025-07-30 13:53:27,493 - running in container 48092a2750f402df72d8d38c2b7bdb284b52d0e13d7f5bf5918327294ed0f411
2025-07-30 13:53:27,493 - failed command "emerge --info"
2025-07-30 13:53:27,493 - 48092a qlop
2025-07-30 13:53:27,558 - 48092a (stderr): qlop: Could not open logfile '/var/log/emerge.log': No such file or directory
2025-07-30 13:53:27,600 - created container 48092a2750f402df72d8d38c2b7bdb284b52d0e13d7f5bf5918327294ed0f411
2025-07-30 13:53:27,600 - 48092a echo emerge --verbose --autounmask-write=y --autounmask-license=y --autounmask-continue=y app-editors/nano --ask >> ~/.bash_history
2025-07-30 13:53:27,738 - emerge attempt 1 (of 5)
2025-07-30 13:53:27,738 - 48092a ['emerge', '--verbose ', '--autounmask-write=y ', '--autounmask-license=y ', '--autounmask-continue=y ', 'app-editors/nano']
2025-07-30 13:53:28,109 - 48092a (stderr): Permission denied: '/var/db/repos/gentoo/eclass'
2025-07-30 13:53:28,190 - running in container 48092a2750f402df72d8d38c2b7bdb284b52d0e13d7f5bf5918327294ed0f411
2025-07-30 13:53:28,191 - failed command "['emerge', '--verbose ', '--autounmask-write=y ', '--autounmask-license=y ', '--autounmask-continue=y ', 'app-editors/nano']"
2025-07-30 13:53:28,191 - stopping container
48092a2750f402df72d8d38c2b7bdb284b52d0e13d7f5bf5918327294ed0f411
2025-07-30 13:53:28,277 - deleting container
48092a2750f402df72d8d38c2b7bdb284b52d0e13d7f5bf5918327294ed0f411Inside the container: 0cd52e4e8810 ~ # ls -al /var/db/repos/gentoo/eclass
ls: cannot access '/var/db/repos/gentoo/eclass': Permission denied
0cd52e4e8810 ~ # ls -al /var/db/repos/gentoo
ls: cannot open directory '/var/db/repos/gentoo': Permission denied
0cd52e4e8810 ~ # ls -al /var/db/repos
total 12
drwxr-xr-x 1 portage portage 4096 Jul 30 10:56 .
drwxr-xr-x 1 root root 4096 Jul 27 17:31 ..
drwxr-x--- 182 nobody nobody 4096 Jul 29 20:27 gentoo
0cd52e4e8810 ~ # ls -aln /var/db/repos
total 12
drwxr-xr-x 1 250 250 4096 Jul 30 12:00 .
drwxr-xr-x 1 0 0 4096 Jul 27 17:31 ..
drwxr-x--- 182 65534 65534 4096 Jul 29 20:27 gentooSeems like it's expected: the host |
|
I tried the same command as you, changing the repository path and it worked just fine. I have the permissions of my user. |
|
@stkw0 , the same. $ ls -aln gentoo/app-alternatives/yacc
total 16
drwxr-x--- 2 1000 1000 4096 Jul 30 17:42 .
drwxr-x--- 13 1000 1000 4096 Jul 30 17:42 ..
-rw-r----- 1 1000 1000 610 Jul 30 17:42 metadata.xml
-rw-r----- 1 1000 1000 1137 Jul 30 17:42 yacc-1-r2.ebuild
$ id
uid=1000(sasha) gid=1000(sasha) groups=1000(sasha),10(wheel),27(video),100(users)
$ umask
0027e0e843fe254a ~ # ls -aln /var/db/repos/gentoo/app-alternatives/yacc
total 16
drwxr-x--- 2 0 0 4096 Jul 30 14:42 .
drwxr-x--- 13 0 0 4096 Jul 30 14:42 ..
-rw-r----- 1 0 0 610 Jul 30 14:42 metadata.xml
-rw-r----- 1 0 0 1137 Jul 30 14:42 yacc-1-r2.ebuildYours? |
|
Also, did you run |
|
@stkw0 , no. But: # cat /etc/subuid
sasha:100000:65536
# cat /etc/subgid
sasha:100000:65536 |
This is the issue. And there is a fundamental problem I don't know yet how to solve. Basically, we could map the same UID and GID so the container has access to the same resources than the calling user. The problem then, is that emerge needs root to run and the options to map the same UID/GID are not present in docker, so even if we did it, then it would diverge between podman and docker, it would stop being a drop-in replacement. If the same UID/GID are not used, then the container uses a different ones so they can not access to files that don't have +r |
|
@stkw0 , What's your permission on an |
|
|
|
|
|
Ok, thanks. |
|
IMHO volumes/paths should be documented. |
|
The ro on --batch should be transparent to the user. It's more an implementation detail. But I would agree it should go on the Changelog of the next release |
The docker container creates the distfiles as "portage" user (UID 250) in the host. This produces an error if then a container is launched as podman. With this ugly hack we can get docker to fetch distfiles as the calling user which avoids an error if then podman is used.
|
@kuraga I just pushed an horrible hack which would help to make it compatible with podman. Now, if you create a bash script like this and use it instead of podman, it should work even if you have an umask of 0027. |
|
@stkw0 , I'm sure it works. Thanks very much for the investigation. But I strictly vote to just add a requirement on permissions. Which would be up to the user. But I don't know, which. Come back later. |
|
Had an idea to add BTW, on the host: $ ls -aln /var/db/repos/gentoo/app-alternatives/yacc
total 20
drwxr-xr-x 2 0 0 4096 июл 8 23:07 .
drwxr-xr-x 13 0 0 4096 июл 19 16:07 ..
-rw-r--r-- 1 0 0 597 мар 20 15:38 Manifest
-rw-r--r-- 1 0 0 610 мар 20 15:38 metadata.xml
-rw-r--r-- 1 0 0 1137 мар 20 15:38 yacc-1-r2.ebuildSo, the owner is |
Another BTW: Which user does |
Yeah, if you don't use --userns=keep-id it would create a different UIDs which would mismatch |
did you create a podman wrapper script as I described above? |
It was instead about:
(which groups does this user have?) |
No, all my thoughts were without the new patch. It works, I'm sure. But I wanna something cleaner :) |
|
Instead of the wrapper, reading the manual I found you can use an env var. for userns This way you could configure it once and forget about it. While writing this I realized we could just export that env var before calling "docker" (or podman). As docker don't have it implemented, it will just be ignored, making it compatible with both :D |
@stkw0 , can it be some "non-superuser" In the container, I need On the host, under Why? Thanks. |
|
No, a rootless podman container can not have an EUID=0 in the container, that would defear the purpose of a rootless container. It can have a root inside the container, but it's like a new user. I never used podman before, so I don't yet precisely how podman works under the hood |
|
So yes, that's the case... @stkw0 , does your patch require to start under |
|
No, I tested it with normal rootless podman containers. The patch is just to trick docker into creating distfiles as the calling user instead of root, to make it compatible with podman. It doesn't helps with sourcing ebuilds from portdir. For that, the wrapper is needed. |
|
Well, does wrapper need |
|
No |
@stkw0 ,
|
|
Assuming a rootless podman with --userns=keep-id:uid=0
Which is expected |
|
|
||
| # Avoid wasting time generating the whole set | ||
| self.execute('echo "C.UTF-8 UTF-8" > /etc/locale.gen') | ||
| self.execute(f'sed -i "s/250/{os.getuid()}/g" /etc/passwd') |
There was a problem hiding this comment.
@nicolasbock can you check if this hack is acceptable? Thank you!
There was a problem hiding this comment.
@stkw0 Yes, that works. Nice hack 😄
@stkw0 , I was asking about Docker. And without the hack. But okay, I give up. The hack is required. |
in docker is more simple. It's just the same than in the host. A file owned by the user, in Docker is the UID of the user, a file owned by root is root and owned by portage would appear as owned by portage. |
|
@stkw0 , @nicolasbock , I see four ways. The first way. The hack. The second way. 2-A. We 2-B. Or the user does it by itself before start. The third way. The user have to
The fourth way. Different ways (from above) for Docker and Podman. I'd like the |
@stkw0 , if so, the only sufficient item on Docker (without the hack!) is:
|
|
Docker doesn't need to chown or anything. It just works. It's rootless podman and the ability to reuse distfiles/pkgfiles created by Docker in a rootless podman what is troublesome. |
|
Ha, yes. Even so. @stkw0 , why not just I thought we need rootFUL for this. |
|
@stkw0 wrote in IRC chat:
|
|
@stkw0 , I guess you mean "to work after Well, yes. But we already pollute the repo directory. Ok, on your choice. |
How so? |
In non- |
|
We don't. A user has the freedom to modify it, but nothing is modified by
itself
El ds., 2 d’ag. 2025, 12:52, Alexander Kurakin ***@***.***>
va escriure:
… *kuraga* left a comment (nicolasbock/ebuildtester#245)
<#245 (comment)>
Well, yes. But we already pollute the repo directory.
How so?
In non---batch mode we mount in rw and modify the directory.
—
Reply to this email directly, view it on GitHub
<#245 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABFD7GTLBEWJSYMSXUPAYML3LSKAFAVCNFSM6AAAAACCUIES3KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCNBWGQZTCMRYHE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
BTW, I realize now that even for Podman #245 (comment) has been updated. |
|
Any updates? |
I wanted to improve it adding what I commented about the env var to have a cleaner solution, but I didn't had time and will probably not have time until the end of the month |
No description provided.