[RFC] Record installed slices in --root and install only what's missing

[woky]

Currently it's not efficient to use chisel for an image that's based on another image that was also built using chisel. This is related to the current lack of any metadata/database of installed slices in the final images.

There's a desire to have 2 kind of images for some programming language environments (for example .NET or Go):

• runtime-deps image: This image contains required dependencies to run self-contained applications compiled from an language SDK. Examples are self-contained application bundles in .NET or static binaries in Golang. These final products have no dependencies on the language runtime as they bundle it. But they still require libraries on which the language runtime depends.
• runtime image: This image is super-set of runtime-deps containing language runtime in addition to its dependencies.

The way we currently use chisel is something a little bit more complex than this illustrative example (that also pretends chisel is in PATH):

FROM ubuntu:22.04 as builder
RUN chisel cut --release ubuntu-22.04 --root /rootfs slice1_libs slice2_libs

FROM scratch
COPY --from=builder /rootfs /

If we apply this for the 2 images described above, we would end up with something like the following 2 Dokcerfiles.

For runtime-deps image:

FROM ubuntu:22.04 as builder
RUN chisel cut --release ubuntu-22.04 --root /rootfs libc6_libs libgcc-s1_libs libstdc++6_libs

FROM scratch
COPY --from=builder /rootfs /

For runtime image (here .NET is used as an example):

FROM ubuntu:22.04 as builder
RUN chisel cut --release ubuntu-22.04 --root /rootfs-dotnet dotnet-runtime-6.0_libs

FROM runtime-deps
COPY --from=builder /rootfs-dotnet /

Even though we base our final runtime image on runtime-deps, we effectively overwrite everything that was installed in runtime-deps. Chisel cannot know that dependencies of dotnet-runtime-6.0 have already been installed so whole dependency tree is populated in /rootfs-dotnet.

In this simple example one could suggest to not base runtime image on runtime-deps image but instead build it from scratch. But more complex runtime-deps image could have something more in addition to root filesystem created by chisel, e.g.:

FROM ubuntu:22.04 as builder
RUN chisel cut --release ubuntu-22.04 --root /rootfs libc6_libs libgcc-s1_libs libstdc++6_libs \
  && echo app:x:999:999:app:/nonexistent:/nonexistent >/etc/passwd \
  && echo app:x:999: >/etc/group

FROM scratch
COPY --from=builder /rootfs /
ENV DOTNET_VERSION=6.0

Here it makes sense to base runtime image on runtime-deps image (above) because of the additional files created outside of chisel and ENV directives. (There could be also HEALTHCHECK, PORTS and other directives that we would like to inherit in runtime image.)

It'd be nice if chisel had an ability to record what has already been installed, and install only missing dependencies that were not recorded. It's a bit complicated by the fact that we execute chisel in stage container that doesn't have access to the image on top of which we want to copy new slices. But that can be solved by copying the suggested chisel metadata/database into the stage container, e.g. like this:

FROM ubuntu:22.04 as builder
COPY --from=dotnet-runtime-deps /var/lib/chisel.db /rootfs-dotnet/var/lib/chisel.db
RUN chisel cut --release ubuntu-22.04 --root /rootfs-dotnet dotnet-runtime-6.0_libs

FROM runtime-deps
COPY --from=builder /rootfs-dotnet /

Now chisel could read state of /rootfs-dotnet from /rootfs-dotnet/var/lib/chisel.db (or wherever we decide to store the metadata).

One major problem with this is that it'll probably break the assumption that scripts run in a closed world and will lead to non-determinism. Currently each chisel run creates new world from scratch and so scripts runs can be scheduled to produce deterministic output. With such split installation it would no longer hold.

[cjdcordeiro]

Additional piece of information wrt security monitoring:

USN notification
service for ROCKs is only being able to monitor ROCKs based on Ubuntu
(i.e. the rock has a dpkg.query file listing Ubuntu pkgs and versions).

[mthalman]

I have a post which describes a workaround for this issue.

[loewenstein]

#25 got closed with

I think we can close this PR, since the underlying feature will be split into multiple feature requests

Can't seem to find those smaller feature requests linked here though.

[rebornplusplus]

Update: we added Chisel manifest support in #142.

[lbussell]

Update: we added Chisel manifest support in #142.

@rebornplusplus is there an ETA for a new release with this feature?

[rebornplusplus]

@rebornplusplus is there an ETA for a new release with this feature?

I don't have the exact date, but I believe it should be within the next few weeks. We are currently fine-tuning a few stuffs (#150, #159, #162) in preparation to a v1.0.0 release.

[rebornplusplus]

@lbussell v1.0.0 is released!

[cjdcordeiro]

Indeed, v1.0.0 is out :)

Closing this issue as:

• the recording of installed content is now available via the Chisel manifest
• subsequent installations are also possible
  • although the strategy for optimizing subsequent installations might be improved in the future, the current behaviour should be sufficient in a way that respects the chiselled system's integrity (i.e. Chisel will bail out if trying to touch existing paths that don't match the planned installation)

[norrisjeremy]

@rebornplusplus @cjdcordeiro Where would a user go to learn how to use this new "Chisel manifest" feature?
I'm not finding any seeing any obvious documentation and when I execute chisel cut ... using the new 1.0.0 release, I'm not seeing any sort of manifest being generated?

$ ./chisel cut --release ubuntu-24.04 --root foo libc6_libs
2024/12/04 08:53:47 Consulting release repository...
2024/12/04 08:53:47 Cached ubuntu-24.04 release is still up-to-date.
2024/12/04 08:53:47 Processing ubuntu-24.04 release...
2024/12/04 08:53:59 Selecting slices...
2024/12/04 08:53:59 Fetching ubuntu 24.04 noble suite details...
2024/12/04 08:54:00 Release date: Thu, 25 Apr 2024 15:10:33 UTC
2024/12/04 08:54:00 Fetching index for ubuntu 24.04 noble main component...
2024/12/04 08:54:00 Fetching index for ubuntu 24.04 noble universe component...
2024/12/04 08:54:00 Fetching ubuntu 24.04 noble-security suite details...
2024/12/04 08:54:00 Release date: Wed, 04 Dec 2024 12:45:27 UTC
2024/12/04 08:54:00 Fetching index for ubuntu 24.04 noble-security main component...
2024/12/04 08:54:00 Fetching index for ubuntu 24.04 noble-security universe component...
2024/12/04 08:54:00 Fetching ubuntu 24.04 noble-updates suite details...
2024/12/04 08:54:00 Release date: Wed, 04 Dec 2024 12:45:57 UTC
2024/12/04 08:54:00 Fetching index for ubuntu 24.04 noble-updates main component...
2024/12/04 08:54:00 Fetching index for ubuntu 24.04 noble-updates universe component...
2024/12/04 08:54:00 Fetching pool/main/b/base-files/base-files_13ubuntu10.1_amd64.deb...
2024/12/04 08:54:00 Fetching pool/main/g/glibc/libc6_2.39-0ubuntu8.3_amd64.deb...
2024/12/04 08:54:00 Extracting files from package "base-files"...
2024/12/04 08:54:00 Extracting files from package "libc6"...

$ find foo
foo
foo/lib
foo/usr
foo/usr/lib
foo/usr/lib/x86_64-linux-gnu
foo/usr/lib/x86_64-linux-gnu/libthread_db.so.1
foo/usr/lib/x86_64-linux-gnu/libanl.so.1
foo/usr/lib/x86_64-linux-gnu/libpthread.so.0
foo/usr/lib/x86_64-linux-gnu/libdl.so.2
foo/usr/lib/x86_64-linux-gnu/libnss_files.so.2
foo/usr/lib/x86_64-linux-gnu/libutil.so.1
foo/usr/lib/x86_64-linux-gnu/libpcprofile.so
foo/usr/lib/x86_64-linux-gnu/libc_malloc_debug.so.0
foo/usr/lib/x86_64-linux-gnu/libnsl.so.1
foo/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
foo/usr/lib/x86_64-linux-gnu/libmemusage.so
foo/usr/lib/x86_64-linux-gnu/libnss_dns.so.2
foo/usr/lib/x86_64-linux-gnu/librt.so.1
foo/usr/lib/x86_64-linux-gnu/libnss_hesiod.so.2
foo/usr/lib/x86_64-linux-gnu/libmvec.so.1
foo/usr/lib/x86_64-linux-gnu/libBrokenLocale.so.1
foo/usr/lib/x86_64-linux-gnu/libc.so.6
foo/usr/lib/x86_64-linux-gnu/libresolv.so.2
foo/usr/lib/x86_64-linux-gnu/libnss_compat.so.2
foo/usr/lib/x86_64-linux-gnu/libm.so.6
foo/usr/share
foo/usr/share/doc
foo/usr/share/doc/libc6
foo/usr/share/doc/libc6/copyright
foo/usr/share/doc/base-files
foo/usr/share/doc/base-files/copyright
foo/usr/lib64
foo/usr/lib64/ld-linux-x86-64.so.2
foo/lib64

Am I misunderstanding how this feature should work?

[cjdcordeiro]

Hi @norrisjeremy

@rebornplusplus is working on bootstrapping dedicated documentation pages for Chisel. We're hoping to get the first draft ready within the next 2 weeks.

In the meantime, the manifest is an opt-in feature that you can solicit via the base-files_chisel slice. Just add that slice and you'll get the manifest created.

This post should help you: https://discourse.ubuntu.com/t/chisel-manifest-is-supported-in-newly-released-v1-0-0/48944

[norrisjeremy]

Hi @cjdcordeiro,

Excellent, thanks! Adding the base-files_chisel slice was the missing piece of the puzzle I was looking for.

Thanks,
Jeremy