Skip to content

Facilitate docker 'layering' #1652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
olivier-lacroix opened this issue Jul 23, 2024 · 8 comments · May be fixed by #3092
Open

Facilitate docker 'layering' #1652

olivier-lacroix opened this issue Jul 23, 2024 · 8 comments · May be fixed by #3092
Labels
enhancement Feature request

Comments

@olivier-lacroix
Copy link
Contributor

Problem description

To speed up docker image building, it is fairly typical to separate "things that change often", to "things that don't".

For dependency management, this usually means it is recommended to install dependencies in a docker layer first (as they do not change often), then install the local package(s) (as it changes at ~each build, so a lot more often).

A very simple python Dockerfile may look like

FROM python:3.12.2-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install -r requirements.txt

COPY sample.py .

where requirements are installed before copying local code, to maximise cache hit.

It is not clear how something similar could be achieved with pixi currently. A couple of options that come to mind

  1. Not include local code in an environment, and use this environment to install dependencies. In that case, local code would need to be installed by something else than pixi, which does not feel ideal
  2. Have two environments, one 'dependencies-only' and one 'dependencies + local code' and start by installing one, then copy the code, then install the second one. this may be better, but feels like a bit of a hack, having two environments when we actually only need one, and pixi will do extra work to solve environment.

Another alternative (and better?) option could be extra options for pixi install. For instance --no-path-dependencies and --only-path-dependencies to respectively exclude path dependencies from solve & install, or deal only with them (ideally without doing any solve work, bypassing as many steps as possible to speed things up).

The Dockerfile may then look like

FROM python:3.12.2-slim

WORKDIR /app

COPY pixi.lock pixi.toml .

RUN pixi install --locked --no-path-dependencies

COPY sample-package . 

RUN pixi install --locked --only-path-dependencies
@olivier-lacroix olivier-lacroix added the enhancement Feature request label Jul 23, 2024
@tdejager
Copy link
Contributor

Thanks for the issue, seems like something we should facilitate @pavelzw could you chime in here with your experience?

@pavelzw
Copy link
Contributor

pavelzw commented Jul 26, 2024

Hmm, I get the advantage of this, although i'm worrying a bit that this makes the CLI more complicated and cluttered. I can't think of any better solution than Olivier's so I guess this could make sense in pixi...

Side note: i've not really used path dependencies because of #1046 and #1340 (comment)

With the postinstall method that I am using until now, we can just ship around this problem:

FROM ghcr.io/prefix-dev/pixi

WORKDIR /app
COPY pixi.lock pixi.toml .
RUN pixi install --locked
COPY pyproject.toml my_app .
RUN pixi run postinstall

But since postinstall could be considered a hack as well, this might not be what we want.

@olivier-lacroix
Copy link
Contributor Author

olivier-lacroix commented Jul 29, 2024 •
edited
Loading

Thanks @pavelzw for your thoughts on this!

As additional background, looking at other tools:

  • conda install has --no-deps and --only-deps options
  • pip install has --no-deps options
  • poetry install has --no-root, --only-root as well as --no-directory options

conda and pip options are a bit different from what is proposed here, which can be explained by the fact they do not use lockfiles. Poetry's --no-directory option however feel very similar to what is proposed here.

@olivier-lacroix olivier-lacroix mentioned this issue Aug 2, 2024
Closed
2 tasks
@ruben-arts
Copy link
Contributor

With pixi build in the back of my mind I think it would make sense to have some of these options available, even without the pypi workflow described in this Issue but including that.

I believe pixi install could be more feature full

  • pixi install package_x package_y: install package_x & package_y from the pixi.lock and their dependencies.
  • pixi install package_x --force-reinstall: Install the package without cache and do whatever required to install as if installing a clean machine. (helpful to avoid issues like Cython source file is not compiled after pixi install/update #1695 and problems with the cache because of locally build conda packages.)
  • pixi install package_x --only-deps: Don't install package_x but only it's dependencies.

I don't see why --no-deps would make sense but please enlighten me.

@olivier-lacroix
Copy link
Contributor Author

@ruben-arts I agree on the lower usefulness of —no-deps. This tends to be done to bypass costly checks, and/or defer the installation of dependencies to a later step/stage.

@olivier-lacroix
Copy link
Contributor Author

I am not sure using the same command pixi install for both environment level operation (as it is today) and package level is the best option; I feel it may be confusing.

Maybe a separate command for package level operation would be better ? e.g. pixi force or something like that? After all, these operations at likely outside of the « happy path ».

@gohweixun
Copy link

Hello, just checking back in to see if there have been any updates on this front. I think that separating out the dependencies installation step from the actual app installation could greatly facilitate development for containerised workflows (and this is a fairly common use case).

@olivier-lacroix olivier-lacroix linked a pull request Feb 9, 2025 that will close this issue
Open
@olivier-lacroix
Copy link
Contributor Author

I have created a PR that intends to solve this issue. Let me know what you guys think.

@baszalmstra baszalmstra added the pixi label Mar 27, 2025 — with Linear
@baszalmstra baszalmstra removed the pixi label Apr 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Feature request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Add the ability to skip install of local source dependencies olivier-lacroix/pixi
6 participants
@tdejager @gohweixun @baszalmstra @ruben-arts @olivier-lacroix @pavelzw