How to activate/deactivate venvs when running tasks on nested Poetry projects?

[bkeryan]

I have a monorepo that currently has a top-level Poetry package and several nested auxiliary packages. Each auxiliary package has different dependencies, its own poetry.lock file (for now), and its own in-project virtualenv (for now). Note that different packages' poetry.lock files may reference different versions of the same dependency.

The directory layout is something like this:

/
    pyproject.toml
    packages/
        a/
            pyproject.toml
        b/
            pyproject.toml

I would like to use Poe to make it easier to run commands (such as lint tools, mypy, etc.) on all of the packages at once. For most commands, this works, but I'm having problems when running mypy. Since each package has different dependencies, it's important to activate the correct virtualenv so that mypy can import the installed dependencies.

Here are a couple of my Poe tasks:

[tool.poe.tasks.install]
shell = "for dir in . packages/*/; do pushd $dir; poetry install; popd; done"

[tool.poe.tasks.check-types]
sequence = [
  { cmd = "mypy -p top_level_package" },
  { shell = "for dir in packages/*/; do pushd $dir; mypy .; popd; done" }
]

The problem is that instead of creating separate venvs for the top-level package, packages/a, and packages/b, this installs all of their dependencies into the top-level package's venv. Not all of the packages' poetry.lock files agree about which dependency versions to install, so this upgrades and downgrades dependencies in the top-level package's venv.

This is related to the Poetry behavior described in Managing environments:

What this means is that it will always work isolated from your global Python installation. To achieve this, it will first check if it’s currently running inside a virtual environment. If it is, it will use it directly without creating a new one. But if it’s not, it will use one that it has already created or create a brand new one for you.

When I run poetry run poe install, Poetry activates the top-level package's venv before running Poe, and when Poe runs poetry install for each package, Poetry reuses the top-level package's venv instead of activating a new one.

One potential solution is to set up the Poe tasks to deactivate the current virtualenv before running other commands.

• Prepending deactivate; to the front of the task's shell command prints a "command not found error" (possibly because I'm using Windows and Poe's shell tasks are using Git Bash?).
• Prepending unset VIRTUAL_ENV; seems to work. This makes Poetry print "Creating virtualenv a in ...\packages\a.venv" and create a venv.
  • Caveat 1: This approach requires using shell tasks instead of cmd tasks.
  • Caveat 2: This doesn't undo all of the other environment variable changes (PATH, PYTHONHOME, PROMPT, ...). However, running poetry run set doesn't list any _OLD_VIRTUAL_* environment variables, so perhaps Poetry doesn't set these? (Note: I'm still using Poetry 1.2.2 and haven't checked whether the latest version changes this behavior.)

Another potential solution is to set up the Poe tasks to directly activate the in-project virtualenv. However, the location of the activate script is different for Linux vs. Windows, so this is more complicated than adding unset VIRTUAL_ENV.

Installing all of the monorepo's packages into a single venv could probably work, if done in a way that resolves dependency versions for all packages at once, rather than upgrading/downgrading dependencies when visiting each package. If there was one poetry.lock per workspace, then having one virtualenv per workspace would work a lot better. I'm aware of a couple of workspace/subproject plugins for Poetry, but I haven't tried them out yet.

Any other ideas?

[nat-n]

hi @bkeryan, that's an interesting problem. I'm not immediately sure about the solution but I can offer a few hints.

1. One thing to try is having an init task in each sub-project like:

    [tool.poe.tasks]
    init = "poetry install"

   and then invoke that poe task from the root project like

   [tool.poe.tasks]
   setup = [
     {cmd = "poe --root packages/a init"},
     {cmd = "poe --root packages/a init"}
   ]

   I think in this configuration poe should take care of replacing the venv of the root with that of the sub-project so that poetry then runs with the right environment.

2. If you need to control environment variables on the running task (to stop poetry picking them up), then configuring the env option on the task might come in handy, e.g.

   [tool.poe.tasks._setup_subproject]
   cmd = "poetry install"
   cwd = "packages/${SUB_PROJECT}"
   env = { VIRTUAL_ENV = "" , ...}
   args = [{ name = "SUB_PROJECT", positional = true }]
   
   [tool.poe.tasks.setup]
   sequence = ["_setup_subproject a", "_setup_subproject b"]

Another thing to be aware of about shell tasks is that you can configure them to use something other than bash, e.g. pwsh, although this may complicate sharing tasks across teams where different users run different platforms.

I hope this helps, let me know how you get on?

---

Another approach I just thought of is that if you are using the poe cli outside of your root poetry venv (having installed poethepoet via pipx for example) then you could use an "undocumented" task option to configure the task to use the "simple" executor instead of the usual poetry executor, so poe won't explicitly use the poetry venv to run the task. You just need to set the executor = "simple" option on the task, and make sure you're not using poetry run or inside poetry shell or any other venv when running poe. The downside of this of course is that it matters how you run the task, which is usually something to avoid.