Python provider not used for pants package of pex_binary target

[kaos]

Describe the bug
When using pants.backend.python.providers.experimental.pyenv test, lint and check works fine, but package fails.

Pants version
2.21.0

OS
Ubuntu

Additional info

13:03:09  pants lint check test package ::
13:03:15  13:03:15.48 [INFO] waiting for pantsd to start...
13:03:18  13:03:17.78 [INFO] pantsd started
13:03:18  13:03:18.30 [INFO] Initializing scheduler...
13:03:24  13:03:24.02 [INFO] Scheduler initialized.
13:03:24  13:03:24.51 [INFO] Starting: Choose specific version for Python 3.9
13:03:24  13:03:24.68 [INFO] Completed: Choose specific version for Python 3.9
13:03:24  13:03:24.92 [INFO] Starting: Building isort.pex from resource://pants.backend.python.lint.isort/isort.lock
13:03:24  13:03:24.92 [INFO] Starting: Building black.pex from resource://pants.backend.python.lint.black/black.lock
13:03:25  13:03:24.97 [INFO] Starting: Building autoflake.pex from resource://pants.backend.python.lint.autoflake/autoflake.lock
13:03:25  13:03:24.97 [INFO] Starting: Building docformatter.pex from resource://pants.backend.python.lint.docformatter/docformatter.lock
13:03:25  13:03:24.97 [INFO] Starting: Building flake8.pex from resource://pants.backend.python.lint.flake8/flake8.lock
13:03:25  13:03:25.01 [INFO] Canceled: Building black.pex from resource://pants.backend.python.lint.black/black.lock
13:03:25  13:03:25.01 [INFO] Starting: Building black.pex from resource://pants.backend.python.lint.black/black.lock
13:03:25  13:03:25.01 [INFO] Canceled: Building black.pex from resource://pants.backend.python.lint.black/black.lock
13:03:25  13:03:25.01 [INFO] Starting: Building black.pex from resource://pants.backend.python.lint.black/black.lock
13:03:27  13:03:26.58 [INFO] Completed: Building isort.pex from resource://pants.backend.python.lint.isort/isort.lock
13:03:27  13:03:26.62 [INFO] Completed: Building flake8.pex from resource://pants.backend.python.lint.flake8/flake8.lock
13:03:27  13:03:26.63 [INFO] Completed: Building autoflake.pex from resource://pants.backend.python.lint.autoflake/autoflake.lock
13:03:27  13:03:26.67 [INFO] Completed: Building docformatter.pex from resource://pants.backend.python.lint.docformatter/docformatter.lock
13:03:27  13:03:26.85 [INFO] Completed: Building black.pex from resource://pants.backend.python.lint.black/black.lock
13:03:27  13:03:26.85 [INFO] Completed: Format with docformatter - docformatter made no changes.
13:03:27  13:03:26.86 [INFO] Completed: Format with isort - isort made no changes.
13:03:27  13:03:26.93 [INFO] Completed: Fix with Autoflake - autoflake made no changes.
13:03:27  13:03:26.97 [INFO] Completed: Lint with Flake8 - flake8 succeeded.
13:03:27  Partition: ['CPython==3.9.*']
13:03:27  
13:03:27  
13:03:27  13:03:27.26 [INFO] Completed: Format with Black - black made no changes.
13:03:27  
13:03:27  ✓ autoflake succeeded.
13:03:27  ✓ black succeeded.
13:03:27  ✓ docformatter succeeded.
13:03:27  ✓ flake8 succeeded.
13:03:27  ✓ isort succeeded.
13:03:27  13:03:27.51 [INFO] Starting: Building mypy.pex from resource://pants.backend.python.typecheck.mypy/mypy.lock
13:03:27  13:03:27.57 [INFO] Starting: Building 4 requirements for requirements.pex from the 3rdparty/default.lock resolve: click, pantsbuild.pants.testutil==2.21.0, pantsbuild.pants==2.21.0, pytest
13:03:29  13:03:29.32 [INFO] Completed: Building mypy.pex from resource://pants.backend.python.typecheck.mypy/mypy.lock
13:03:31  13:03:31.13 [INFO] Completed: Building 4 requirements for requirements.pex from the 3rdparty/default.lock resolve: click, pantsbuild.pants.testutil==2.21.0, pantsbuild.pants==2.21.0, pytest
13:03:31  13:03:31.18 [INFO] Starting: Building requirements_venv.pex
13:03:32  13:03:32.72 [INFO] Completed: Building requirements_venv.pex
13:03:34  13:03:34.63 [INFO] Completed: Typecheck using MyPy - mypy - mypy succeeded.
13:03:34  Success: no issues found in 9 source files
13:03:34  
13:03:34  
13:03:34  
13:03:34  ✓ mypy succeeded.
13:03:34  13:03:34.74 [INFO] Starting: Building pytest.pex from resource://pants.backend.python.subsystems/pytest.lock
13:03:34  13:03:34.77 [INFO] Starting: Building 3 requirements for requirements.pex from the 3rdparty/default.lock resolve: pantsbuild.pants.testutil==2.21.0, pantsbuild.pants==2.21.0, pytest
13:03:34  13:03:34.77 [INFO] Starting: Building requirements.pex
13:03:34  13:03:34.86 [INFO] Canceled: Building pytest.pex from resource://pants.backend.python.subsystems/pytest.lock
13:03:34  13:03:34.86 [INFO] Starting: Building pytest.pex from resource://pants.backend.python.subsystems/pytest.lock
13:03:36  13:03:36.38 [INFO] Completed: Building requirements.pex
13:03:36  13:03:36.52 [INFO] Completed: Building pytest.pex from resource://pants.backend.python.subsystems/pytest.lock
13:03:36  13:03:36.55 [INFO] Starting: Building local_dists.pex
13:03:37  13:03:37.55 [INFO] Completed: Building 3 requirements for requirements.pex from the 3rdparty/default.lock resolve: pantsbuild.pants.testutil==2.21.0, pantsbuild.pants==2.21.0, pytest
13:03:38  13:03:38.05 [INFO] Completed: Building local_dists.pex
13:03:38  13:03:38.10 [INFO] Starting: Building pytest_runner.pex
13:03:38  13:03:38.10 [INFO] Starting: Building pytest_runner.pex
13:03:39  13:03:39.64 [INFO] Completed: Building pytest_runner.pex
13:03:39  13:03:39.73 [INFO] Completed: Building pytest_runner.pex
13:03:40  13:03:40.09 [INFO] Completed: Run Pytest - src/.../x:tests - succeeded.
13:03:42  13:03:41.70 [INFO] Completed: Run Pytest - src/.../y:tests - succeeded.
13:03:42  
13:03:42  ✓ src/.../x:tests succeeded in 0.43s.
13:03:42  ✓ src/.../y:tests succeeded in 1.91s.

# --vvvvv--  Package begins here  --vvvvv--

13:03:42  13:03:41.82 [INFO] Starting: Building 1 requirement for x.pex from the 3rdparty/default.lock resolve: click
13:03:42  13:03:41.86 [INFO] Starting: Building build_backend.pex from resource://pants.backend.python.subsystems/setuptools.lock
13:03:43  13:03:43.30 [INFO] Completed: Building 1 requirement for x.pex from the 3rdparty/default.lock resolve: click
13:03:43  13:03:43.31 [ERROR] 1 Exception encountered:
13:03:43  
13:03:43  Engine traceback:
13:03:43    in `package` goal
13:03:43  
13:03:43  ProcessExecutionFailure: Process 'Building 1 requirement for x.pex from the 3rdparty/default.lock resolve: click' failed with exit code 102.
13:03:43  stdout:
13:03:43  
13:03:43  stderr:
13:03:43  Could not find a compatible interpreter.
13:03:43  
13:03:43  Examined the following interpreters:
13:03:43  1.) /usr/bin/python2.7 CPython==2.7.18
13:03:43  2.) /usr/bin/python3.8 CPython==3.8.10
13:03:43  
13:03:43  No interpreter compatible with the requested constraints was found:
13:03:43  
13:03:43    Version matches CPython==3.9.*
13:03:43  
13:03:43  
13:03:43  
13:03:43  Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.

[kaos]

This fails for building a pex_binary. Building python_distribution targets works fine. I can understand if running the pex once built could be problematic, but building I think should work?

[huonw]

This reproduces with pants.backend.python.providers.experimental.python_build_standalone too.

Reproducer:

cd $(mktemp -d)

# NB. adjust the 3.10.8 to be an interpreter that's not available on your system
cat > pants.toml <<EOF
[GLOBAL]
pants_version = "2.24.0a0"

backend_packages = [
  "pants.backend.python",
  "pants.backend.python.providers.experimental.python_build_standalone",
]

[python]
interpreter_constraints = ["CPython==3.10.8"]
EOF

cat > BUILD <<EOF
python_sources(name="s")
pex_binary(name="p", entry_point="./example.py")
EOF

echo "print('hi')" > example.py

# Okay: runs and prints `hi` just fine
pants run ./example.py

# BUG: cannot find interpreter
pants package :p

[kaos]

:p, what a great example target name 🤣

[huonw]

In #21842 we have another reproducer with more exploration, which leads to two new observations:

• Setting [python-bootstrap].search_path to only look at <PYENV> makes packaging work
• Including things like <PEXRC> which is valid input to the config causes a list index out of range error

[tgolsson]

I ran across this today when rebuilding our CI images, and I seemingly cannot make it work with the above workaround. Instead, it fails with the following error:

Could not find a compatible interpreter.

1.) /root/.cache/nce/67912efc04f9156d8f5b48a0348983defb964de043b8c13ddc6cc8a002f8e691/cpython-3.9.18+20240107-x86_64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.9 CPython==3.9.18
2.)                                                                                                                                                         /usr/bin/python3.12 CPython==3.12.3

If I don't add the boostrap entry, only the second exists. Interestingly, pyenv is at least evaluated at some point before getting here; as I only saw this once I had added all the native libs needed to build CPython. I'm a bit confused why it also still lists the entry for /usr/bin/python3.12, but I haven't explored much further yet.

[tgolsson]

I think after playing around with a few reproducing cases that we're simply not passing the pyenv Python paths to the Pex process at all. In my logs I can see the correct Python being selected to run the pex, but that path is nowhere in the build invocation. So when Pex tries to locate the compatible interpreter it barfs, because all it has is $PATH.

[tgolsson]

I'm a bit too unfamiliar with the Python backend but I think this PR does workaround the issue, following the reasoning above. #21948. But I don't think it's correct generally and probably runs afoul of other assumptions in the (quite convoluted) Pex/Python build flow.