explicitly resolve Python for pex run/package

[tgolsson]

This effectively ensures that when we try to build a pex, it's done using the executable that we're resolving either way. I'm sure this specific implementation is suboptimal, but it resolves the specific issues seens in #21048 and makes my repro pass.

As I noted on the issue, what is currently happening is that if we pick a pyenv interpreter to run the pex, it won't be available in the Pex (i.e., the tool) invocation sandbox. This causes Pex to barf when finding a compatible interpreter for its work.

[huonw]

Thanks for taking this on.

There's CI errors that'll seem to be indicating real problems:

E       ValueError: Only one of complete_platforms or a specific interpreter may be set. Got both CompletePlatforms(['src/py/project/platform-linux-py38.json', 'src/py/project/platform-mac-py38.json']) and PythonExecutable(path='/opt/hostedtoolcache/Python/3.8.18/x64/bin/python3.8', fingerprint='bd4b1f72112b6ae3a218e3d093f562f7a8ce7ecc07c13973df10a3ac5375adcf', append_only_caches=FrozenDict({})).

---

(Apologies if I end up telling you thinks you already know. This investigation is also firming up my understanding of pex.)

As background, (AIUI) I think there's two broad categories for pex builds:

• "manual" builds for a specific Python/machine (including cross-builds, for a completely different platform)
• "automatic" builds for some Python on the current machine

There's seem to be a few ways to specify specific platforms to build a PEX for:

• --complete-platforms: fully specify an external system so that PEX can choose the right wheels even without running on that system. The current system doesn't even need the relevant Python version installed (e.g. I think you can use a x86-64 Linux system with only Python 2.7 to build a PEX that (only) runs on Python 3.12 on arm64 macOS, with the right complete-platform JSON). (Exposed by Pants as complete_platforms=[...])
• --platform: similar to --complete-platform, but abbreviated, and doesn't fully capture the system info and can lead to PEXes that don't work as intended (deprecated and removed from Pants)
• --python: specific (local) Pythons to be compatible with

These can all be specified multiple times and are additive, e.g., on my arm64 macOS machine, this builds a PEX compatible with (i.e. contains wheels for those platforms):

• Python 3.10 on x86-64 linux
• 3.11 on arm64 macOS
• 3.12 on arm64 linux

pex \
  -v \
  --layout=packed \
  --complete-platform linux_cp310_x86_64.json \
  --python $(which python3.11) \
  --platform manylinux_2_17_aarch64-cp-3.12.9-cp312 \
  numpy -o numpy-pex
# ...
# pex:       Resolving for:
#   cp311-cp311-macosx_15_0_arm64 interpreter at /opt/homebrew/Cellar/[email protected]/3.11.9/Frameworks/Python.framework/Versions/3.11/bin/python3.11
#   abbreviated platform cp312-cp312-manylinux_2_17_aarch64
#   complete platform cp310-cp310-manylinux_2_31_x86_64: 1153.3ms
# ...


ls numpy-pex/.deps
# numpy-2.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
# numpy-2.2.2-cp311-cp311-macosx_14_0_arm64.whl
# numpy-2.2.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

By not providing any of these options, one can also let PEX do an search on the current platform for an appropriate interpreter, which seems to be guided by (among a few other options):

• --python-path: similar to $PATH, guiding where PEX looks
• --interpreter-constraint: limiting the versions of interpreters to use

For instance, I happen to have a few PBS interpreters on my system:

pex \
  -v \
  --layout=packed  \
  --python-path=$HOME/.cache/pants/named_caches/python_build_standalone/3.12.6/bin:$HOME/.cache/pants/named_caches/python_build_standalone/3.10.15/bin \
   --interpreter-constraint '==3.10.*' \
   numpy -o numpy-force310
# ...
# pex:       Resolving for:
#   cp310-cp310-macosx_15_0_arm64 interpreter at /Users/huon/.cache/pants/named_caches/python_build_standalone/3.10.15/bin/python3.10: 1322.9ms
# ...

ls numpy-force310/.deps
# numpy-2.2.2-cp310-cp310-macosx_14_0_arm64.whl

Specifying --python-path doesn't seem to interfere with the specific-platform options (i.e. this still builds for only the platform implied by the --complete-platform), e.g.

pex \
  -v \
  --layout=packed \
  --python-path=$HOME/.cache/pants/named_caches/python_build_standalone/3.12.6/bin:$HOME/.cache/pants/named_caches/python_build_standalone/3.10.15/bin \
  --complete-platform linux_cp310_x86_64.json \
   numpy -o numpy-path-and-platforms
# ...
# pex:       Resolving for:
#   complete platform cp310-cp310-manylinux_2_31_x86_64: 1167.8ms
# ...

ls numpy-path-and-platforms/.deps
# numpy-2.2.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

---

So, I think unconditionally specifying --python=... is undesirable, as it is forcing all PEXes to be compatible with that specific local interpreter, even when attempting to do a pure cross-build (e.g. building linux artifacts for a docker image, while on a macOS machine). The error we're seeing in CI is catching this.

The investigation above leads me to think of two possible solutions:

1. only specify --python when doing a current platform build (i.e. no complete platforms or other similar args)
2. include the PBS path(s) in --python-path for all builds

What do you think?