feat(python.toolchain): support file-based default Python version #2588

vonschultz · 2025-01-28T15:22:03Z

This change adds a new default_version_file attribute to python.toolchain. If set, the toolchain compares the file's contents to its python_version, and if they match, treats that toolchain as default (ignoring is_default). This allows Bazel to synchronize the default Python version with external tools (e.g., pyenv) that use a .python-version file or environment variables.

Fixes #2587.

This change adds a new `default_version_file` attribute to `python.toolchain`. If set, the toolchain compares the file's contents to its `python_version`, and if they match, treats that toolchain as default (ignoring `is_default`). This allows Bazel to synchronize the default Python version with external tools (e.g., pyenv) that use a `.python-version` file or environment variables. Fixes bazel-contrib#2587.

aignas

The implementation in general looks to me and I think this is probably a fine way to support such usecase. My only ask would be to see if there is a way to add a unit test in //tests/python/ .bzl file.

EDIT: sorry it took a while to dedicate attention to this and I really appreciate the write up and the explanation behind the usecase. Thank you once more for your contributions.

python/private/python.bzl

aignas

I actually wanted to write an example here and realized that the current API is not very clear to the user. What one would have to do is do something like below:

python.toolchain(
    version = "3.9",
    default_version_file = ".python-version"
)
python.toolchain(
    version = "3.10",
)

which is very similar and the user may be confused as to where and how to use the API. Is the user supposed to have:

python.toolchain(version = "3.10", default_version_file = ".python-version")
python.toolchain(version = "3.11", default_version_file = ".python-version")
python.toolchain(version = "3.12", default_version_file = ".python-version")

?

Should we have a separate tag_class that sets the defaults? I think I want to finish #2578 PR to understand if we can learn something from the builder API implemented there. The potential is to have a more DRY version:

python.defaults(version_file = ".python-version)
python.toolchain(version = "3.9")
python.toolchain(version = "3.10")
python.toolchain(version = "3.11")

vonschultz · 2025-03-11T09:17:31Z

Exactly, it would be

python.toolchain(version = "3.10", default_version_file = ".python-version")
python.toolchain(version = "3.11", default_version_file = ".python-version")
python.toolchain(version = "3.12", default_version_file = ".python-version")

yes. I don't think that's ideal, though. I like the idea of a separate tag_class for defaults. Then we could potentially support both something like

python.defaults(python_version_file = ".python-version")
python.toolchain(version = "3.9")
python.toolchain(version = "3.10")
python.toolchain(version = "3.11")

and something like

python.defaults(python_version = "3.11")
python.toolchain(version = "3.9")
python.toolchain(version = "3.10")
python.toolchain(version = "3.11")

and perhaps even implement a default of looking in the root module .python-version file if nothing else has been specified.

aignas · 2025-03-11T10:21:29Z

#2578 implements the design for defaults that got approved, so I think we could almost copy paste that here.

As an alternative to python.toolchain.is_default, introduce a python.defaults tag class with attributes python_version, python_version_env and python_version_file. This allows to read the default python version from your projects .python-version files, similar to other tools. It also allows using an environment variable, with a fallback if the environment variable is not set.

vonschultz · 2025-03-19T14:44:07Z

I originally had the idea that supporting reading the Python version from a file would be sufficient to also solve the use case where you want to read it from an environment variable, since you could generate a file with whatever logic you need. However, I read in the documentation of module_ctx.path that "remote repositories and module extensions are executed during the analysis phase and thus cannot depends on a target result (the label should point to a non-generated file)." For this reason I also included a python_version_env attribute in my latest patch, so both the file use case and the environment variable use case can be solved directly in rules_python.

aignas

Thanks a lot for the new design. I like how clean the interfaces look and I really like that rules_python reads .python-version by default, which makes it integrate with existing python tooling better.

I am torn if we should include this in 1.3.0 or if the ship has sailed. Since we are releasing right now and there may be a few more comments here and there, I would vote to just rebase and put this under unreleased CHANGELOG section.

python/private/python.bzl

aignas · 2025-03-20T04:56:16Z

python/private/python.bzl

+                    getenv = module_ctx.os.environ.get
+                default_python_version = getenv(default_python_version_env, default_python_version)
+    if not default_python_version:
+        fallback_python_version_file = module_ctx.path("@@//:.python-version")


How does this actually work? Will @@ not expand to rules_python in all cases? Is it possible to add a test for this? Ah, because we are not using Label("@@//:.python-version") we are going to use the root module's file. This may be good to document as an inline comment.

It didn't actually work. 🙈 It does work when I switch to using Label("@@//:.python-version"), which resolves to the root module's file. There is a potential issue if there is a .python-version file in the root module, but no BUILD or BUILD.bazel file in the root module. I don't know any way to detect it and ignore it:

fallback_python_version_file = module_ctx.path(Label("@@//:.python-version")) Error in path: Unable to load package for //:.python-version: BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.

For testing, there's the mocked tests in //tests/python, which doesn't really catch what .python-version we resolve to because of the mocking, and I've been playing around in examples/multi_python_versions, but I can't have the example test both the python.defaults tag class and the .python-version fallback, unless I copy the entire example. Not sure how to proceed.

We can get the path to module file of the root module in the same way and then try reading .python-version in the same dir.

aignas · 2025-03-21T06:31:43Z

python/private/python.bzl

+                if default_python_version:
+                    is_default = default_python_version == toolchain_version
+                    if toolchain_attr.is_default and not is_default:
+                        fail("The 'is_default' attribute doesn't work if you set " +


Suggested change

fail("The 'is_default' attribute doesn't work if you set " +

logger.warn("The 'is_default' attribute doesn't work if you set " +

Maybe having the logger warn the user is enough here? I imagine people may have a .python-version file that they expect bazel to not read and then suddenly rules_python will start failing, which would be surprising behaviour.

If we enter this branch, wouldn't there be surprising behavior no matter what? If the user has said python.toolchain(python_version = "3.11", is_default = True) and the .python-version file says 3.12, suddenly rules_python will start using 3.12 even though the user has explicitly asked for 3.11 to be the default. It's not that some obscure logic used to choose 3.11 and now some obscure logic chooses 3.12 and that's worth a warning — the user explicitly asked for something that we're now actively refusing to do. That feels more like a fail than a logger.warn, doesn't it? In this scenario, if the user wants rules_python to ignore the .python-version file and keep using 3.11, they would have to migrate to the python.defaults(python_version = "3.11") syntax and stop using is_default. I suppose this may be considered a breaking change, which we might need to document better in the change log and perhaps the failure message or warning. If we don't want a breaking change, we'd have to change the logic so the is_default overrides the .python-version file: in that case, I think a warning would be suitable.

If we enter this branch because the user is using both python.defaults(python_version = "3.12") and python.toolchain(python_version = "3.11", is_default = True), I think that should be a fail rather than a logger.warn, because the user is explicitly asking for things that aren't consistent. We could try to keep track of whether the .python-version file or the python.defaults tag class is the source of the discrepancy and do different error messages (or warning, perhaps, in the case of the .python-version file).

Thanks, makes sense.

aignas · 2025-03-21T06:33:23Z

python/private/python.bzl

+File saying what the default Python version should be. If the contents
+of the file match the {attr}`python_version` attribute of a toolchain,
+this toolchain is the default version. If this attribute is set, the
+{attr}`is_default` attribute of the toolchain is ignored.


Please document that we default to .python-version if we find it.

python/private/python.bzl

aignas · 2025-03-21T06:36:25Z

python/private/python.bzl

+:::{versionchanged} VERSION_NEXT_FEATURE
+This setting is ignored if the default version is set using the `defaults`
+tag class.
+:::


Should we deprecate this in favour of the defaults tag_class?

cc: @rickeylev what do you think?

aignas · 2025-03-21T06:41:48Z

I originally had the idea that supporting reading the Python version from a file would be sufficient to also solve the use case where you want to read it from an environment variable, since you could generate a file with whatever logic you need. However, I read in the documentation of module_ctx.path that "remote repositories and module extensions are executed during the analysis phase and thus cannot depends on a target result (the label should point to a non-generated file)." For this reason I also included a python_version_env attribute in my latest patch, so both the file use case and the environment variable use case can be solved directly in rules_python.

I think you could read the env in a repo rule and write a file (but everything has to be in repository_ctx and no build actions). However, this solution seems to be much cleaner.

Label("@@//:.python-version") resolves to the .python-version file of the root module, if such a file exists.

Implement python.defaults.python_version_env in terms of module_ctx.getenv, which was introduced in Bazel 7.1. This means that Bazel 7.0 users who wish to use the new python_version_env will have to upgrade to Bazel 7.1 or later.

If we read a python_version_file (specified with the defaults tag class or the default @@//:.python-version file), explicitly start watching that file. There are some circumstances where such a watch wouldn't be allowed, but since it's new functionality it doesn't break anything for anyone to insist on the watch. If a specific use case requires the relaxation of the requirement, that can always be considered later.

vonschultz requested review from rickeylev and aignas as code owners January 28, 2025 15:22

aignas reviewed Feb 25, 2025

View reviewed changes

python/private/python.bzl Outdated Show resolved Hide resolved

python/private/python.bzl Outdated Show resolved Hide resolved

python/private/python.bzl Outdated Show resolved Hide resolved

aignas added 6 commits March 7, 2025 14:57

Update python/private/python.bzl

e0b7a3d

Update python/private/python.bzl

f2cc08a

Update python/private/python.bzl

31cb09c

doc: add changelog

86d17aa

Merge branch 'main' into default_version_file

0228b9d

adjust changelog

2b1286d

aignas requested changes Mar 7, 2025

View reviewed changes

vonschultz and others added 2 commits March 18, 2025 13:36

Merge branch 'bazel-contrib:main' into default_version_file

f4631dc

aignas reviewed Mar 21, 2025

View reviewed changes

vonschultz added 3 commits March 21, 2025 12:52

Use Label when resolving @@//:.python-version

678a608

Label("@@//:.python-version") resolves to the .python-version file of the root module, if such a file exists.

Drop support for Bazel 7.0 in python_version_env

b5a337b

Implement python.defaults.python_version_env in terms of module_ctx.getenv, which was introduced in Bazel 7.1. This means that Bazel 7.0 users who wish to use the new python_version_env will have to upgrade to Bazel 7.1 or later.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(python.toolchain): support file-based default Python version #2588

feat(python.toolchain): support file-based default Python version #2588

vonschultz commented Jan 28, 2025

aignas left a comment •

edited

Loading

aignas left a comment •

edited

Loading

vonschultz commented Mar 11, 2025

aignas commented Mar 11, 2025

vonschultz commented Mar 19, 2025

aignas left a comment

aignas Mar 20, 2025

vonschultz Mar 21, 2025

aignas Mar 21, 2025

aignas Mar 21, 2025

vonschultz Mar 21, 2025

aignas Mar 21, 2025

aignas Mar 21, 2025

aignas Mar 21, 2025

aignas commented Mar 21, 2025

	fail("The 'is_default' attribute doesn't work if you set " +
	logger.warn("The 'is_default' attribute doesn't work if you set " +

feat(python.toolchain): support file-based default Python version #2588

Are you sure you want to change the base?

feat(python.toolchain): support file-based default Python version #2588

Conversation

vonschultz commented Jan 28, 2025

aignas left a comment • edited Loading

Choose a reason for hiding this comment

aignas left a comment • edited Loading

Choose a reason for hiding this comment

vonschultz commented Mar 11, 2025

aignas commented Mar 11, 2025

vonschultz commented Mar 19, 2025

aignas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aignas commented Mar 21, 2025

aignas left a comment •

edited

Loading

aignas left a comment •

edited

Loading