Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

experimental_index_url can only be used in one pip.parse(), subsequent calls fail #2648

Open
1 task
chrisirhc opened this issue Mar 5, 2025 · 8 comments · May be fixed by #2658
Open
1 task

experimental_index_url can only be used in one pip.parse(), subsequent calls fail #2648

chrisirhc opened this issue Mar 5, 2025 · 8 comments · May be fixed by #2658
Labels
type: pip pip/pypi integration

Comments

@chrisirhc
Copy link
Contributor

chrisirhc commented Mar 5, 2025

🐞 bug report

Affected Rule

pip.parse()

Is this a regression?

I'm not sure.

Description

I have 4 lock files for 2 different Python versions over 2 platforms. When I try to use experimental_index_url on both pip.parse calls, it fails.

🔬 Minimal Reproduction

  • I'll put together a minimum repro repo when I get a chance

Here's an excerpt from MODULE.bazel:

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    experimental_extra_index_urls = PYPI_EXTRA_INDEX_URLS,
    experimental_index_url = PYPI_INDEX_URL,
    experimental_requirement_cycles = base_requirement_cycles,
    extra_pip_args = pip_parse_extra_pip_args,
    hub_name = "third_party_python_base",
    python_version = "3.9",
    requirements_darwin = "//third_party/python/base:requirements_3_9_darwin.txt",
    requirements_linux = "//third_party/python/base:requirements_3_9.txt",
)
pip.parse(
    experimental_extra_index_urls = PYPI_EXTRA_INDEX_URLS,
    experimental_index_url = PYPI_INDEX_URL,
    experimental_requirement_cycles = base_requirement_cycles,
    extra_pip_args = pip_parse_extra_pip_args,
    hub_name = "third_party_python_base",
    python_version = "3.11",
    requirements_darwin = "//third_party/python/base:requirements_3_11_darwin.txt",
    requirements_linux = "//third_party/python/base:requirements_3_11.txt",
)

🔥 Exception or Error


ERROR: /home/user/.cache/bazel/…/6cbf57f86cd7ffd2869796107fcd24ff/external/rules_python~/python/private/pypi/extension.bzl:247:25: Traceback (most recent call last):
        File "/home/user/.cache/bazel/…/6cbf57f86cd7ffd2869796107fcd24ff/external/rules_python~/python/private/pypi/extension.bzl", line 610, column 25, in _pip_impl
                mods = parse_modules(module_ctx)
        File "/home/user/.cache/bazel/…/6cbf57f86cd7ffd2869796107fcd24ff/external/rules_python~/python/private/pypi/extension.bzl", line 485, column 36, in parse_modules
                out = _create_whl_repos(
        File "/home/user/.cache/bazel/…/6cbf57f86cd7ffd2869796107fcd24ff/external/rules_python~/python/private/pypi/extension.bzl", line 247, column 25, in _create_whl_repos
                fail("Attempting to creating a duplicate library {} for {}".format(
Error in fail: Attempting to creating a duplicate library third_party_python_base_311_python_bidi_sdist_0665a082 for python_bidi

🌍 Your Environment

Operating System:

  
Linux
  

Output of bazel version:

  
Build label: 7.4.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Nov 11 21:24:53 2024 (1731360293)
Build timestamp: 1731360293
Build timestamp as int: 1731360293
  

Rules_python version:

  
1.2.0
  

Anything else relevant?

@aignas
Copy link
Collaborator

aignas commented Mar 6, 2025

Ah yes, the sdist repos need to be prefixed/suffixed with the python version. The fix should be reasonably easy - only the repo name should be updated.

PRs are welcome here if someone needs this before I can take a look at it.

@chrisirhc
Copy link
Contributor Author

chrisirhc commented Mar 6, 2025

Thanks for the quick response. Making a reproduction was more difficult than I expected.

I'm assuming you mean somewhere in the following lines, is that right?
https://github.com/bazelbuild/rules_python/blob/a816962e509311c23230730b4b28f9d52a229949/python/private/pypi/whl_repo_name.bzl#L36

@aignas
Copy link
Collaborator

aignas commented Mar 6, 2025

Yes, exactly. I think passing python_version to that function would be enough. Using something like python_version.replace(".", "_") would be a sufficient suffix to not cause clashes, but still work (beware, there should not be any . in the name or the python importing machinery gets angry).

@aignas
Copy link
Collaborator

aignas commented Mar 6, 2025

For reference this comment was wrong, it is unclear yet why this is happening.

@chrisirhc
Copy link
Contributor Author

More debug information on the 2nd sdist, logged on the line:https://github.com/bazelbuild/rules_python/blob/e7d2f09394dd14816310c4c661d2fefab33b2b1b/python/private/pypi/extension.bzl#L246

 repo_name:  third_party_python_base_311_python_bidi_sdist_0665a082
 whl_name:  python_bidi
 args:  {"repo": "third_party_python_base_311", "dep_template": "@third_party_python_base//{name}:{target}", "python_interpreter_target": Label("@@rules_python~~python~python_3_11_host//:python"), ..., "requirement": "python-bidi==0.6.0", "urls": ["…pi/packages/packages/42/72/798318217bf0ee8e68af67e6e23c5e1017b9ab83b96019bf240da7b7a72f/python_bidi-0.6.0.tar.gz"], "sha256": "0665a0826074a9ff8d29640c0c405a2710b671db14fcc8b1c3ee6615ff10b837", "filename": "python_bidi-0.6.0.tar.gz", "experimental_target_platforms": ["cp311_osx_aarch64", "cp311_osx_x86_64"]}
 config_setting:  struct(config_setting = None, filename = "python_bidi-0.6.0.tar.gz", target_platforms = ("cp311_osx_aarch64", "cp311_osx_x86_64"), version = "3.11")

 repo_name:  third_party_python_base_311_python_bidi_sdist_0665a082
 whl_name:  python_bidi
 args:  {"repo": "third_party_python_base_311", "dep_template": "@third_party_python_base//{name}:{target}", "python_interpreter_target": Label("@@rules_python~~python~python_3_11_host//:python"), ..., "requirement": "python-bidi==0.6.0", "urls": ["…pi/packages/packages/42/72/798318217bf0ee8e68af67e6e23c5e1017b9ab83b96019bf240da7b7a72f/python_bidi-0.6.0.tar.gz"], "sha256": "0665a0826074a9ff8d29640c0c405a2710b671db14fcc8b1c3ee6615ff10b837", "filename": "python_bidi-0.6.0.tar.gz", "experimental_target_platforms": ["cp311_linux_aarch64", "cp311_linux_arm", "cp311_linux_ppc", "cp311_linux_s390x", "cp311_linux_x86_64"]}
 config_setting:  struct(config_setting = None, filename = "python_bidi-0.6.0.tar.gz", target_platforms = ("cp311_linux_aarch64", "cp311_linux_arm", "cp311_linux_ppc", "cp311_linux_s390x", "cp311_linux_x86_64"), version = "3.11")

Key part is that the experimental_target_platforms from args and target_platforms from config_setting differ for the same sdist file.

@aignas
Copy link
Collaborator

aignas commented Mar 10, 2025

Is there a bug in parse_requirements.bzl? Hmmm. I think I may know what is happening.

  1. The requirements.txt files for the darwin and linux have different lines for the bidi package. Could you paste them here?
  2. Because they are different, the logic in requirements.txt will think that all of the shas for each platform are unique, but in sdist they are not.

I think we need to fix the parse_requirements.bzl logic to not make assumptions and do the merging of the requirements.txt sources in a more clever way. This has unit tests, so should be doable.

Another option is to handle this at the extension level where the extension is not blindly creating sdists, but that option is less appealing to me because the former is easier to test.

@chrisirhc
Copy link
Contributor Author

chrisirhc commented Mar 10, 2025

  1. The requirements.txt files for the darwin and linux have different lines for the bidi package. Could you paste them here?

Yes, this was exactly the issue. The lines are slightly different, with one hash appearing on darwin lock file but not linux linux lock file, and one hash appearing on linux but not the darwin.

When I initially tried to create the reproduction, I wasn't using the exact same set of hashes as my main repo.
I've updated the reproduction now (requirements_3_11.txt and requirements_3_11_darwin.txt are now different in terms of hashes) and it's showing the same error, https://github.com/chrisirhc/rules_python-repro-multiple-exp/

@chrisirhc
Copy link
Contributor Author

Took a stab at fixing it via #2658 but looks like it breaks some behavior on mac/universal, and any whls. Potentially due to extension.bzl's handling. Let me know what you think.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: pip pip/pypi integration
Projects
None yet
2 participants