compile_pip_requirements does not use credential helper

[jml-derek]

🐞 Bug Report

Affected Rule

The issue is caused by the pip.parse rule with experimental_index_url and experimental_index_url_overrides configurations.

Is this a regression?

I have not tried previous versions. This is my first attempt following the documentation at https://github.com/bazelbuild/rules_python/blob/389431bba6f9a4b46b6cf15dd9cd24a1f52f6e16/docs/pypi-dependencies.md?plain=1#L417.

Description

When using pip.parse with experimental_index_url and experimental_index_url_overrides configured with a credential helper script that directly attaches a token generated from the gcloud CLI, authentication fails. The command still prompts for username and password instead of using the token provided by the helper script.

🔬 Minimal Reproduction

I've created a test repository that demonstrates the issue: https://github.com/derek-jml/test_repo

Steps to reproduce:

1. Clone the repository
2. Run bazel run //:requirements.update
3. Observe that it prompts for username and password instead of using the token

🔥 Exception or Error

When running `bazel run //:requirements.update`, it prompts for username and password authentication instead of using the token provided by the credential helper script.

🌍 Your Environment

Operating System:

[Your OS information]

Output of bazel version:

8.1.1

Rules_python version:

1.2.0

Anything else relevant?
I'm following the documentation for using private PyPI repositories with authentication as described in the pypi-dependencies.md file. The specific section I'm referencing is about using credential helper scripts with experimental_index_url and experimental_index_url_overrides.

The credential helper script generates a token from the gcloud CLI that should be used automatically, but the authentication mechanism appears to ignore this and still prompts for username/password credentials.

[rickeylev]

This potentially sounds like #2640 ?

Getting some more details about when it's prompting for the user/pass would be helpful. If it's because of the pip wheel command, then its the same cause.

Maybe try running with < /dev/null piped in? this would close stdin so the prompt immediately fails and, hopefully, provide a traceback with more details.

[jml-derek]

This potentially sounds like #2640 ?

Getting some more details about when it's prompting for the user/pass would be helpful. If it's because of the pip wheel command, then its the same cause.

Maybe try running with < /dev/null piped in? this would close stdin so the prompt immediately fails and, hopefully, provide a traceback with more details.

there is not a lot of information in bug trace.
//:requirements.update is a compile_pip_requirements target

bazel run --sandbox_debug //:requirements.update < /dev/null
INFO: Analyzed target //:requirements.update (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:requirements.update up-to-date:
bazel-bin/requirements.update
INFO: Elapsed time: 0.235s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/requirements.update '--src=_main/requirements.in' _main/requirements_lock.txt //:requirements.update '--resolver=backtracking' --allow-unsafe --generate-hashes
Updating requirements_lock.txt
User for us-central1-python.pkg.dev:
Aborted!

[dougthor42]

Looks like you're missing some quotes in your JSON output from the cred helper. Give this a try:

-echo {"headers": {"Authorization: Bearer $TOKEN"}}
+echo '{"headers": {"Authorization": ["Bearer '$TOKEN'"]}}'

[dougthor42]

Replied in chat, too, but copying here for posterity and for other's benefit.

1. Your cred helper needs to be executable chmod +x tools/artifact_registry/cred_helper.sh. I assume that's just a minor oversight in the example repo.
2. If you're using rules_python to lock your requirements via compile_pip_requirements, then your requirements.in file needs oauth2accesstoken@.

diff --git a/requirements.in b/requirements.in
index aa57009..b7da3a2 100644
--- a/requirements.in
+++ b/requirements.in
@@ -1,3 +1,3 @@
 --extra-index-url https://pypi.org/simple
---extra-index-url https://us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple
+--extra-index-url https://oauth2accesstoken@us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple
 opacus

With that change, I get auth failures (as expected, as I'm using a dummy $TOKEN.

$ bazel run //:requirements.update                                                                          
INFO: Analyzed target //:requirements.update (90 packages loaded, 4369 targets configured).
INFO: Found 1 target...                                                                                    
Target //:requirements.update up-to-date:                                                                  
  bazel-bin/requirements.update                                                                            
INFO: Elapsed time: 1.225s, Critical Path: 0.03s                                                           
INFO: 1 process: 4 action cache hit, 1 internal.                                                           
INFO: Build completed successfully, 1 total action                                                         
INFO: Running command line: bazel-bin/requirements.update '--src=_main/requirements.in' _main/requirements_lock.txt //:requirements.update '--resolver=backtracking' --allow-unsafe --generate-hashes
Updating requirements_lock.txt                                                                             
  WARNING: 401 Error, Credentials not correct for https://us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple/opacus/
  WARNING: 401 Error, Credentials not correct for https://us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple/numpy/
  WARNING: 401 Error, Credentials not correct for https://us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple/torch/
  WARNING: 401 Error, Credentials not correct for https://us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple/scipy/
...

Your lockfile will then also include oauth2accesstoken@ after running the lock:

diff --git a/requirements_lock.txt b/requirements_lock.txt
index 809b052..5a23e8b 100644
--- a/requirements_lock.txt
+++ b/requirements_lock.txt
@@ -4,7 +4,7 @@
 #
 #    bazel run //:requirements.update
 #
---extra-index-url https://us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple
+--extra-index-url https://oauth2accesstoken@us-central1-python.pkg.dev/steady-state-msmt-simulations/python-repo/simple
 
 filelock==3.17.0 \
     --hash=sha256:533dc2f7ba78dc2f0f531fc6c4940addf7b70a481e269a5a3b93be94ffbe8338 \

[jml-derek]

Root Cause Identified

I believe we've found the issue. There's a fundamental difference in how authentication works between different commands:

bazel run //:requirements.update Authentication

• Does not work with Bazel credential helpers at all
• Requires [keyring](https://cloud.google.com/artifact-registry/docs/python/authentication#keyring) to be installed locally
• When installed, it uses the keyring gcloud toolchain to generate the requirement.txt

Dependency Resolution Authentication

• The credential helper works only when experimental_index_url and experimental_index_url_overrides are set in pip.parse
• This is because these settings control the Bazel downloader's behavior
• If these settings are not provided (meaning we're not using the Bazel downloader), then pip's default downloader is used instead
• The pip downloader will still use keyring in this case, functioning the same way as bazel run //:requirements.update

[aignas]

This is working as intended, I think, because repository_ctx and module_ctx is not used.

However, PRs to integrate are welcome, #2657 may be a good base for that once it is merged.