Windows test collection fails with UnicodeDecodeError in provider dependency refresh

[KristoferFangrat]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

main checkout at commit 49674b7

What happened and how to reproduce it?

On Windows, running a focused provider test can fail during pytest collection before any tests are collected. The tests_common.pytest_plugin invokes scripts/ci/prek/update_providers_dependencies.py, which reads UTF-8 project files with Path.read_text() without an explicit encoding. With the Windows cp1252 locale this raises UnicodeDecodeError.

Observed error:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position ...
...
subprocess.CalledProcessError: ... update_providers_dependencies.py

The failure occurs before the test file runs.

What you think should happen instead?

Airflow's provider dependency refresh and pytest collection should read repository files as UTF-8 explicitly and work on Windows without requiring a locale override.

Operating System

Windows 11

Deployment

None

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

Airflow main checkout (2026-09-22), Python 3.11, pytest 9.1.1

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

Reproduction on Windows with a non-UTF-8 default locale:

set PYTHONUTF8=0
set PYTHONPATH=devel-common/src;providers/google/src
.venv\Scripts\python.exe -m pytest providers/google/tests/unit/google/cloud/operators/test_stackdriver.py -q

Pytest collection fails before tests run. Setting PYTHONUTF8=1 allows collection to progress, which points to locale-dependent file reads.

Relevant files:

• devel-common/src/tests_common/pytest_plugin.py
• scripts/ci/prek/update_providers_dependencies.py

Both contain repository-file reads without an explicit UTF-8 encoding. GitHub search found no existing Airflow issue for this exact Windows/cp1252 failure.

Are you willing to submit a PR?

• Yes

Code of Conduct:

• I agree to follow the project's Code of Conduct.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[kartik00052]

please assign this issue to me
I have to work on it

[schizophrenicmaniac]

I'd like to work on this. My approach: find all Path.read_text()/open() calls without explicit encoding in pytest_plugin.py and update_providers_dependencies.py, add encoding="utf-8" to them (and matching write calls), then verify collection succeeds on Windows without needing PYTHONUTF8=1. Will submit a PR once validated.

[kartik00052]

I can confirm and narrow this down exactly — grepped both files for .read_text() / .write_text() / open() calls without an explicit encoding:

devel-common/src/tests_common/pytest_plugin.py

• L169: AIRFLOW_PYPROJECT_TOML_FILE_PATH.read_text().splitlines()
• L201: PROVIDER_DEPENDENCIES_JSON_HASH_PATH.read_text().strip()
• L203: PROVIDER_DEPENDENCIES_JSON_HASH_PATH.write_text(calculated_provider_deps_hash)

scripts/ci/prek/update_providers_dependencies.py

• L73: pyproject_toml_file_path.read_text() (fed into tomllib.loads)
• L98: provider_yaml_file.read_text() (fed into yaml.safe_load)
• L230, L233: DEPENDENCIES_JSON_FILE_PATH.read_text()
• L235: DEPENDENCIES_JSON_FILE_PATH.write_text(new_content)

That's 8 call sites total, no bare open() calls in either file — so the fix is contained to adding encoding="utf-8" to these 8 read_text/write_text calls, nothing broader in scope.

Planning to validate by reproducing with set PYTHONUTF8=0 per the repro steps above, confirming collection fails first, then confirming it passes after the change — and separately confirming behavior is unchanged on Linux/macOS (these are already UTF-8 by default there, so this should be a no-op change on non-Windows). Will submit a PR once that's verified.

[KristoferFangrat]

@kartik00052
Thanks for investigating this and for listing the affected call sites. I’ll let you proceed with this issue and look for another contribution opportunity.