Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache _is_setuptools_namespace results to avoid repetitive I/O #2705

Merged

Conversation

correctmost
Copy link
Contributor

Type of Changes

Type
🔨 Refactoring

Description

_is_setuptools_namespace can end up reading the same files over and over.

For example, when running pylint's import-error checks on yt-dlp, ~23,500 redundant open() calls were performed prior to caching.

Closes pylint-dev/pylint#9603.

Stats

Before

Command Mean [s] Min [s] Max [s] Relative
pylint --recursive=y . 17.126 ± 0.084 17.047 17.292 1.00
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    28477    0.140    0.000    0.512    0.000 {built-in method _io.open}

After

Command Mean [s] Min [s] Max [s] Relative
pylint --recursive=y . 16.969 ± 0.065 16.882 17.079 1.00
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     4949    0.047    0.000    0.064    0.000 {built-in method _io.open}

Cache stats at Pylint exit

CacheInfo(hits=23528, misses=1165, maxsize=1024, currsize=1024)

_is_setuptools_namespace can end up reading the same files over
and over.

For example, when running pylint's import-error checks on yt-dlp,
~23,500 redundant open() calls were performed prior to caching.

Closes pylint-dev/pylint#9603.
Copy link
Member

@Pierre-Sassoulas Pierre-Sassoulas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌

@jacobtylerwalls jacobtylerwalls merged commit 8628425 into pylint-dev:main Mar 16, 2025
18 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

E0401 (import-error) checks perform repeated file reads
3 participants