Improve performance and output of pyenv virtualenvs#502
Improve performance and output of pyenv virtualenvs#502native-api merged 35 commits intopyenv:masterfrom
pyenv virtualenvs#502Conversation
|
I could use some help getting the tests correct. I made a few changes but I don't understand why the |
|
Seems like only two tests are failing 😄 I just randomly checked into virtualenv. I recently did some perf improvements on the main I am not sure if this applies to your situation as well, haven't looked much into it. The main idea I have is:
|
|
The tests do need some adjusting since I've changed the output a bit. I'm just having a hard time figuring out what exactly |
|
Hehe, and the test file hasn't been touched in 10 years 😆 I did not encounter |
|
The code itself that was the source of the performance issue is 10-12 years old. Not that old code is bad, but there were some inefficiencies in there. |
|
No it's not that the old code is bad. It was probably good at the time it was written. It's more of a problem that the code has design principles which made sense back then, but fast forward 10 years and it's not as intuitive anymore. And if you're unlucky, you're facing tests which were quick-fix set up 10 years ago, and suddenly you need to fix the problem of ... It's a challenge for sure 😄 |
|
Thank you for the feedback @native-api. I'll address this later in the week. |
|
I had a crazy week and a full weekend away from home. I will work on this during the week. |
|
Thanks for all the feedback. I didn't get to fixing up the tests. I need to get some sleep for now. |
|
Did this get dropped? Would love a speedup on virtualenvs 👍 |
|
I have not forgotten about this. My day job has consumed all my capacity lately so I haven't circled back to address the test failures. I'm kinda cheating because I started using the changes in this branch locally so haven't felt the pain in a while. |
|
The prompt hook performance issue (#259) is fixed in v1.3.0 via #523, which caches the active version to skip redundant |
|
I'll try out the fixes in v1.3.0 and see how they affect the issue I was seeing. Thanks for the improvements you made. |
|
So it does seem like this helps with the prompt completion, but |
samdoran
force-pushed
the
improve-performance-and-output
branch
2 times, most recently
from
bdcfb08 to
cbaa29a
Compare
|
Sorry this took so long, but I finally circled back and got the tests in order. I fixed some existing issues and added more tests cases. Please review when you have a chance. Thank you! |
|
Thanks for fixing those things up. |
samdoran
force-pushed
the
improve-performance-and-output
branch
from
894a488 to
64b8644
Compare
Use the code from pyenv-versions for efficiency and consistent output. The main performance problem was in the call to pyenv-virtualenv-prefix, which called pyenv-prefix, which then enumerated every virtual environment. This was done inside a loop, compounding the problem. Simply the virtual environment listing so that it does not have to call pyenv-virtualenv-prefix anymore.
This is useful for listing the “frinedly” virtualenv names instead of the short path which includes the Python version.
|
The echo "${VIRTUALENV_PREFIX_PATHS[*]}" statement at the end of the file
only worked by accident because IFS=: was set earlier in the file.
Properly set/unset IFS.
|
Doh, let me see what's up with the test failures. |
This refelects how call changes have changed in the code.
|
I needed to remove the |
The warning was removed, so testing no empty output doesn’t make much sense.
Also have create_m_venv call setup_m_venv.
Venvs created from system Python are directories, not sym links. The previous code only displayed venvs that were a directory and a symlink. Now check for directory existence and symlink status, or directory existence and the presence of bin/activate within the directory.
This was only used by the warning code, which was removed.
Change the main setup to only set PYENV_ROOT. Each test now sets up the required virtual environments. This results in some duplication but makes it clearer what each test is doing. An alternative approach would be to modify the venv created by create_m_venv to make it look like a system venv, but two distinct functions make it much clearer what they are doing even though there is some overlap between the functions.
|
The tests should pass. 😁 |
This makes test setup more resilient.
|
I added a test to prevent performance regressions. I can omit that if you think it's excessive. |
Also stub out calls to pyenv-virtualenv-prefix in the test that’s listing venvs.
Making this an integration test ended up giving mixed results. This test when run without the performance changes in this branch did not reflect the performance gains seen when running pyenv-virtualenvs outside of the test environment. The test was not accomplishing its goal.
|
After running the performance test on |
no longer needed now that PYENV_ROOT is under BATS_TEST_DIRNAME
|
Okay, as a highly-requested change, this qualifies for a payout from donated money (tax-exempt). If you are interested, please register at https://opencollective.com and provide your handle name so I can set it up. |
|
Thank you so much for working with me through this. |
5 participants
The biggest change is reusing most of the code used for
pyenv versionsinpyenv virtualenvs. This dramatically improves performance while making the output consistent.Related to #490. Maybe fixes the problem?
With 128 virtual environments, it currently takes about 13 seconds to run
pyenv virtualenvs --bare.Output
With the changes in this branch it is 2.9s:
Output
Output formatting changes
This PR also includes changes to the output format to make it more consistent with
pyenv versions.The completions for
pyenv virtualenvnow omit envs.