-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add file handle API #375
base: main
Are you sure you want to change the base?
Add file handle API #375
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #375 +/- ##
==========================================
- Coverage 87.84% 87.78% -0.06%
==========================================
Files 85 85
Lines 11246 11257 +11
Branches 2084 2087 +3
==========================================
+ Hits 9879 9882 +3
- Misses 868 871 +3
- Partials 499 504 +5 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…upports recent towncrier version
68ca85b
to
18a0553
Compare
18a0553
to
6d4a543
Compare
Co-authored-by: Jonas Lähnemann <[email protected]>
@CSSFrancis, what do you think about this API? |
@ericpre I really like this change and think it was a long time coming. That being said I think we need to think a little bit about more how we are handling files and what the relationship between hyperspy --> dask --> underlying file. More and more it's seeming like More than likely what we need is a abstract class which is something like https://docs.xarray.dev/en/latest/generated/xarray.backends.CachingFileManager.html which would allow File Objects to be passed from one process to the next without pickling. This is how To maybe take it a step further, I would like to build a bit more context support for things like rechunking. I think this would involve something like the optimized slicing that https://github.com/Blosc/b2h5py/blob/317f4a0c31ac61623bf49861f3f5c6ebce46f633/b2h5py/blosc2.py#L248 Mostly it would just be optimized slicing for things like virtual images, plotting etc. |
@CSSFrancis, thank you, this sounds good. Regarding this PR, would it be better to put it on hold until a file manager is implemented to return the file manager object instead of the file handle? Even if they may have similar API, there are not the same and therefore can't be swapped without breaking the API? |
Closes #331.
Progress of the PR
file_handle
API,file_handle
API for the following format,upcoming_changes
folder (seeupcoming_changes/README.rst
),docs/readthedocs.org:rosettasciio
build of this PR (link in github checks)