-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement chunked foreach
/non allocating chunked iteration.
#37
Comments
Maybe we can achieve this in a more general way by On the first iteration we can allocate the required memory for the number of chunks along the column, and copy to it from disk when the iterator gets to the next chunk. This may also resolve issues with methods like |
I am still not sure I completely understand the issue here. What exactly is the use case for iteration here that can not be accomplished using |
As pointed out in #15 broadcast can do pretty much everything. But one issue is you have to call
collect
and allocate an array?In GeoData.jl I'm using broadcast to iterate over arrays to e.g. calculate ranges of non nodata values for a
trim
method, usingbroadcast
so the operation is chunked. But I still need to callcollect
on the broadcast object, which allocates.It would be good if there was a way to iterate over the object with chunking that didn't allocate. Maybe that's possible already?
The text was updated successfully, but these errors were encountered: