-
-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Predicate validation in parquet could happen before compute #753
Comments
cc @rjzamora if you have thoughts where this belongs |
It should raise once dask-expr actually passes the filters to pyarrow, but this won't happen until we need to know the number of partitions. So, |
Just to clarify: We can still add an extra/earlier validation step to dask-expr. I'll take a look to see where that would be easiest. |
Sorry, you are completely correct, it's raising when I call compute. Ideally we would fail earlier, but this is by no means urgent, I'll change the title |
Ideally, this would raise before we trigger compute
The text was updated successfully, but these errors were encountered: