Eagerly computing results limits potential for optimization

[hendrikmakait]

Problem

By eagerly computing, e.g., when calling .head(), we limit the potential for optimizations. For example, it shouldn't matter whether I call df.head(n)[["a", "b"]] or df[["a", "b"]].head(n). With eager computations, we lose the opportunity to push the column projection before the head selection.

Apart from that, in my opinion that eager computation comes as a surprise and makes it harder to argue about when Dask actually computes things.

Proposed solution(s)

In general, limit eager computation as much as possible. For this particular example: Deprecate compute=True as default and switch to compute=False in the future.

[phofl]

With "eagerly computing" you are referring to stuff like head, tail, Len, ...., correct? Just to avoid confusion with the set_index space

[hendrikmakait]

Yes, I loosely refer to everything that calls compute() under the hood to retrieve a result, not to compute data required to perform the actual delayed computation.

[crusaderky]

IMHO it depends.
head() is meant as a quick and dirty development tool. I don't see a problem with it being eager.
An eager len() is, on the other hand, catastrophic as there's a strong user expectation for it to be always trivial.

[hendrikmakait]

Why would head be considered a "quick and dirty" development tool? IMO, there are plenty of use cases where I just want to compute the top K of something.

Honestly, I don't care too much about len in the context of optimization because I don't see any use cases where it would prohibit further optimization even if it's executed eagerly.

[crusaderky]

I guess that's just my coding habits. Yes, if head() is used in production code it should definitely be lazy by default.