Only create one native plan for a query on an executor

[viirya]

What is the problem the feature request solves?

Currently we create a native plan per task for a query. Even those tasks are on same executor, they still have separate native plans. For the approach, one problem is the additional cost for deserializing the same query plan multiple times and extra memory storing duplicate native plans.

Describe the potential solution

No response

Additional context

No response

[andygrove]

One of the challenges related to this is that we start executing ScanExec during construction, as part of query planning. If we want to share one plan per executor (which I agree would be good), then we will need to postpone execution until the execute function is called.

[viirya]

Okay. I forgot this point. It should be the cause of many test failures in the draft PR #1203. I think it is a good direction to go, not just for performance but also for one possible feature I would like to achieve (the feature is only possible when only one native plan on an executor).

[viirya]

I will think if there is a possible way to work around the issue.

[andygrove]

I've been considering proposing a change in DataFusion to relax the rules when creating record batches in an operator. Rather than requiring the same physical types for each batch, it would be better just to enforce the same logical type and then coerce the physical types when needed. For example, with a string column, we would then be able to use either Utf8 or Dictionary<_, Utf8>.

I'm not sure how large a change it would be, but I think that would solve our problem.

[viirya]

It seems a fundamental rule in DataFusion physical plan. In many places, physical schema of children operator are used. So once any difference is found between physical schema during planning and actual physical schema during execution, an error would be happened. In these places, some type coercions might be needed to add, maybe it also requires some array casting during execution.