Open
Description
With #2981 we adopt a level-based traversal execution strategy. The memory footprint of this strategy is dependent on the shape of the hierarchy; so for extremely wide hierarchies, it would consume a lot of memory which is not very efficient.
Under certain conditions, we can adopt a more efficient DFS traversal strategy that is much more efficient for shallow (but extremely wide) hierarchies since the memory consumption in this case is O(depth)
of the hierarchy. Implementing this requires a new execution runtime that is more or less a special implementation of FlatMapPipelinedCursor
.