[EPIC] Improve Comet planning

[andygrove]

What is the problem the feature request solves?

Comet currently eagerly tries to convert each operator to a Comet operator. This sometimes leads to inefficient plans that can be slower than just falling back to Spark.

One example is where a Comet JVM shuffle is inserted for a child plan that runs in Spark, to convert to columner shuffle, and then the next stage converts back to row-based right away.

+- HashAggregate
  +- CometNativeColumnarToRow
     +- CometColumnarExchange
        +- HashAggregate

Another example we saw recently was with DPP fallback, and there was a fix applied for that specific case.

I think it is time to start looking at the overall planning strategy.

Document to discuss:

https://docs.google.com/document/d/1ux_SwZPd64VTtuTC9lC46oIZGKcoH6KhWOS8xlqwdx8/edit?usp=sharing

Tracking for related issues:

• Avoid JVM shuffle when parent stage will just convert back to rows #4004

Describe the potential solution

No response

Additional context

No response

[andygrove]

Another example is where we fall back to Spark for SortMergeJoin but keep the inputs as CometSort. It would be better to fallback for the sorts as well.