3.4 AgnosticEncoder support - Spark Connect

[chris-twiner]

Spark Connect adds another interaction approach, splitting driver and executor. The remote app using spark does so over Arrow instead of InternalRow / Expressions per SPARK-41690. It's definitely not a must for 3.4 support (as shown by #698) but it would require a large chunk of TypedEncoder , TypedDataset etc. to be either largely dupicated or have some clever implicit strategy pushed through and left to only the TypedEncoder.scala to be largely duped.

Place holder issue.

[chris-twiner]

Based on the source code the ConvertToArrow (called by connects' SparkSession) and ExpressionEncoder only support the built in AgnosticEncoders (logic present in ScalaReflection) - it's a locked in system, no way to inject behaviour without classpath hackery.

So custom types, typed datasets (different api ) and injections - i.e. all the cool stuff - don't seem to be possible with Spark Connect as it stands in 3.4.