[DISCUSS] commit to ExpressionEncoder?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] commit to ExpressionEncoder?

Erik Erlandson-2
Currently the design of Encoder implies the possibility that encoders might be customized, or at least that there are other internal alternatives to ExpressionEncoder. However there are both implicit and explicit restrictions in spark-sql, such that ExpressionEncoder is the only functional option, including but not limited to:

  def encoderFor[A : Encoder]: ExpressionEncoder[A] = implicitly[Encoder[A]] match {
    case e: ExpressionEncoder[A] =>
      e.assertUnresolved()
      e
    case _ => sys.error(s"Only expression encoders are supported today")
  }

My impression, based on recent work around the SQL code, is that:
1) ExpressionEncoder does support the space of plausible data types and objects, including custom ones.
2) Supporting other subclasses of Encoder in actuality would be nontrivial

Should spark SQL more explicitly commit to an "ExpressionEncoder only" position?