Quantcast

[SQL] Registering custom Rule[LogicalPlan] using extendedResolutionRules by overriding SparkSession, SessionState, and Analyzer only?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[SQL] Registering custom Rule[LogicalPlan] using extendedResolutionRules by overriding SparkSession, SessionState, and Analyzer only?

Jacek Laskowski
Hi,

We had a Spark-a-thon in Warsaw, Poland [1] where we set on learning
QueryPlan API. My initial idea was to start with Analyzer and register
a custom Rule[LogicalPlan] using extendedResolutionRules [2].

We were glad to have seen the scaladoc:

"Override to provide additional rules for the "Resolution" batch."

after we had got stuck how to register the custom rule. It turned out
to be very different from what you could do using ExperimentalMethods
[3] that offers you two extension points for the query planner (i.e.
SparkPlanner and SparkOptimizer).

We ended up overriding SparkSession, SessionState, and Analyzer which
was an excellent coding exercise for the sparkathon, but could be too
much given the simplicity of ExperimentalMethods for the query
planner.

See a solution: https://gist.github.com/loostro/5a14dbca9b52b841cb97d17ba952943f

Can we do better? Could we have done it using other extension points
to Analyzer? Are there any plans on opening Analyzer (akin to
SparkPlanner)? Why? When? That could be another excellent coding
exercise for sparkathon, couldn't it?

Please help.

[1] https://www.meetup.com/WarsawScala/events/238558617/
[2] https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L107
[3] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/ExperimentalMethods.scala

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Loading...