Permanent UDF support across session

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Permanent UDF support across session

Ajith shetty
I have a question related to Permanent UDF for spark enabled hive support. 

When we do create function, this is registered with hive via
 spark-sql>create function customfun as 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFLastDay' using jar 'hdfs:///tmp/hive-exec.jar';
 call stack:
 org.apache.spark.sql.hive.client.HiveClientImpl#createFunction
 org.apache.spark.sql.hive.HiveExternalCatalog#createFunction
 org.apache.spark.sql.catalyst.catalog.SessionCatalog#createFunction
 org.apache.spark.sql.execution.command.CreateFunctionCommand#run


but when we call a registered UDF, we do ADD JAR call to hive
 spark-sql> select customfun('2015-08-22');
 call stack:
 org.apache.spark.sql.hive.client.HiveClientImpl#addJar
 org.apache.spark.sql.hive.HiveSessionResourceLoader#addJar
 org.apache.spark.sql.internal.SessionResourceLoader#loadResource
 org.apache.spark.sql.catalyst.catalog.SessionCatalog#loadFunctionResources
 org.apache.spark.sql.catalyst.catalog.SessionCatalog#lookupFunction

so is the ADD JAR call to hive necessary when we invoke a already registered UDF.? as i see if we follow current code, 
1. hive can lookup already registered UDFs without explicit add jar call from spark , Refer https://cwiki.apache.org/confluence/display/Hive/HivePlugins fixed via https://issues.apache.org/jira/browse/HIVE-6380 ( When the function is referenced for the first time by a Hive session, these resources will be added to the environment. )
2. We cannot have across session as the new session again need to do add jar internally on UDF call, which will fail as caller neeed to have a admin role set ( hive requires add jar to be run only via admin role )

Please correct me if i am wrong, can we avoid add jar when we invoke a registered UDF.? any side-effects if i modify this flow.?