start spark code without applicatio.jar and submit

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

start spark code without applicatio.jar and submit


we want to execute spark code with out submit application.jar,like this code:

public static void main(String args[]) throws Exception{
        SparkSession spark = SparkSession
                .appName("spark test")
        Dataset<Row> testData =".\\src\\main\\java\\Resources\\no_schema_iris.scv");

the above code can work well with idea , do not need to generate jar file and submit , but if we replace master("local[*]") with master("yarn") , it can't work , so is there a way to use cluster sparkSession like local sparkSession ?  we need to dynamically execute spark code in web server according to the different request ,  such as filter request will call dataset.filter() , so there is no application.jar to submit .
--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]