classes for YARN 2.2

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

classes for YARN 2.2

Ted Yu
Hi,
There're two copies of classes w.r.t. YARN:

-rw-r--r--  1 tyu  staff  9710 Jan  3 10:14
/Users/tyu/spark//yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
-rw-r--r--  1 tyu  staff  8189 Jan  3 10:14
/Users/tyu/spark//yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala

The class under yarn/stable is for YARN 2.2, right ?

Cheers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: classes for YARN 2.2

Patrick Wendell
Yes

On Fri, Jan 3, 2014 at 2:33 PM, Ted Yu <[hidden email]> wrote:

> Hi,
> There're two copies of classes w.r.t. YARN:
>
> -rw-r--r--  1 tyu  staff  9710 Jan  3 10:14
> /Users/tyu/spark//yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
> -rw-r--r--  1 tyu  staff  8189 Jan  3 10:14
> /Users/tyu/spark//yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
>
> The class under yarn/stable is for YARN 2.2, right ?
>
> Cheers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: classes for YARN 2.2

Patrick Wendell
Just to give a bit more color - these exist because the yarn API
changed and we'd like to support both for a while. Rather than
introduce shims we just factored out the common code and maintain
isolated builds for either one.

- Patrick

On Fri, Jan 3, 2014 at 2:34 PM, Patrick Wendell <[hidden email]> wrote:

> Yes
>
> On Fri, Jan 3, 2014 at 2:33 PM, Ted Yu <[hidden email]> wrote:
>> Hi,
>> There're two copies of classes w.r.t. YARN:
>>
>> -rw-r--r--  1 tyu  staff  9710 Jan  3 10:14
>> /Users/tyu/spark//yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
>> -rw-r--r--  1 tyu  staff  8189 Jan  3 10:14
>> /Users/tyu/spark//yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
>>
>> The class under yarn/stable is for YARN 2.2, right ?
>>
>> Cheers
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: classes for YARN 2.2

Ted Yu
In yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
, at line 86:

    // Workaround until hadoop moves to something which has
    // https://issues.apache.org/jira/browse/HADOOP-8406 - fixed in
(2.0.2-alpha but no 0.23 line)
    //
org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(conf)

Looks like the above comment can be removed since hadoop 2.2 has HADOOP-8406

Cheers


On Fri, Jan 3, 2014 at 2:39 PM, Patrick Wendell <[hidden email]> wrote:

> Just to give a bit more color - these exist because the yarn API
> changed and we'd like to support both for a while. Rather than
> introduce shims we just factored out the common code and maintain
> isolated builds for either one.
>
> - Patrick
>
> On Fri, Jan 3, 2014 at 2:34 PM, Patrick Wendell <[hidden email]>
> wrote:
> > Yes
> >
> > On Fri, Jan 3, 2014 at 2:33 PM, Ted Yu <[hidden email]> wrote:
> >> Hi,
> >> There're two copies of classes w.r.t. YARN:
> >>
> >> -rw-r--r--  1 tyu  staff  9710 Jan  3 10:14
> >>
> /Users/tyu/spark//yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
> >> -rw-r--r--  1 tyu  staff  8189 Jan  3 10:14
> >>
> /Users/tyu/spark//yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
> >>
> >> The class under yarn/stable is for YARN 2.2, right ?
> >>
> >> Cheers
>
Loading...