Quantcast

Updating docs for running on Mesos

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Updating docs for running on Mesos

Andrew Ash
The docs for how to run Spark on Mesos have changed very little since
0.6.0, but setting it up is much easier now than then.  Does it make sense
to revamp with the below changes?


You no longer need to build mesos yourself as pre-built versions are
available from Mesosphere: http://mesosphere.io/downloads/

And the instructions guide you towards compiling your own distribution of
Spark, when you can use the prebuilt versions of Spark as well.


I'd like to split that portion of the documentation into two sections, a
build-from-scratch section and a use-prebuilt section.  The new outline
would look something like this:


*Running Spark on Mesos*

Installing Mesos
- using prebuilt (recommended)
 - pointer to mesosphere's packages
- from scratch
 - (similar to current)


Connecting Spark to Mesos
- loading distribution into an accessible location
- Spark settings

Mesos Run Modes
- (same as current)

Running Alongside Hadoop
- (trim this down)



Does that work for people?


Thanks!
Andrew


PS Basically all the same:

http://spark.apache.org/docs/0.6.0/running-on-mesos.html
http://spark.apache.org/docs/0.6.2/running-on-mesos.html
http://spark.apache.org/docs/0.7.3/running-on-mesos.html
http://spark.apache.org/docs/0.8.1/running-on-mesos.html
http://spark.apache.org/docs/0.9.1/running-on-mesos.html
https://people.apache.org/~pwendell/spark-1.0.0-rc3-docs/running-on-mesos.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updating docs for running on Mesos

Patrick Wendell
Andrew,

Updating these docs would be great! I think this would be a welcome change.

In terms of packaging, it would be good to mention the binaries
produced by the upstream project as well, in addition to Mesosphere.

- Patrick

On Thu, May 8, 2014 at 12:51 AM, Andrew Ash <[hidden email]> wrote:

> The docs for how to run Spark on Mesos have changed very little since
> 0.6.0, but setting it up is much easier now than then.  Does it make sense
> to revamp with the below changes?
>
>
> You no longer need to build mesos yourself as pre-built versions are
> available from Mesosphere: http://mesosphere.io/downloads/
>
> And the instructions guide you towards compiling your own distribution of
> Spark, when you can use the prebuilt versions of Spark as well.
>
>
> I'd like to split that portion of the documentation into two sections, a
> build-from-scratch section and a use-prebuilt section.  The new outline
> would look something like this:
>
>
> *Running Spark on Mesos*
>
> Installing Mesos
> - using prebuilt (recommended)
>  - pointer to mesosphere's packages
> - from scratch
>  - (similar to current)
>
>
> Connecting Spark to Mesos
> - loading distribution into an accessible location
> - Spark settings
>
> Mesos Run Modes
> - (same as current)
>
> Running Alongside Hadoop
> - (trim this down)
>
>
>
> Does that work for people?
>
>
> Thanks!
> Andrew
>
>
> PS Basically all the same:
>
> http://spark.apache.org/docs/0.6.0/running-on-mesos.html
> http://spark.apache.org/docs/0.6.2/running-on-mesos.html
> http://spark.apache.org/docs/0.7.3/running-on-mesos.html
> http://spark.apache.org/docs/0.8.1/running-on-mesos.html
> http://spark.apache.org/docs/0.9.1/running-on-mesos.html
> https://people.apache.org/~pwendell/spark-1.0.0-rc3-docs/running-on-mesos.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updating docs for running on Mesos

andy
Administrator
In reply to this post by Andrew Ash
Thanks for suggesting this and volunteering to do it.

On May 11, 2014 3:32 AM, "Andrew Ash" <[hidden email]> wrote:

>
> The docs for how to run Spark on Mesos have changed very little since
> 0.6.0, but setting it up is much easier now than then.  Does it make sense
> to revamp with the below changes?
>
>
> You no longer need to build mesos yourself as pre-built versions are
> available from Mesosphere: http://mesosphere.io/downloads/
>
> And the instructions guide you towards compiling your own distribution of
> Spark, when you can use the prebuilt versions of Spark as well.
>
>
> I'd like to split that portion of the documentation into two sections, a
> build-from-scratch section and a use-prebuilt section.  The new outline
> would look something like this:
>
>
> *Running Spark on Mesos*
>
> Installing Mesos
> - using prebuilt (recommended)
>  - pointer to mesosphere's packages
> - from scratch
>  - (similar to current)
>
>
> Connecting Spark to Mesos
> - loading distribution into an accessible location
> - Spark settings
>
> Mesos Run Modes
> - (same as current)
>
> Running Alongside Hadoop
> - (trim this down)

What trimming do you have in mind here?
https://people.apache.org/~pwendell/spark-1.0.0-rc3-docs/running-on-mesos.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updating docs for running on Mesos

Andrew Ash
For trimming the Running Alongside Hadoop section I mostly think there
should be a separate Spark+HDFS section and have the CDH+HDP page be merged
into that one, but I supposed that's a separate docs change.


On Sun, May 11, 2014 at 4:28 PM, Andy Konwinski <[hidden email]>wrote:

> Thanks for suggesting this and volunteering to do it.
>
> On May 11, 2014 3:32 AM, "Andrew Ash" <[hidden email]> wrote:
> >
> > The docs for how to run Spark on Mesos have changed very little since
> > 0.6.0, but setting it up is much easier now than then.  Does it make
> sense
> > to revamp with the below changes?
> >
> >
> > You no longer need to build mesos yourself as pre-built versions are
> > available from Mesosphere: http://mesosphere.io/downloads/
> >
> > And the instructions guide you towards compiling your own distribution of
> > Spark, when you can use the prebuilt versions of Spark as well.
> >
> >
> > I'd like to split that portion of the documentation into two sections, a
> > build-from-scratch section and a use-prebuilt section.  The new outline
> > would look something like this:
> >
> >
> > *Running Spark on Mesos*
> >
> > Installing Mesos
> > - using prebuilt (recommended)
> >  - pointer to mesosphere's packages
> > - from scratch
> >  - (similar to current)
> >
> >
> > Connecting Spark to Mesos
> > - loading distribution into an accessible location
> > - Spark settings
> >
> > Mesos Run Modes
> > - (same as current)
> >
> > Running Alongside Hadoop
> > - (trim this down)
>
> What trimming do you have in mind here?
>
> >
> >
> >
> > Does that work for people?
> >
> >
> > Thanks!
> > Andrew
> >
> >
> > PS Basically all the same:
> >
> > http://spark.apache.org/docs/0.6.0/running-on-mesos.html
> > http://spark.apache.org/docs/0.6.2/running-on-mesos.html
> > http://spark.apache.org/docs/0.7.3/running-on-mesos.html
> > http://spark.apache.org/docs/0.8.1/running-on-mesos.html
> > http://spark.apache.org/docs/0.9.1/running-on-mesos.html
> >
>
> https://people.apache.org/~pwendell/spark-1.0.0-rc3-docs/running-on-mesos.html
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updating docs for running on Mesos

Andrew Ash
In reply to this post by Patrick Wendell
As far as I know, the upstream doesn't release binaries, only source code.
 The downloads page <https://mesos.apache.org/downloads/> for 0.18.0 only
has a source tarball.  Is there a binary release somewhere from Mesos that
I'm missing?


On Sun, May 11, 2014 at 2:16 PM, Patrick Wendell <[hidden email]> wrote:

> Andrew,
>
> Updating these docs would be great! I think this would be a welcome change.
>
> In terms of packaging, it would be good to mention the binaries
> produced by the upstream project as well, in addition to Mesosphere.
>
> - Patrick
>
> On Thu, May 8, 2014 at 12:51 AM, Andrew Ash <[hidden email]> wrote:
> > The docs for how to run Spark on Mesos have changed very little since
> > 0.6.0, but setting it up is much easier now than then.  Does it make
> sense
> > to revamp with the below changes?
> >
> >
> > You no longer need to build mesos yourself as pre-built versions are
> > available from Mesosphere: http://mesosphere.io/downloads/
> >
> > And the instructions guide you towards compiling your own distribution of
> > Spark, when you can use the prebuilt versions of Spark as well.
> >
> >
> > I'd like to split that portion of the documentation into two sections, a
> > build-from-scratch section and a use-prebuilt section.  The new outline
> > would look something like this:
> >
> >
> > *Running Spark on Mesos*
> >
> > Installing Mesos
> > - using prebuilt (recommended)
> >  - pointer to mesosphere's packages
> > - from scratch
> >  - (similar to current)
> >
> >
> > Connecting Spark to Mesos
> > - loading distribution into an accessible location
> > - Spark settings
> >
> > Mesos Run Modes
> > - (same as current)
> >
> > Running Alongside Hadoop
> > - (trim this down)
> >
> >
> >
> > Does that work for people?
> >
> >
> > Thanks!
> > Andrew
> >
> >
> > PS Basically all the same:
> >
> > http://spark.apache.org/docs/0.6.0/running-on-mesos.html
> > http://spark.apache.org/docs/0.6.2/running-on-mesos.html
> > http://spark.apache.org/docs/0.7.3/running-on-mesos.html
> > http://spark.apache.org/docs/0.8.1/running-on-mesos.html
> > http://spark.apache.org/docs/0.9.1/running-on-mesos.html
> >
> https://people.apache.org/~pwendell/spark-1.0.0-rc3-docs/running-on-mesos.html
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updating docs for running on Mesos

Andrew Ash
I have a draft of my proposed changes here:

https://github.com/apache/spark/pull/756
https://issues.apache.org/jira/browse/SPARK-1818

Thanks!
Andrew


On Mon, May 12, 2014 at 9:57 PM, Andrew Ash <[hidden email]> wrote:

> As far as I know, the upstream doesn't release binaries, only source code.
>  The downloads page <https://mesos.apache.org/downloads/> for 0.18.0 only
> has a source tarball.  Is there a binary release somewhere from Mesos that
> I'm missing?
>
>
> On Sun, May 11, 2014 at 2:16 PM, Patrick Wendell <[hidden email]>wrote:
>
>> Andrew,
>>
>> Updating these docs would be great! I think this would be a welcome
>> change.
>>
>> In terms of packaging, it would be good to mention the binaries
>> produced by the upstream project as well, in addition to Mesosphere.
>>
>> - Patrick
>>
>> On Thu, May 8, 2014 at 12:51 AM, Andrew Ash <[hidden email]> wrote:
>> > The docs for how to run Spark on Mesos have changed very little since
>> > 0.6.0, but setting it up is much easier now than then.  Does it make
>> sense
>> > to revamp with the below changes?
>> >
>> >
>> > You no longer need to build mesos yourself as pre-built versions are
>> > available from Mesosphere: http://mesosphere.io/downloads/
>> >
>> > And the instructions guide you towards compiling your own distribution
>> of
>> > Spark, when you can use the prebuilt versions of Spark as well.
>> >
>> >
>> > I'd like to split that portion of the documentation into two sections, a
>> > build-from-scratch section and a use-prebuilt section.  The new outline
>> > would look something like this:
>> >
>> >
>> > *Running Spark on Mesos*
>> >
>> > Installing Mesos
>> > - using prebuilt (recommended)
>> >  - pointer to mesosphere's packages
>> > - from scratch
>> >  - (similar to current)
>> >
>> >
>> > Connecting Spark to Mesos
>> > - loading distribution into an accessible location
>> > - Spark settings
>> >
>> > Mesos Run Modes
>> > - (same as current)
>> >
>> > Running Alongside Hadoop
>> > - (trim this down)
>> >
>> >
>> >
>> > Does that work for people?
>> >
>> >
>> > Thanks!
>> > Andrew
>> >
>> >
>> > PS Basically all the same:
>> >
>> > http://spark.apache.org/docs/0.6.0/running-on-mesos.html
>> > http://spark.apache.org/docs/0.6.2/running-on-mesos.html
>> > http://spark.apache.org/docs/0.7.3/running-on-mesos.html
>> > http://spark.apache.org/docs/0.8.1/running-on-mesos.html
>> > http://spark.apache.org/docs/0.9.1/running-on-mesos.html
>> >
>> https://people.apache.org/~pwendell/spark-1.0.0-rc3-docs/running-on-mesos.html
>>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updating docs for running on Mesos

Gerard Maas
In reply to this post by Andrew Ash
Andrew,

Mesosphere has binary releases here:
http://mesosphere.io/downloads/

(Anecdote: I actually burned a CPU building Mesos from source. No kidding -
it was coming, as the laptop was crashing from time to time, but the mesos
build was that one drop too much)

kr, Gerard.



On Tue, May 13, 2014 at 6:57 AM, Andrew Ash <[hidden email]> wrote:

> As far as I know, the upstream doesn't release binaries, only source code.
>  The downloads page <https://mesos.apache.org/downloads/> for 0.18.0 only
> has a source tarball.  Is there a binary release somewhere from Mesos that
> I'm missing?
>
>
> On Sun, May 11, 2014 at 2:16 PM, Patrick Wendell <[hidden email]>
> wrote:
>
> > Andrew,
> >
> > Updating these docs would be great! I think this would be a welcome
> change.
> >
> > In terms of packaging, it would be good to mention the binaries
> > produced by the upstream project as well, in addition to Mesosphere.
> >
> > - Patrick
> >
> > On Thu, May 8, 2014 at 12:51 AM, Andrew Ash <[hidden email]>
> wrote:
> > > The docs for how to run Spark on Mesos have changed very little since
> > > 0.6.0, but setting it up is much easier now than then.  Does it make
> > sense
> > > to revamp with the below changes?
> > >
> > >
> > > You no longer need to build mesos yourself as pre-built versions are
> > > available from Mesosphere: http://mesosphere.io/downloads/
> > >
> > > And the instructions guide you towards compiling your own distribution
> of
> > > Spark, when you can use the prebuilt versions of Spark as well.
> > >
> > >
> > > I'd like to split that portion of the documentation into two sections,
> a
> > > build-from-scratch section and a use-prebuilt section.  The new outline
> > > would look something like this:
> > >
> > >
> > > *Running Spark on Mesos*
> > >
> > > Installing Mesos
> > > - using prebuilt (recommended)
> > >  - pointer to mesosphere's packages
> > > - from scratch
> > >  - (similar to current)
> > >
> > >
> > > Connecting Spark to Mesos
> > > - loading distribution into an accessible location
> > > - Spark settings
> > >
> > > Mesos Run Modes
> > > - (same as current)
> > >
> > > Running Alongside Hadoop
> > > - (trim this down)
> > >
> > >
> > >
> > > Does that work for people?
> > >
> > >
> > > Thanks!
> > > Andrew
> > >
> > >
> > > PS Basically all the same:
> > >
> > > http://spark.apache.org/docs/0.6.0/running-on-mesos.html
> > > http://spark.apache.org/docs/0.6.2/running-on-mesos.html
> > > http://spark.apache.org/docs/0.7.3/running-on-mesos.html
> > > http://spark.apache.org/docs/0.8.1/running-on-mesos.html
> > > http://spark.apache.org/docs/0.9.1/running-on-mesos.html
> > >
> >
> https://people.apache.org/~pwendell/spark-1.0.0-rc3-docs/running-on-mesos.html
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updating docs for running on Mesos

Matei Zaharia
Administrator
I’ll ask the Mesos folks about this. Unfortunately it might be tough to link only to a company’s builds; but we can perhaps include them in addition to instructions for building Mesos from Apache.

Matei

On May 12, 2014, at 11:55 PM, Gerard Maas <[hidden email]> wrote:

> Andrew,
>
> Mesosphere has binary releases here:
> http://mesosphere.io/downloads/
>
> (Anecdote: I actually burned a CPU building Mesos from source. No kidding -
> it was coming, as the laptop was crashing from time to time, but the mesos
> build was that one drop too much)
>
> kr, Gerard.
>
>
>
> On Tue, May 13, 2014 at 6:57 AM, Andrew Ash <[hidden email]> wrote:
>
>> As far as I know, the upstream doesn't release binaries, only source code.
>> The downloads page <https://mesos.apache.org/downloads/> for 0.18.0 only
>> has a source tarball.  Is there a binary release somewhere from Mesos that
>> I'm missing?
>>
>>
>> On Sun, May 11, 2014 at 2:16 PM, Patrick Wendell <[hidden email]>
>> wrote:
>>
>>> Andrew,
>>>
>>> Updating these docs would be great! I think this would be a welcome
>> change.
>>>
>>> In terms of packaging, it would be good to mention the binaries
>>> produced by the upstream project as well, in addition to Mesosphere.
>>>
>>> - Patrick
>>>
>>> On Thu, May 8, 2014 at 12:51 AM, Andrew Ash <[hidden email]>
>> wrote:
>>>> The docs for how to run Spark on Mesos have changed very little since
>>>> 0.6.0, but setting it up is much easier now than then.  Does it make
>>> sense
>>>> to revamp with the below changes?
>>>>
>>>>
>>>> You no longer need to build mesos yourself as pre-built versions are
>>>> available from Mesosphere: http://mesosphere.io/downloads/
>>>>
>>>> And the instructions guide you towards compiling your own distribution
>> of
>>>> Spark, when you can use the prebuilt versions of Spark as well.
>>>>
>>>>
>>>> I'd like to split that portion of the documentation into two sections,
>> a
>>>> build-from-scratch section and a use-prebuilt section.  The new outline
>>>> would look something like this:
>>>>
>>>>
>>>> *Running Spark on Mesos*
>>>>
>>>> Installing Mesos
>>>> - using prebuilt (recommended)
>>>> - pointer to mesosphere's packages
>>>> - from scratch
>>>> - (similar to current)
>>>>
>>>>
>>>> Connecting Spark to Mesos
>>>> - loading distribution into an accessible location
>>>> - Spark settings
>>>>
>>>> Mesos Run Modes
>>>> - (same as current)
>>>>
>>>> Running Alongside Hadoop
>>>> - (trim this down)
>>>>
>>>>
>>>>
>>>> Does that work for people?
>>>>
>>>>
>>>> Thanks!
>>>> Andrew
>>>>
>>>>
>>>> PS Basically all the same:
>>>>
>>>> http://spark.apache.org/docs/0.6.0/running-on-mesos.html
>>>> http://spark.apache.org/docs/0.6.2/running-on-mesos.html
>>>> http://spark.apache.org/docs/0.7.3/running-on-mesos.html
>>>> http://spark.apache.org/docs/0.8.1/running-on-mesos.html
>>>> http://spark.apache.org/docs/0.9.1/running-on-mesos.html
>>>>
>>>
>> https://people.apache.org/~pwendell/spark-1.0.0-rc3-docs/running-on-mesos.html
>>>
>>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updating docs for running on Mesos

Andrew Ash
Completely agree about preferring to link to the upstream project rather
than a company's -- the only reason I'm using mesosphere's now is that I
see no alternative from mesos.apache.org

I included instructions for both using Mesosphere's packages and building
from scratch in the PR: https://github.com/apache/spark/pull/756


On Tue, May 13, 2014 at 12:59 AM, Matei Zaharia <[hidden email]>wrote:

> I’ll ask the Mesos folks about this. Unfortunately it might be tough to
> link only to a company’s builds; but we can perhaps include them in
> addition to instructions for building Mesos from Apache.
>
> Matei
>
> On May 12, 2014, at 11:55 PM, Gerard Maas <[hidden email]> wrote:
>
> > Andrew,
> >
> > Mesosphere has binary releases here:
> > http://mesosphere.io/downloads/
> >
> > (Anecdote: I actually burned a CPU building Mesos from source. No
> kidding -
> > it was coming, as the laptop was crashing from time to time, but the
> mesos
> > build was that one drop too much)
> >
> > kr, Gerard.
> >
> >
> >
> > On Tue, May 13, 2014 at 6:57 AM, Andrew Ash <[hidden email]>
> wrote:
> >
> >> As far as I know, the upstream doesn't release binaries, only source
> code.
> >> The downloads page <https://mesos.apache.org/downloads/> for 0.18.0
> only
> >> has a source tarball.  Is there a binary release somewhere from Mesos
> that
> >> I'm missing?
> >>
> >>
> >> On Sun, May 11, 2014 at 2:16 PM, Patrick Wendell <[hidden email]>
> >> wrote:
> >>
> >>> Andrew,
> >>>
> >>> Updating these docs would be great! I think this would be a welcome
> >> change.
> >>>
> >>> In terms of packaging, it would be good to mention the binaries
> >>> produced by the upstream project as well, in addition to Mesosphere.
> >>>
> >>> - Patrick
> >>>
> >>> On Thu, May 8, 2014 at 12:51 AM, Andrew Ash <[hidden email]>
> >> wrote:
> >>>> The docs for how to run Spark on Mesos have changed very little since
> >>>> 0.6.0, but setting it up is much easier now than then.  Does it make
> >>> sense
> >>>> to revamp with the below changes?
> >>>>
> >>>>
> >>>> You no longer need to build mesos yourself as pre-built versions are
> >>>> available from Mesosphere: http://mesosphere.io/downloads/
> >>>>
> >>>> And the instructions guide you towards compiling your own distribution
> >> of
> >>>> Spark, when you can use the prebuilt versions of Spark as well.
> >>>>
> >>>>
> >>>> I'd like to split that portion of the documentation into two sections,
> >> a
> >>>> build-from-scratch section and a use-prebuilt section.  The new
> outline
> >>>> would look something like this:
> >>>>
> >>>>
> >>>> *Running Spark on Mesos*
> >>>>
> >>>> Installing Mesos
> >>>> - using prebuilt (recommended)
> >>>> - pointer to mesosphere's packages
> >>>> - from scratch
> >>>> - (similar to current)
> >>>>
> >>>>
> >>>> Connecting Spark to Mesos
> >>>> - loading distribution into an accessible location
> >>>> - Spark settings
> >>>>
> >>>> Mesos Run Modes
> >>>> - (same as current)
> >>>>
> >>>> Running Alongside Hadoop
> >>>> - (trim this down)
> >>>>
> >>>>
> >>>>
> >>>> Does that work for people?
> >>>>
> >>>>
> >>>> Thanks!
> >>>> Andrew
> >>>>
> >>>>
> >>>> PS Basically all the same:
> >>>>
> >>>> http://spark.apache.org/docs/0.6.0/running-on-mesos.html
> >>>> http://spark.apache.org/docs/0.6.2/running-on-mesos.html
> >>>> http://spark.apache.org/docs/0.7.3/running-on-mesos.html
> >>>> http://spark.apache.org/docs/0.8.1/running-on-mesos.html
> >>>> http://spark.apache.org/docs/0.9.1/running-on-mesos.html
> >>>>
> >>>
> >>
> https://people.apache.org/~pwendell/spark-1.0.0-rc3-docs/running-on-mesos.html
> >>>
> >>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updating docs for running on Mesos

Gerard Maas
In reply to this post by Andrew Ash
Great work!. I just left some comments in the PR. In summary, it would be
great to have more background on how Spark works on Mesos and how the
different elements interact. That will (hopefully) help understanding the
practicalities of the common assembly location (http/hdfs) and how the jobs
are distributed to the Mesos infrastructure.

Also, adding a chapter on troubleshooting (where we have spent most of our
time lately :-) would be a welcome addition.  I'm not use I've figured it
out completely as to attempt to contribute that myself.

-kr, Gerard.


On Tue, May 13, 2014 at 6:56 AM, Andrew Ash <[hidden email]> wrote:

> For trimming the Running Alongside Hadoop section I mostly think there
> should be a separate Spark+HDFS section and have the CDH+HDP page be merged
> into that one, but I supposed that's a separate docs change.
>
>
> On Sun, May 11, 2014 at 4:28 PM, Andy Konwinski <[hidden email]
> >wrote:
>
> > Thanks for suggesting this and volunteering to do it.
> >
> > On May 11, 2014 3:32 AM, "Andrew Ash" <[hidden email]> wrote:
> > >
> > > The docs for how to run Spark on Mesos have changed very little since
> > > 0.6.0, but setting it up is much easier now than then.  Does it make
> > sense
> > > to revamp with the below changes?
> > >
> > >
> > > You no longer need to build mesos yourself as pre-built versions are
> > > available from Mesosphere: http://mesosphere.io/downloads/
> > >
> > > And the instructions guide you towards compiling your own distribution
> of
> > > Spark, when you can use the prebuilt versions of Spark as well.
> > >
> > >
> > > I'd like to split that portion of the documentation into two sections,
> a
> > > build-from-scratch section and a use-prebuilt section.  The new outline
> > > would look something like this:
> > >
> > >
> > > *Running Spark on Mesos*
> > >
> > > Installing Mesos
> > > - using prebuilt (recommended)
> > >  - pointer to mesosphere's packages
> > > - from scratch
> > >  - (similar to current)
> > >
> > >
> > > Connecting Spark to Mesos
> > > - loading distribution into an accessible location
> > > - Spark settings
> > >
> > > Mesos Run Modes
> > > - (same as current)
> > >
> > > Running Alongside Hadoop
> > > - (trim this down)
> >
> > What trimming do you have in mind here?
> >
> > >
> > >
> > >
> > > Does that work for people?
> > >
> > >
> > > Thanks!
> > > Andrew
> > >
> > >
> > > PS Basically all the same:
> > >
> > > http://spark.apache.org/docs/0.6.0/running-on-mesos.html
> > > http://spark.apache.org/docs/0.6.2/running-on-mesos.html
> > > http://spark.apache.org/docs/0.7.3/running-on-mesos.html
> > > http://spark.apache.org/docs/0.8.1/running-on-mesos.html
> > > http://spark.apache.org/docs/0.9.1/running-on-mesos.html
> > >
> >
> >
> https://people.apache.org/~pwendell/spark-1.0.0-rc3-docs/running-on-mesos.html
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updating docs for running on Mesos

Tim St Clair
In reply to this post by Matei Zaharia
Perhaps linking to a Mesos page, which then can list the various package incantations.

Cheers,
Tim

----- Original Message -----

> From: "Matei Zaharia" <[hidden email]>
> To: [hidden email]
> Sent: Tuesday, May 13, 2014 2:59:42 AM
> Subject: Re: Updating docs for running on Mesos
>
> I’ll ask the Mesos folks about this. Unfortunately it might be tough to link
> only to a company’s builds; but we can perhaps include them in addition to
> instructions for building Mesos from Apache.
>
> Matei
>
> On May 12, 2014, at 11:55 PM, Gerard Maas <[hidden email]> wrote:
>
> > Andrew,
> >
> > Mesosphere has binary releases here:
> > http://mesosphere.io/downloads/
> >
> > (Anecdote: I actually burned a CPU building Mesos from source. No kidding -
> > it was coming, as the laptop was crashing from time to time, but the mesos
> > build was that one drop too much)
> >
> > kr, Gerard.
> >
> >
> >
> > On Tue, May 13, 2014 at 6:57 AM, Andrew Ash <[hidden email]> wrote:
> >
> >> As far as I know, the upstream doesn't release binaries, only source code.
> >> The downloads page <https://mesos.apache.org/downloads/> for 0.18.0 only
> >> has a source tarball.  Is there a binary release somewhere from Mesos that
> >> I'm missing?
> >>
> >>
> >> On Sun, May 11, 2014 at 2:16 PM, Patrick Wendell <[hidden email]>
> >> wrote:
> >>
> >>> Andrew,
> >>>
> >>> Updating these docs would be great! I think this would be a welcome
> >> change.
> >>>
> >>> In terms of packaging, it would be good to mention the binaries
> >>> produced by the upstream project as well, in addition to Mesosphere.
> >>>
> >>> - Patrick
> >>>
> >>> On Thu, May 8, 2014 at 12:51 AM, Andrew Ash <[hidden email]>
> >> wrote:
> >>>> The docs for how to run Spark on Mesos have changed very little since
> >>>> 0.6.0, but setting it up is much easier now than then.  Does it make
> >>> sense
> >>>> to revamp with the below changes?
> >>>>
> >>>>
> >>>> You no longer need to build mesos yourself as pre-built versions are
> >>>> available from Mesosphere: http://mesosphere.io/downloads/
> >>>>
> >>>> And the instructions guide you towards compiling your own distribution
> >> of
> >>>> Spark, when you can use the prebuilt versions of Spark as well.
> >>>>
> >>>>
> >>>> I'd like to split that portion of the documentation into two sections,
> >> a
> >>>> build-from-scratch section and a use-prebuilt section.  The new outline
> >>>> would look something like this:
> >>>>
> >>>>
> >>>> *Running Spark on Mesos*
> >>>>
> >>>> Installing Mesos
> >>>> - using prebuilt (recommended)
> >>>> - pointer to mesosphere's packages
> >>>> - from scratch
> >>>> - (similar to current)
> >>>>
> >>>>
> >>>> Connecting Spark to Mesos
> >>>> - loading distribution into an accessible location
> >>>> - Spark settings
> >>>>
> >>>> Mesos Run Modes
> >>>> - (same as current)
> >>>>
> >>>> Running Alongside Hadoop
> >>>> - (trim this down)
> >>>>
> >>>>
> >>>>
> >>>> Does that work for people?
> >>>>
> >>>>
> >>>> Thanks!
> >>>> Andrew
> >>>>
> >>>>
> >>>> PS Basically all the same:
> >>>>
> >>>> http://spark.apache.org/docs/0.6.0/running-on-mesos.html
> >>>> http://spark.apache.org/docs/0.6.2/running-on-mesos.html
> >>>> http://spark.apache.org/docs/0.7.3/running-on-mesos.html
> >>>> http://spark.apache.org/docs/0.8.1/running-on-mesos.html
> >>>> http://spark.apache.org/docs/0.9.1/running-on-mesos.html
> >>>>
> >>>
> >> https://people.apache.org/~pwendell/spark-1.0.0-rc3-docs/running-on-mesos.html
> >>>
> >>
>
>

--
Cheers,
Tim
Freedom, Features, Friends, First -> Fedora
https://fedoraproject.org/wiki/SIGs/bigdata
Loading...