Build Changes for SBT Users

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Build Changes for SBT Users

Patrick Wendell
Hey All,

Due to an ASF requirement, we recently merged a patch which removes
the sbt jar from the build. This is necessary because we aren't
allowed to distributed binary artifacts with our source packages.

This means that instead of building Spark with "sbt/sbt XXX", you'll
need to have sbt yourself and just run "sbt XXX" from within the Spark
directory. This is similar to the maven build, where we expect users
already have maven installed.

You can download sbt at http://www.scala-sbt.org/. It's okay to just
download the most recent version of sbt, since sbt knows how to fetch
other versions of itself and will always use the one we specify in our
build file to compile spark.

- Patrick
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

Holden Karau
Could we ship a shell script which downloads the sbt jar if not present
(like for example https://github.com/holdenk/slashem/blob/master/sbt )?


On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <[hidden email]> wrote:

> Hey All,
>
> Due to an ASF requirement, we recently merged a patch which removes
> the sbt jar from the build. This is necessary because we aren't
> allowed to distributed binary artifacts with our source packages.
>
> This means that instead of building Spark with "sbt/sbt XXX", you'll
> need to have sbt yourself and just run "sbt XXX" from within the Spark
> directory. This is similar to the maven build, where we expect users
> already have maven installed.
>
> You can download sbt at http://www.scala-sbt.org/. It's okay to just
> download the most recent version of sbt, since sbt knows how to fetch
> other versions of itself and will always use the one we specify in our
> build file to compile spark.
>
> - Patrick
>



--
Cell : 425-233-8271
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

Andrew Ash
+1 on bundling a script similar to that one


On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]> wrote:

> Could we ship a shell script which downloads the sbt jar if not present
> (like for example https://github.com/holdenk/slashem/blob/master/sbt )?
>
>
> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <[hidden email]>
> wrote:
>
> > Hey All,
> >
> > Due to an ASF requirement, we recently merged a patch which removes
> > the sbt jar from the build. This is necessary because we aren't
> > allowed to distributed binary artifacts with our source packages.
> >
> > This means that instead of building Spark with "sbt/sbt XXX", you'll
> > need to have sbt yourself and just run "sbt XXX" from within the Spark
> > directory. This is similar to the maven build, where we expect users
> > already have maven installed.
> >
> > You can download sbt at http://www.scala-sbt.org/. It's okay to just
> > download the most recent version of sbt, since sbt knows how to fetch
> > other versions of itself and will always use the one we specify in our
> > build file to compile spark.
> >
> > - Patrick
> >
>
>
>
> --
> Cell : 425-233-8271
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

Patrick Wendell
We thought about this but elected not to do this for a few reasons.

1. Some people build from machines that do not have internet access
for security reasons and retrieve dependency from internal nexus
repositories. So having a build dependency that relies on internet
downloads is not desirable.

2. It's a hard to ensure stability of a particular URL in perpetuity.
This is why maven central and other mirror networks exist. Keep in
mind that we can't change the release code ever once we release it,
and if something changed about the particular URL it could break the
build.

- Patrick

On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <[hidden email]> wrote:

> +1 on bundling a script similar to that one
>
>
> On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]> wrote:
>
>> Could we ship a shell script which downloads the sbt jar if not present
>> (like for example https://github.com/holdenk/slashem/blob/master/sbt )?
>>
>>
>> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <[hidden email]>
>> wrote:
>>
>> > Hey All,
>> >
>> > Due to an ASF requirement, we recently merged a patch which removes
>> > the sbt jar from the build. This is necessary because we aren't
>> > allowed to distributed binary artifacts with our source packages.
>> >
>> > This means that instead of building Spark with "sbt/sbt XXX", you'll
>> > need to have sbt yourself and just run "sbt XXX" from within the Spark
>> > directory. This is similar to the maven build, where we expect users
>> > already have maven installed.
>> >
>> > You can download sbt at http://www.scala-sbt.org/. It's okay to just
>> > download the most recent version of sbt, since sbt knows how to fetch
>> > other versions of itself and will always use the one we specify in our
>> > build file to compile spark.
>> >
>> > - Patrick
>> >
>>
>>
>>
>> --
>> Cell : 425-233-8271
>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

Holden Karau
That makes sense, I think we could structure a script in such a way that it
would overcome these problems though and probably provide a fair a mount of
benefit for people who just want to get started quickly.

The easiest would be to have it use the system sbt if present and then fall
back to downloading the sbt jar. As far as stability of the URL goes we
could solve this by either having it point at a domain we control, or just
with an clear error message indicating it failed to download sbt and the
user needs to install sbt.

If a restructured script in that manner would be useful I could whip up a
pull request :)


On Sat, Jan 4, 2014 at 10:56 AM, Patrick Wendell <[hidden email]> wrote:

> We thought about this but elected not to do this for a few reasons.
>
> 1. Some people build from machines that do not have internet access
> for security reasons and retrieve dependency from internal nexus
> repositories. So having a build dependency that relies on internet
> downloads is not desirable.
>
> 2. It's a hard to ensure stability of a particular URL in perpetuity.
> This is why maven central and other mirror networks exist. Keep in
> mind that we can't change the release code ever once we release it,
> and if something changed about the particular URL it could break the
> build.
>
> - Patrick
>
> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <[hidden email]> wrote:
> > +1 on bundling a script similar to that one
> >
> >
> > On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]>
> wrote:
> >
> >> Could we ship a shell script which downloads the sbt jar if not present
> >> (like for example https://github.com/holdenk/slashem/blob/master/sbt )?
> >>
> >>
> >> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <[hidden email]>
> >> wrote:
> >>
> >> > Hey All,
> >> >
> >> > Due to an ASF requirement, we recently merged a patch which removes
> >> > the sbt jar from the build. This is necessary because we aren't
> >> > allowed to distributed binary artifacts with our source packages.
> >> >
> >> > This means that instead of building Spark with "sbt/sbt XXX", you'll
> >> > need to have sbt yourself and just run "sbt XXX" from within the Spark
> >> > directory. This is similar to the maven build, where we expect users
> >> > already have maven installed.
> >> >
> >> > You can download sbt at http://www.scala-sbt.org/. It's okay to just
> >> > download the most recent version of sbt, since sbt knows how to fetch
> >> > other versions of itself and will always use the one we specify in our
> >> > build file to compile spark.
> >> >
> >> > - Patrick
> >> >
> >>
> >>
> >>
> >> --
> >> Cell : 425-233-8271
> >>
>



--
Cell : 425-233-8271
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

Jey Kottalam
I'm in full agreement with Holden. We should provide a smooth out of the
box experience while also not getting in the way of those who provide their
own sbt installs.

On Saturday, January 4, 2014, Holden Karau wrote:

> That makes sense, I think we could structure a script in such a way that it
> would overcome these problems though and probably provide a fair a mount of
> benefit for people who just want to get started quickly.
>
> The easiest would be to have it use the system sbt if present and then fall
> back to downloading the sbt jar. As far as stability of the URL goes we
> could solve this by either having it point at a domain we control, or just
> with an clear error message indicating it failed to download sbt and the
> user needs to install sbt.
>
> If a restructured script in that manner would be useful I could whip up a
> pull request :)
>
>
> On Sat, Jan 4, 2014 at 10:56 AM, Patrick Wendell <[hidden email]<javascript:;>>
> wrote:
>
> > We thought about this but elected not to do this for a few reasons.
> >
> > 1. Some people build from machines that do not have internet access
> > for security reasons and retrieve dependency from internal nexus
> > repositories. So having a build dependency that relies on internet
> > downloads is not desirable.
> >
> > 2. It's a hard to ensure stability of a particular URL in perpetuity.
> > This is why maven central and other mirror networks exist. Keep in
> > mind that we can't change the release code ever once we release it,
> > and if something changed about the particular URL it could break the
> > build.
> >
> > - Patrick
> >
> > On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <[hidden email]<javascript:;>>
> wrote:
> > > +1 on bundling a script similar to that one
> > >
> > >
> > > On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]<javascript:;>
> >
> > wrote:
> > >
> > >> Could we ship a shell script which downloads the sbt jar if not
> present
> > >> (like for example https://github.com/holdenk/slashem/blob/master/sbt)?
> > >>
> > >>
> > >> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <[hidden email]<javascript:;>
> >
> > >> wrote:
> > >>
> > >> > Hey All,
> > >> >
> > >> > Due to an ASF requirement, we recently merged a patch which removes
> > >> > the sbt jar from the build. This is necessary because we aren't
> > >> > allowed to distributed binary artifacts with our source packages.
> > >> >
> > >> > This means that instead of building Spark with "sbt/sbt XXX", you'll
> > >> > need to have sbt yourself and just run "sbt XXX" from within the
> Spark
> > >> > directory. This is similar to the maven build, where we expect users
> > >> > already have maven installed.
> > >> >
> > >> > You can download sbt at http://www.scala-sbt.org/. It's okay to
> just
> > >> > download the most recent version of sbt, since sbt knows how to
> fetch
> > >> > other versions of itself and will always use the one we specify in
> our
> > >> > build file to compile spark.
> > >> >
> > >> > - Patrick
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Cell : 425-233-8271
> > >>
> >
>
>
>
> --
> Cell : 425-233-8271
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

Patrick Wendell
In reply to this post by Holden Karau
Hey Holden,

That sounds reasonable to me. Where would we get a url we can control
though? Right now the project has web space is at incubator.apache...
but later this will change to a full apache domain. Is there somewhere
in maven central these jars are hosted... that would be the nicest
because things like repo1.maven.org basically never changes.

- Patrick

On Sat, Jan 4, 2014 at 1:20 PM, Holden Karau <[hidden email]> wrote:

> That makes sense, I think we could structure a script in such a way that it
> would overcome these problems though and probably provide a fair a mount of
> benefit for people who just want to get started quickly.
>
> The easiest would be to have it use the system sbt if present and then fall
> back to downloading the sbt jar. As far as stability of the URL goes we
> could solve this by either having it point at a domain we control, or just
> with an clear error message indicating it failed to download sbt and the
> user needs to install sbt.
>
> If a restructured script in that manner would be useful I could whip up a
> pull request :)
>
>
> On Sat, Jan 4, 2014 at 10:56 AM, Patrick Wendell <[hidden email]> wrote:
>
>> We thought about this but elected not to do this for a few reasons.
>>
>> 1. Some people build from machines that do not have internet access
>> for security reasons and retrieve dependency from internal nexus
>> repositories. So having a build dependency that relies on internet
>> downloads is not desirable.
>>
>> 2. It's a hard to ensure stability of a particular URL in perpetuity.
>> This is why maven central and other mirror networks exist. Keep in
>> mind that we can't change the release code ever once we release it,
>> and if something changed about the particular URL it could break the
>> build.
>>
>> - Patrick
>>
>> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <[hidden email]> wrote:
>> > +1 on bundling a script similar to that one
>> >
>> >
>> > On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]>
>> wrote:
>> >
>> >> Could we ship a shell script which downloads the sbt jar if not present
>> >> (like for example https://github.com/holdenk/slashem/blob/master/sbt )?
>> >>
>> >>
>> >> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <[hidden email]>
>> >> wrote:
>> >>
>> >> > Hey All,
>> >> >
>> >> > Due to an ASF requirement, we recently merged a patch which removes
>> >> > the sbt jar from the build. This is necessary because we aren't
>> >> > allowed to distributed binary artifacts with our source packages.
>> >> >
>> >> > This means that instead of building Spark with "sbt/sbt XXX", you'll
>> >> > need to have sbt yourself and just run "sbt XXX" from within the Spark
>> >> > directory. This is similar to the maven build, where we expect users
>> >> > already have maven installed.
>> >> >
>> >> > You can download sbt at http://www.scala-sbt.org/. It's okay to just
>> >> > download the most recent version of sbt, since sbt knows how to fetch
>> >> > other versions of itself and will always use the one we specify in our
>> >> > build file to compile spark.
>> >> >
>> >> > - Patrick
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Cell : 425-233-8271
>> >>
>>
>
>
>
> --
> Cell : 425-233-8271
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

rxin
Doesn't Apache do redirection from incubation. to the normal website also?
 By the time that happens, we can also update the URL in the script?


On Sat, Jan 4, 2014 at 4:13 PM, Patrick Wendell <[hidden email]> wrote:

> Hey Holden,
>
> That sounds reasonable to me. Where would we get a url we can control
> though? Right now the project has web space is at incubator.apache...
> but later this will change to a full apache domain. Is there somewhere
> in maven central these jars are hosted... that would be the nicest
> because things like repo1.maven.org basically never changes.
>
> - Patrick
>
> On Sat, Jan 4, 2014 at 1:20 PM, Holden Karau <[hidden email]> wrote:
> > That makes sense, I think we could structure a script in such a way that
> it
> > would overcome these problems though and probably provide a fair a mount
> of
> > benefit for people who just want to get started quickly.
> >
> > The easiest would be to have it use the system sbt if present and then
> fall
> > back to downloading the sbt jar. As far as stability of the URL goes we
> > could solve this by either having it point at a domain we control, or
> just
> > with an clear error message indicating it failed to download sbt and the
> > user needs to install sbt.
> >
> > If a restructured script in that manner would be useful I could whip up a
> > pull request :)
> >
> >
> > On Sat, Jan 4, 2014 at 10:56 AM, Patrick Wendell <[hidden email]>
> wrote:
> >
> >> We thought about this but elected not to do this for a few reasons.
> >>
> >> 1. Some people build from machines that do not have internet access
> >> for security reasons and retrieve dependency from internal nexus
> >> repositories. So having a build dependency that relies on internet
> >> downloads is not desirable.
> >>
> >> 2. It's a hard to ensure stability of a particular URL in perpetuity.
> >> This is why maven central and other mirror networks exist. Keep in
> >> mind that we can't change the release code ever once we release it,
> >> and if something changed about the particular URL it could break the
> >> build.
> >>
> >> - Patrick
> >>
> >> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <[hidden email]>
> wrote:
> >> > +1 on bundling a script similar to that one
> >> >
> >> >
> >> > On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]>
> >> wrote:
> >> >
> >> >> Could we ship a shell script which downloads the sbt jar if not
> present
> >> >> (like for example https://github.com/holdenk/slashem/blob/master/sbt)?
> >> >>
> >> >>
> >> >> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <[hidden email]
> >
> >> >> wrote:
> >> >>
> >> >> > Hey All,
> >> >> >
> >> >> > Due to an ASF requirement, we recently merged a patch which removes
> >> >> > the sbt jar from the build. This is necessary because we aren't
> >> >> > allowed to distributed binary artifacts with our source packages.
> >> >> >
> >> >> > This means that instead of building Spark with "sbt/sbt XXX",
> you'll
> >> >> > need to have sbt yourself and just run "sbt XXX" from within the
> Spark
> >> >> > directory. This is similar to the maven build, where we expect
> users
> >> >> > already have maven installed.
> >> >> >
> >> >> > You can download sbt at http://www.scala-sbt.org/. It's okay to
> just
> >> >> > download the most recent version of sbt, since sbt knows how to
> fetch
> >> >> > other versions of itself and will always use the one we specify in
> our
> >> >> > build file to compile spark.
> >> >> >
> >> >> > - Patrick
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Cell : 425-233-8271
> >> >>
> >>
> >
> >
> >
> > --
> > Cell : 425-233-8271
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

Xuefeng Wu
In reply to this post by Patrick Wendell
Sound reasonable.  But I think few installed sbt even it is easy to install.  I think can provide this tricky script in online document, user could download this script to install sbt independence. Sound like a yet another brew install sbt?
:)

Yours, Xuefeng Wu 吴雪峰 敬上

> On 2014年1月5日, at 上午2:56, Patrick Wendell <[hidden email]> wrote:
>
> We thought about this but elected not to do this for a few reasons.
>
> 1. Some people build from machines that do not have internet access
> for security reasons and retrieve dependency from internal nexus
> repositories. So having a build dependency that relies on internet
> downloads is not desirable.
>
> 2. It's a hard to ensure stability of a particular URL in perpetuity.
> This is why maven central and other mirror networks exist. Keep in
> mind that we can't change the release code ever once we release it,
> and if something changed about the particular URL it could break the
> build.
>
> - Patrick
>
>> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <[hidden email]> wrote:
>> +1 on bundling a script similar to that one
>>
>>
>>> On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]> wrote:
>>>
>>> Could we ship a shell script which downloads the sbt jar if not present
>>> (like for example https://github.com/holdenk/slashem/blob/master/sbt )?
>>>
>>>
>>> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <[hidden email]>
>>> wrote:
>>>
>>>> Hey All,
>>>>
>>>> Due to an ASF requirement, we recently merged a patch which removes
>>>> the sbt jar from the build. This is necessary because we aren't
>>>> allowed to distributed binary artifacts with our source packages.
>>>>
>>>> This means that instead of building Spark with "sbt/sbt XXX", you'll
>>>> need to have sbt yourself and just run "sbt XXX" from within the Spark
>>>> directory. This is similar to the maven build, where we expect users
>>>> already have maven installed.
>>>>
>>>> You can download sbt at http://www.scala-sbt.org/. It's okay to just
>>>> download the most recent version of sbt, since sbt knows how to fetch
>>>> other versions of itself and will always use the one we specify in our
>>>> build file to compile spark.
>>>>
>>>> - Patrick
>>>
>>>
>>>
>>> --
>>> Cell : 425-233-8271
>>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

Patrick Wendell
Reynold the issue is releases are immutable and we expect them to be
downloaded for several years after the release date.

On Sat, Jan 4, 2014 at 5:57 PM, Xuefeng Wu <[hidden email]> wrote:

> Sound reasonable.  But I think few installed sbt even it is easy to install.  I think can provide this tricky script in online document, user could download this script to install sbt independence. Sound like a yet another brew install sbt?
> :)
>
> Yours, Xuefeng Wu 吴雪峰 敬上
>
>> On 2014年1月5日, at 上午2:56, Patrick Wendell <[hidden email]> wrote:
>>
>> We thought about this but elected not to do this for a few reasons.
>>
>> 1. Some people build from machines that do not have internet access
>> for security reasons and retrieve dependency from internal nexus
>> repositories. So having a build dependency that relies on internet
>> downloads is not desirable.
>>
>> 2. It's a hard to ensure stability of a particular URL in perpetuity.
>> This is why maven central and other mirror networks exist. Keep in
>> mind that we can't change the release code ever once we release it,
>> and if something changed about the particular URL it could break the
>> build.
>>
>> - Patrick
>>
>>> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <[hidden email]> wrote:
>>> +1 on bundling a script similar to that one
>>>
>>>
>>>> On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]> wrote:
>>>>
>>>> Could we ship a shell script which downloads the sbt jar if not present
>>>> (like for example https://github.com/holdenk/slashem/blob/master/sbt )?
>>>>
>>>>
>>>> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <[hidden email]>
>>>> wrote:
>>>>
>>>>> Hey All,
>>>>>
>>>>> Due to an ASF requirement, we recently merged a patch which removes
>>>>> the sbt jar from the build. This is necessary because we aren't
>>>>> allowed to distributed binary artifacts with our source packages.
>>>>>
>>>>> This means that instead of building Spark with "sbt/sbt XXX", you'll
>>>>> need to have sbt yourself and just run "sbt XXX" from within the Spark
>>>>> directory. This is similar to the maven build, where we expect users
>>>>> already have maven installed.
>>>>>
>>>>> You can download sbt at http://www.scala-sbt.org/. It's okay to just
>>>>> download the most recent version of sbt, since sbt knows how to fetch
>>>>> other versions of itself and will always use the one we specify in our
>>>>> build file to compile spark.
>>>>>
>>>>> - Patrick
>>>>
>>>>
>>>>
>>>> --
>>>> Cell : 425-233-8271
>>>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

Tathagata Das
Patrick, that is right. All we are trying to ensure is to make a
"best-effort" attempt to make it smooth for a new user. The script will try
its best to automatically install / download sbt for the user. The fallback
will be that the user will have to install sbt on their own. If the URL
happens to change and our script fails to automatically download, then we
are *no worse* than not providing the script at all.

TD


On Sat, Jan 4, 2014 at 7:06 PM, Patrick Wendell <[hidden email]> wrote:

> Reynold the issue is releases are immutable and we expect them to be
> downloaded for several years after the release date.
>
> On Sat, Jan 4, 2014 at 5:57 PM, Xuefeng Wu <[hidden email]> wrote:
> > Sound reasonable.  But I think few installed sbt even it is easy to
> install.  I think can provide this tricky script in online document, user
> could download this script to install sbt independence. Sound like a yet
> another brew install sbt?
> > :)
> >
> > Yours, Xuefeng Wu 吴雪峰 敬上
> >
> >> On 2014年1月5日, at 上午2:56, Patrick Wendell <[hidden email]> wrote:
> >>
> >> We thought about this but elected not to do this for a few reasons.
> >>
> >> 1. Some people build from machines that do not have internet access
> >> for security reasons and retrieve dependency from internal nexus
> >> repositories. So having a build dependency that relies on internet
> >> downloads is not desirable.
> >>
> >> 2. It's a hard to ensure stability of a particular URL in perpetuity.
> >> This is why maven central and other mirror networks exist. Keep in
> >> mind that we can't change the release code ever once we release it,
> >> and if something changed about the particular URL it could break the
> >> build.
> >>
> >> - Patrick
> >>
> >>> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <[hidden email]>
> wrote:
> >>> +1 on bundling a script similar to that one
> >>>
> >>>
> >>>> On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]>
> wrote:
> >>>>
> >>>> Could we ship a shell script which downloads the sbt jar if not
> present
> >>>> (like for example https://github.com/holdenk/slashem/blob/master/sbt)?
> >>>>
> >>>>
> >>>> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <[hidden email]>
> >>>> wrote:
> >>>>
> >>>>> Hey All,
> >>>>>
> >>>>> Due to an ASF requirement, we recently merged a patch which removes
> >>>>> the sbt jar from the build. This is necessary because we aren't
> >>>>> allowed to distributed binary artifacts with our source packages.
> >>>>>
> >>>>> This means that instead of building Spark with "sbt/sbt XXX", you'll
> >>>>> need to have sbt yourself and just run "sbt XXX" from within the
> Spark
> >>>>> directory. This is similar to the maven build, where we expect users
> >>>>> already have maven installed.
> >>>>>
> >>>>> You can download sbt at http://www.scala-sbt.org/. It's okay to just
> >>>>> download the most recent version of sbt, since sbt knows how to fetch
> >>>>> other versions of itself and will always use the one we specify in
> our
> >>>>> build file to compile spark.
> >>>>>
> >>>>> - Patrick
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Cell : 425-233-8271
> >>>>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

Patrick Wendell
I agree TD - I was just saying that Reynold's proposal that we could
update the release post-hoc is unfortunately not possible.

On Sat, Jan 4, 2014 at 7:13 PM, Tathagata Das
<[hidden email]> wrote:

> Patrick, that is right. All we are trying to ensure is to make a
> "best-effort" attempt to make it smooth for a new user. The script will try
> its best to automatically install / download sbt for the user. The fallback
> will be that the user will have to install sbt on their own. If the URL
> happens to change and our script fails to automatically download, then we
> are *no worse* than not providing the script at all.
>
> TD
>
>
> On Sat, Jan 4, 2014 at 7:06 PM, Patrick Wendell <[hidden email]> wrote:
>
>> Reynold the issue is releases are immutable and we expect them to be
>> downloaded for several years after the release date.
>>
>> On Sat, Jan 4, 2014 at 5:57 PM, Xuefeng Wu <[hidden email]> wrote:
>> > Sound reasonable.  But I think few installed sbt even it is easy to
>> install.  I think can provide this tricky script in online document, user
>> could download this script to install sbt independence. Sound like a yet
>> another brew install sbt?
>> > :)
>> >
>> > Yours, Xuefeng Wu 吴雪峰 敬上
>> >
>> >> On 2014年1月5日, at 上午2:56, Patrick Wendell <[hidden email]> wrote:
>> >>
>> >> We thought about this but elected not to do this for a few reasons.
>> >>
>> >> 1. Some people build from machines that do not have internet access
>> >> for security reasons and retrieve dependency from internal nexus
>> >> repositories. So having a build dependency that relies on internet
>> >> downloads is not desirable.
>> >>
>> >> 2. It's a hard to ensure stability of a particular URL in perpetuity.
>> >> This is why maven central and other mirror networks exist. Keep in
>> >> mind that we can't change the release code ever once we release it,
>> >> and if something changed about the particular URL it could break the
>> >> build.
>> >>
>> >> - Patrick
>> >>
>> >>> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <[hidden email]>
>> wrote:
>> >>> +1 on bundling a script similar to that one
>> >>>
>> >>>
>> >>>> On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]>
>> wrote:
>> >>>>
>> >>>> Could we ship a shell script which downloads the sbt jar if not
>> present
>> >>>> (like for example https://github.com/holdenk/slashem/blob/master/sbt)?
>> >>>>
>> >>>>
>> >>>> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <[hidden email]>
>> >>>> wrote:
>> >>>>
>> >>>>> Hey All,
>> >>>>>
>> >>>>> Due to an ASF requirement, we recently merged a patch which removes
>> >>>>> the sbt jar from the build. This is necessary because we aren't
>> >>>>> allowed to distributed binary artifacts with our source packages.
>> >>>>>
>> >>>>> This means that instead of building Spark with "sbt/sbt XXX", you'll
>> >>>>> need to have sbt yourself and just run "sbt XXX" from within the
>> Spark
>> >>>>> directory. This is similar to the maven build, where we expect users
>> >>>>> already have maven installed.
>> >>>>>
>> >>>>> You can download sbt at http://www.scala-sbt.org/. It's okay to just
>> >>>>> download the most recent version of sbt, since sbt knows how to fetch
>> >>>>> other versions of itself and will always use the one we specify in
>> our
>> >>>>> build file to compile spark.
>> >>>>>
>> >>>>> - Patrick
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Cell : 425-233-8271
>> >>>>
>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

Holden Karau
So I've put together a pull request which depends on the typesafe URLs
(they have been stable for a long time)
https://github.com/apache/incubator-spark/pull/331 and uses the system sbt
if it is present. How do people feel about this?


On Sat, Jan 4, 2014 at 9:07 PM, Patrick Wendell <[hidden email]> wrote:

> I agree TD - I was just saying that Reynold's proposal that we could
> update the release post-hoc is unfortunately not possible.
>
> On Sat, Jan 4, 2014 at 7:13 PM, Tathagata Das
> <[hidden email]> wrote:
> > Patrick, that is right. All we are trying to ensure is to make a
> > "best-effort" attempt to make it smooth for a new user. The script will
> try
> > its best to automatically install / download sbt for the user. The
> fallback
> > will be that the user will have to install sbt on their own. If the URL
> > happens to change and our script fails to automatically download, then we
> > are *no worse* than not providing the script at all.
> >
> > TD
> >
> >
> > On Sat, Jan 4, 2014 at 7:06 PM, Patrick Wendell <[hidden email]>
> wrote:
> >
> >> Reynold the issue is releases are immutable and we expect them to be
> >> downloaded for several years after the release date.
> >>
> >> On Sat, Jan 4, 2014 at 5:57 PM, Xuefeng Wu <[hidden email]> wrote:
> >> > Sound reasonable.  But I think few installed sbt even it is easy to
> >> install.  I think can provide this tricky script in online document,
> user
> >> could download this script to install sbt independence. Sound like a yet
> >> another brew install sbt?
> >> > :)
> >> >
> >> > Yours, Xuefeng Wu 吴雪峰 敬上
> >> >
> >> >> On 2014年1月5日, at 上午2:56, Patrick Wendell <[hidden email]> wrote:
> >> >>
> >> >> We thought about this but elected not to do this for a few reasons.
> >> >>
> >> >> 1. Some people build from machines that do not have internet access
> >> >> for security reasons and retrieve dependency from internal nexus
> >> >> repositories. So having a build dependency that relies on internet
> >> >> downloads is not desirable.
> >> >>
> >> >> 2. It's a hard to ensure stability of a particular URL in perpetuity.
> >> >> This is why maven central and other mirror networks exist. Keep in
> >> >> mind that we can't change the release code ever once we release it,
> >> >> and if something changed about the particular URL it could break the
> >> >> build.
> >> >>
> >> >> - Patrick
> >> >>
> >> >>> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <[hidden email]>
> >> wrote:
> >> >>> +1 on bundling a script similar to that one
> >> >>>
> >> >>>
> >> >>>> On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]
> >
> >> wrote:
> >> >>>>
> >> >>>> Could we ship a shell script which downloads the sbt jar if not
> >> present
> >> >>>> (like for example
> https://github.com/holdenk/slashem/blob/master/sbt)?
> >> >>>>
> >> >>>>
> >> >>>> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <
> [hidden email]>
> >> >>>> wrote:
> >> >>>>
> >> >>>>> Hey All,
> >> >>>>>
> >> >>>>> Due to an ASF requirement, we recently merged a patch which
> removes
> >> >>>>> the sbt jar from the build. This is necessary because we aren't
> >> >>>>> allowed to distributed binary artifacts with our source packages.
> >> >>>>>
> >> >>>>> This means that instead of building Spark with "sbt/sbt XXX",
> you'll
> >> >>>>> need to have sbt yourself and just run "sbt XXX" from within the
> >> Spark
> >> >>>>> directory. This is similar to the maven build, where we expect
> users
> >> >>>>> already have maven installed.
> >> >>>>>
> >> >>>>> You can download sbt at http://www.scala-sbt.org/. It's okay to
> just
> >> >>>>> download the most recent version of sbt, since sbt knows how to
> fetch
> >> >>>>> other versions of itself and will always use the one we specify in
> >> our
> >> >>>>> build file to compile spark.
> >> >>>>>
> >> >>>>> - Patrick
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Cell : 425-233-8271
> >> >>>>
> >>
>



--
Cell : 425-233-8271
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

rxin
In reply to this post by Patrick Wendell
Why is it not possible? You always update the script; just can't update
scripts for released versions.




On Sat, Jan 4, 2014 at 9:07 PM, Patrick Wendell <[hidden email]> wrote:

> I agree TD - I was just saying that Reynold's proposal that we could
> update the release post-hoc is unfortunately not possible.
>
> On Sat, Jan 4, 2014 at 7:13 PM, Tathagata Das
> <[hidden email]> wrote:
> > Patrick, that is right. All we are trying to ensure is to make a
> > "best-effort" attempt to make it smooth for a new user. The script will
> try
> > its best to automatically install / download sbt for the user. The
> fallback
> > will be that the user will have to install sbt on their own. If the URL
> > happens to change and our script fails to automatically download, then we
> > are *no worse* than not providing the script at all.
> >
> > TD
> >
> >
> > On Sat, Jan 4, 2014 at 7:06 PM, Patrick Wendell <[hidden email]>
> wrote:
> >
> >> Reynold the issue is releases are immutable and we expect them to be
> >> downloaded for several years after the release date.
> >>
> >> On Sat, Jan 4, 2014 at 5:57 PM, Xuefeng Wu <[hidden email]> wrote:
> >> > Sound reasonable.  But I think few installed sbt even it is easy to
> >> install.  I think can provide this tricky script in online document,
> user
> >> could download this script to install sbt independence. Sound like a yet
> >> another brew install sbt?
> >> > :)
> >> >
> >> > Yours, Xuefeng Wu 吴雪峰 敬上
> >> >
> >> >> On 2014年1月5日, at 上午2:56, Patrick Wendell <[hidden email]> wrote:
> >> >>
> >> >> We thought about this but elected not to do this for a few reasons.
> >> >>
> >> >> 1. Some people build from machines that do not have internet access
> >> >> for security reasons and retrieve dependency from internal nexus
> >> >> repositories. So having a build dependency that relies on internet
> >> >> downloads is not desirable.
> >> >>
> >> >> 2. It's a hard to ensure stability of a particular URL in perpetuity.
> >> >> This is why maven central and other mirror networks exist. Keep in
> >> >> mind that we can't change the release code ever once we release it,
> >> >> and if something changed about the particular URL it could break the
> >> >> build.
> >> >>
> >> >> - Patrick
> >> >>
> >> >>> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <[hidden email]>
> >> wrote:
> >> >>> +1 on bundling a script similar to that one
> >> >>>
> >> >>>
> >> >>>> On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]
> >
> >> wrote:
> >> >>>>
> >> >>>> Could we ship a shell script which downloads the sbt jar if not
> >> present
> >> >>>> (like for example
> https://github.com/holdenk/slashem/blob/master/sbt)?
> >> >>>>
> >> >>>>
> >> >>>> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <
> [hidden email]>
> >> >>>> wrote:
> >> >>>>
> >> >>>>> Hey All,
> >> >>>>>
> >> >>>>> Due to an ASF requirement, we recently merged a patch which
> removes
> >> >>>>> the sbt jar from the build. This is necessary because we aren't
> >> >>>>> allowed to distributed binary artifacts with our source packages.
> >> >>>>>
> >> >>>>> This means that instead of building Spark with "sbt/sbt XXX",
> you'll
> >> >>>>> need to have sbt yourself and just run "sbt XXX" from within the
> >> Spark
> >> >>>>> directory. This is similar to the maven build, where we expect
> users
> >> >>>>> already have maven installed.
> >> >>>>>
> >> >>>>> You can download sbt at http://www.scala-sbt.org/. It's okay to
> just
> >> >>>>> download the most recent version of sbt, since sbt knows how to
> fetch
> >> >>>>> other versions of itself and will always use the one we specify in
> >> our
> >> >>>>> build file to compile spark.
> >> >>>>>
> >> >>>>> - Patrick
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Cell : 425-233-8271
> >> >>>>
> >>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Build Changes for SBT Users

Patrick Wendell
Ya I was referring to already released version. Of course we can
update for subsequent releases...

On Sun, Jan 5, 2014 at 4:24 PM, Reynold Xin <[hidden email]> wrote:

> Why is it not possible? You always update the script; just can't update
> scripts for released versions.
>
>
>
>
> On Sat, Jan 4, 2014 at 9:07 PM, Patrick Wendell <[hidden email]> wrote:
>
>> I agree TD - I was just saying that Reynold's proposal that we could
>> update the release post-hoc is unfortunately not possible.
>>
>> On Sat, Jan 4, 2014 at 7:13 PM, Tathagata Das
>> <[hidden email]> wrote:
>> > Patrick, that is right. All we are trying to ensure is to make a
>> > "best-effort" attempt to make it smooth for a new user. The script will
>> try
>> > its best to automatically install / download sbt for the user. The
>> fallback
>> > will be that the user will have to install sbt on their own. If the URL
>> > happens to change and our script fails to automatically download, then we
>> > are *no worse* than not providing the script at all.
>> >
>> > TD
>> >
>> >
>> > On Sat, Jan 4, 2014 at 7:06 PM, Patrick Wendell <[hidden email]>
>> wrote:
>> >
>> >> Reynold the issue is releases are immutable and we expect them to be
>> >> downloaded for several years after the release date.
>> >>
>> >> On Sat, Jan 4, 2014 at 5:57 PM, Xuefeng Wu <[hidden email]> wrote:
>> >> > Sound reasonable.  But I think few installed sbt even it is easy to
>> >> install.  I think can provide this tricky script in online document,
>> user
>> >> could download this script to install sbt independence. Sound like a yet
>> >> another brew install sbt?
>> >> > :)
>> >> >
>> >> > Yours, Xuefeng Wu 吴雪峰 敬上
>> >> >
>> >> >> On 2014年1月5日, at 上午2:56, Patrick Wendell <[hidden email]> wrote:
>> >> >>
>> >> >> We thought about this but elected not to do this for a few reasons.
>> >> >>
>> >> >> 1. Some people build from machines that do not have internet access
>> >> >> for security reasons and retrieve dependency from internal nexus
>> >> >> repositories. So having a build dependency that relies on internet
>> >> >> downloads is not desirable.
>> >> >>
>> >> >> 2. It's a hard to ensure stability of a particular URL in perpetuity.
>> >> >> This is why maven central and other mirror networks exist. Keep in
>> >> >> mind that we can't change the release code ever once we release it,
>> >> >> and if something changed about the particular URL it could break the
>> >> >> build.
>> >> >>
>> >> >> - Patrick
>> >> >>
>> >> >>> On Sat, Jan 4, 2014 at 9:34 AM, Andrew Ash <[hidden email]>
>> >> wrote:
>> >> >>> +1 on bundling a script similar to that one
>> >> >>>
>> >> >>>
>> >> >>>> On Sat, Jan 4, 2014 at 4:48 AM, Holden Karau <[hidden email]
>> >
>> >> wrote:
>> >> >>>>
>> >> >>>> Could we ship a shell script which downloads the sbt jar if not
>> >> present
>> >> >>>> (like for example
>> https://github.com/holdenk/slashem/blob/master/sbt)?
>> >> >>>>
>> >> >>>>
>> >> >>>> On Sat, Jan 4, 2014 at 12:02 AM, Patrick Wendell <
>> [hidden email]>
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>>> Hey All,
>> >> >>>>>
>> >> >>>>> Due to an ASF requirement, we recently merged a patch which
>> removes
>> >> >>>>> the sbt jar from the build. This is necessary because we aren't
>> >> >>>>> allowed to distributed binary artifacts with our source packages.
>> >> >>>>>
>> >> >>>>> This means that instead of building Spark with "sbt/sbt XXX",
>> you'll
>> >> >>>>> need to have sbt yourself and just run "sbt XXX" from within the
>> >> Spark
>> >> >>>>> directory. This is similar to the maven build, where we expect
>> users
>> >> >>>>> already have maven installed.
>> >> >>>>>
>> >> >>>>> You can download sbt at http://www.scala-sbt.org/. It's okay to
>> just
>> >> >>>>> download the most recent version of sbt, since sbt knows how to
>> fetch
>> >> >>>>> other versions of itself and will always use the one we specify in
>> >> our
>> >> >>>>> build file to compile spark.
>> >> >>>>>
>> >> >>>>> - Patrick
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> --
>> >> >>>> Cell : 425-233-8271
>> >> >>>>
>> >>
>>
Loading...