[VOTE] SPARK 2.4.0 (RC4)

classic Classic list List threaded Threaded
49 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: What if anything to fix about k8s for the 2.4.0 RC5?

Sean Owen-2
Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :)

On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos
<[hidden email]> wrote:

>
> I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right?
>
> For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 .
>
>
> On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos <[hidden email]> wrote:
>>
>> I will open a jira for the profile propagation issue and have a look to fix it.
>>
>> Stavros
>>
>> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson <[hidden email]> wrote:
>>>
>>>
>>> I would be comfortable making the integration testing manual for now.  A JIRA for ironing out how to make it reliable for automatic as a goal for 3.0 seems like a good idea.
>>>
>>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen <[hidden email]> wrote:
>>>>
>>>> Forking this thread.
>>>>
>>>> Because we'll have another RC, we could possibly address these two
>>>> issues. Only if we have a reliable change of course.
>>>>
>>>> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt.
>>>>
>>>> And is it reasonable to essentially 'disable'
>>>> kubernetes/integration-tests by removing it from the kubernetes
>>>> profile? it doesn't mean it goes away, just means it's run manually,
>>>> not automatically. Is that actually how it's meant to be used anyway?
>>>> in the short term? given the discussion around its requirements and
>>>> minikube and all that?
>>>>
>>>> (Actually, this would also 'solve' the Scala 2.12 build problem too)
>>>>
>>>> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen <[hidden email]> wrote:
>>>> >
>>>> > To be clear I'm currently +1 on this release, with much commentary.
>>>> >
>>>> > OK, the explanation for kubernetes tests makes sense. Yes I think we need to propagate the scala-2.12 build profile to make it work. Go for it, if you have a lead on what the change is.
>>>> > This doesn't block the release as it's an issue for tests, and only affects 2.12. However if we had a clean fix for this and there were another RC, I'd include it.
>>>> >
>>>> > Dongjoon has a good point about the spark-kubernetes-integration-tests artifact. That doesn't sound like it should be published in this way, though, of course, we publish the test artifacts from every module already. This is only a bit odd in being a non-test artifact meant for testing. But it's special testing! So I also don't think that needs to block a release.
>>>> >
>>>> > This happens because the integration tests module is enabled with the 'kubernetes' profile too, and also this output is copied into the release tarball at kubernetes/integration-tests/tests. Do we need that in a binary release?
>>>> >
>>>> > If these integration tests are meant to be run ad hoc, manually, not part of a normal test cycle, then I think we can just not enable it with -Pkubernetes. If it is meant to run every time, then it sounds like we need a little extra work shown in recent PRs to make that easier, but then, this test code should just be the 'test' artifact parts of the kubernetes module, no?
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: [hidden email]
>>>>
>>
>>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: What if anything to fix about k8s for the 2.4.0 RC5?

Xiao Li
Hopefully, this will not delay RC5. Since this is not a blocker ticket, RC5 will start if all the blocker tickets are resolved.

Thanks,

Xiao 

Sean Owen <[hidden email]> 于2018年10月25日周四 上午8:44写道:
Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :)

On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos
<[hidden email]> wrote:
>
> I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right?
>
> For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 .
>
>
> On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos <[hidden email]> wrote:
>>
>> I will open a jira for the profile propagation issue and have a look to fix it.
>>
>> Stavros
>>
>> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson <[hidden email]> wrote:
>>>
>>>
>>> I would be comfortable making the integration testing manual for now.  A JIRA for ironing out how to make it reliable for automatic as a goal for 3.0 seems like a good idea.
>>>
>>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen <[hidden email]> wrote:
>>>>
>>>> Forking this thread.
>>>>
>>>> Because we'll have another RC, we could possibly address these two
>>>> issues. Only if we have a reliable change of course.
>>>>
>>>> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt.
>>>>
>>>> And is it reasonable to essentially 'disable'
>>>> kubernetes/integration-tests by removing it from the kubernetes
>>>> profile? it doesn't mean it goes away, just means it's run manually,
>>>> not automatically. Is that actually how it's meant to be used anyway?
>>>> in the short term? given the discussion around its requirements and
>>>> minikube and all that?
>>>>
>>>> (Actually, this would also 'solve' the Scala 2.12 build problem too)
>>>>
>>>> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen <[hidden email]> wrote:
>>>> >
>>>> > To be clear I'm currently +1 on this release, with much commentary.
>>>> >
>>>> > OK, the explanation for kubernetes tests makes sense. Yes I think we need to propagate the scala-2.12 build profile to make it work. Go for it, if you have a lead on what the change is.
>>>> > This doesn't block the release as it's an issue for tests, and only affects 2.12. However if we had a clean fix for this and there were another RC, I'd include it.
>>>> >
>>>> > Dongjoon has a good point about the spark-kubernetes-integration-tests artifact. That doesn't sound like it should be published in this way, though, of course, we publish the test artifacts from every module already. This is only a bit odd in being a non-test artifact meant for testing. But it's special testing! So I also don't think that needs to block a release.
>>>> >
>>>> > This happens because the integration tests module is enabled with the 'kubernetes' profile too, and also this output is copied into the release tarball at kubernetes/integration-tests/tests. Do we need that in a binary release?
>>>> >
>>>> > If these integration tests are meant to be run ad hoc, manually, not part of a normal test cycle, then I think we can just not enable it with -Pkubernetes. If it is meant to run every time, then it sounds like we need a little extra work shown in recent PRs to make that easier, but then, this test code should just be the 'test' artifact parts of the kubernetes module, no?
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: [hidden email]
>>>>
>>
>>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: What if anything to fix about k8s for the 2.4.0 RC5?

Sean Owen-2
I think it's worth getting in a change to just not enable this module,
which ought to be entirely safe, and avoid two of the issues we
identified.
that said it didn't block RC4 so need not block RC5.
But should happen today if we're doing it.
On Thu, Oct 25, 2018 at 10:47 AM Xiao Li <[hidden email]> wrote:

>
> Hopefully, this will not delay RC5. Since this is not a blocker ticket, RC5 will start if all the blocker tickets are resolved.
>
> Thanks,
>
> Xiao
>
> Sean Owen <[hidden email]> 于2018年10月25日周四 上午8:44写道:
>>
>> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :)
>>
>> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos
>> <[hidden email]> wrote:
>> >
>> > I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right?
>> >
>> > For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 .
>> >
>> >
>> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos <[hidden email]> wrote:
>> >>
>> >> I will open a jira for the profile propagation issue and have a look to fix it.
>> >>
>> >> Stavros
>> >>
>> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson <[hidden email]> wrote:
>> >>>
>> >>>
>> >>> I would be comfortable making the integration testing manual for now.  A JIRA for ironing out how to make it reliable for automatic as a goal for 3.0 seems like a good idea.
>> >>>
>> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen <[hidden email]> wrote:
>> >>>>
>> >>>> Forking this thread.
>> >>>>
>> >>>> Because we'll have another RC, we could possibly address these two
>> >>>> issues. Only if we have a reliable change of course.
>> >>>>
>> >>>> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt.
>> >>>>
>> >>>> And is it reasonable to essentially 'disable'
>> >>>> kubernetes/integration-tests by removing it from the kubernetes
>> >>>> profile? it doesn't mean it goes away, just means it's run manually,
>> >>>> not automatically. Is that actually how it's meant to be used anyway?
>> >>>> in the short term? given the discussion around its requirements and
>> >>>> minikube and all that?
>> >>>>
>> >>>> (Actually, this would also 'solve' the Scala 2.12 build problem too)
>> >>>>
>> >>>> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen <[hidden email]> wrote:
>> >>>> >
>> >>>> > To be clear I'm currently +1 on this release, with much commentary.
>> >>>> >
>> >>>> > OK, the explanation for kubernetes tests makes sense. Yes I think we need to propagate the scala-2.12 build profile to make it work. Go for it, if you have a lead on what the change is.
>> >>>> > This doesn't block the release as it's an issue for tests, and only affects 2.12. However if we had a clean fix for this and there were another RC, I'd include it.
>> >>>> >
>> >>>> > Dongjoon has a good point about the spark-kubernetes-integration-tests artifact. That doesn't sound like it should be published in this way, though, of course, we publish the test artifacts from every module already. This is only a bit odd in being a non-test artifact meant for testing. But it's special testing! So I also don't think that needs to block a release.
>> >>>> >
>> >>>> > This happens because the integration tests module is enabled with the 'kubernetes' profile too, and also this output is copied into the release tarball at kubernetes/integration-tests/tests. Do we need that in a binary release?
>> >>>> >
>> >>>> > If these integration tests are meant to be run ad hoc, manually, not part of a normal test cycle, then I think we can just not enable it with -Pkubernetes. If it is meant to run every time, then it sounds like we need a little extra work shown in recent PRs to make that easier, but then, this test code should just be the 'test' artifact parts of the kubernetes module, no?
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe e-mail: [hidden email]
>> >>>>
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: What if anything to fix about k8s for the 2.4.0 RC5?

Stavros Kontopoulos-3
I think it's worth getting in a change to just not enable this module,
which ought to be entirely safe, and avoid two of the issues we
identified.

Besides disabling it, when someone wants to run the tests with 2.12 he should be able to do so. So propagating the Scala profile still makes sense
but it is not related to the release other than making sure things work fine. 

On Thu, Oct 25, 2018 at 7:02 PM, Sean Owen <[hidden email]> wrote:
I think it's worth getting in a change to just not enable this module,
which ought to be entirely safe, and avoid two of the issues we
identified.
that said it didn't block RC4 so need not block RC5.
But should happen today if we're doing it.
On Thu, Oct 25, 2018 at 10:47 AM Xiao Li <[hidden email]> wrote:
>
> Hopefully, this will not delay RC5. Since this is not a blocker ticket, RC5 will start if all the blocker tickets are resolved.
>
> Thanks,
>
> Xiao
>
> Sean Owen <[hidden email]> 于2018年10月25日周四 上午8:44写道:
>>
>> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :)
>>
>> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos
>> <[hidden email]> wrote:
>> >
>> > I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right?
>> >
>> > For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 .
>> >
>> >
>> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos <[hidden email]> wrote:
>> >>
>> >> I will open a jira for the profile propagation issue and have a look to fix it.
>> >>
>> >> Stavros
>> >>
>> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson <[hidden email]> wrote:
>> >>>
>> >>>
>> >>> I would be comfortable making the integration testing manual for now.  A JIRA for ironing out how to make it reliable for automatic as a goal for 3.0 seems like a good idea.
>> >>>
>> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen <[hidden email]> wrote:
>> >>>>
>> >>>> Forking this thread.
>> >>>>
>> >>>> Because we'll have another RC, we could possibly address these two
>> >>>> issues. Only if we have a reliable change of course.
>> >>>>
>> >>>> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt.
>> >>>>
>> >>>> And is it reasonable to essentially 'disable'
>> >>>> kubernetes/integration-tests by removing it from the kubernetes
>> >>>> profile? it doesn't mean it goes away, just means it's run manually,
>> >>>> not automatically. Is that actually how it's meant to be used anyway?
>> >>>> in the short term? given the discussion around its requirements and
>> >>>> minikube and all that?
>> >>>>
>> >>>> (Actually, this would also 'solve' the Scala 2.12 build problem too)
>> >>>>
>> >>>> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen <[hidden email]> wrote:
>> >>>> >
>> >>>> > To be clear I'm currently +1 on this release, with much commentary.
>> >>>> >
>> >>>> > OK, the explanation for kubernetes tests makes sense. Yes I think we need to propagate the scala-2.12 build profile to make it work. Go for it, if you have a lead on what the change is.
>> >>>> > This doesn't block the release as it's an issue for tests, and only affects 2.12. However if we had a clean fix for this and there were another RC, I'd include it.
>> >>>> >
>> >>>> > Dongjoon has a good point about the spark-kubernetes-integration-tests artifact. That doesn't sound like it should be published in this way, though, of course, we publish the test artifacts from every module already. This is only a bit odd in being a non-test artifact meant for testing. But it's special testing! So I also don't think that needs to block a release.
>> >>>> >
>> >>>> > This happens because the integration tests module is enabled with the 'kubernetes' profile too, and also this output is copied into the release tarball at kubernetes/integration-tests/tests. Do we need that in a binary release?
>> >>>> >
>> >>>> > If these integration tests are meant to be run ad hoc, manually, not part of a normal test cycle, then I think we can just not enable it with -Pkubernetes. If it is meant to run every time, then it sounds like we need a little extra work shown in recent PRs to make that easier, but then, this test code should just be the 'test' artifact parts of the kubernetes module, no?
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe e-mail: [hidden email]
>> >>>>
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>



--
Stavros Kontopoulos
Senior Software Engineer
Lightbend, Inc.
<a href="tel:%2B1%20650%20678%200020" value="+16506780020" target="_blank">p:  +30 6977967274
Reply | Threaded
Open this post in threaded view
|

Re: What if anything to fix about k8s for the 2.4.0 RC5?

cloud0fan
Any updates on this topic? https://github.com/apache/spark/pull/22827 is merged and 2.4 is unblocked.

I'll cut RC5 shortly after the weekend, and it will be great to include the change proposed here.

Thanks,
Wenchen

On Fri, Oct 26, 2018 at 12:55 AM Stavros Kontopoulos <[hidden email]> wrote:
I think it's worth getting in a change to just not enable this module,
which ought to be entirely safe, and avoid two of the issues we
identified.

Besides disabling it, when someone wants to run the tests with 2.12 he should be able to do so. So propagating the Scala profile still makes sense
but it is not related to the release other than making sure things work fine. 

On Thu, Oct 25, 2018 at 7:02 PM, Sean Owen <[hidden email]> wrote:
I think it's worth getting in a change to just not enable this module,
which ought to be entirely safe, and avoid two of the issues we
identified.
that said it didn't block RC4 so need not block RC5.
But should happen today if we're doing it.
On Thu, Oct 25, 2018 at 10:47 AM Xiao Li <[hidden email]> wrote:
>
> Hopefully, this will not delay RC5. Since this is not a blocker ticket, RC5 will start if all the blocker tickets are resolved.
>
> Thanks,
>
> Xiao
>
> Sean Owen <[hidden email]> 于2018年10月25日周四 上午8:44写道:
>>
>> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :)
>>
>> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos
>> <[hidden email]> wrote:
>> >
>> > I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right?
>> >
>> > For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 .
>> >
>> >
>> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos <[hidden email]> wrote:
>> >>
>> >> I will open a jira for the profile propagation issue and have a look to fix it.
>> >>
>> >> Stavros
>> >>
>> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson <[hidden email]> wrote:
>> >>>
>> >>>
>> >>> I would be comfortable making the integration testing manual for now.  A JIRA for ironing out how to make it reliable for automatic as a goal for 3.0 seems like a good idea.
>> >>>
>> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen <[hidden email]> wrote:
>> >>>>
>> >>>> Forking this thread.
>> >>>>
>> >>>> Because we'll have another RC, we could possibly address these two
>> >>>> issues. Only if we have a reliable change of course.
>> >>>>
>> >>>> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt.
>> >>>>
>> >>>> And is it reasonable to essentially 'disable'
>> >>>> kubernetes/integration-tests by removing it from the kubernetes
>> >>>> profile? it doesn't mean it goes away, just means it's run manually,
>> >>>> not automatically. Is that actually how it's meant to be used anyway?
>> >>>> in the short term? given the discussion around its requirements and
>> >>>> minikube and all that?
>> >>>>
>> >>>> (Actually, this would also 'solve' the Scala 2.12 build problem too)
>> >>>>
>> >>>> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen <[hidden email]> wrote:
>> >>>> >
>> >>>> > To be clear I'm currently +1 on this release, with much commentary.
>> >>>> >
>> >>>> > OK, the explanation for kubernetes tests makes sense. Yes I think we need to propagate the scala-2.12 build profile to make it work. Go for it, if you have a lead on what the change is.
>> >>>> > This doesn't block the release as it's an issue for tests, and only affects 2.12. However if we had a clean fix for this and there were another RC, I'd include it.
>> >>>> >
>> >>>> > Dongjoon has a good point about the spark-kubernetes-integration-tests artifact. That doesn't sound like it should be published in this way, though, of course, we publish the test artifacts from every module already. This is only a bit odd in being a non-test artifact meant for testing. But it's special testing! So I also don't think that needs to block a release.
>> >>>> >
>> >>>> > This happens because the integration tests module is enabled with the 'kubernetes' profile too, and also this output is copied into the release tarball at kubernetes/integration-tests/tests. Do we need that in a binary release?
>> >>>> >
>> >>>> > If these integration tests are meant to be run ad hoc, manually, not part of a normal test cycle, then I think we can just not enable it with -Pkubernetes. If it is meant to run every time, then it sounds like we need a little extra work shown in recent PRs to make that easier, but then, this test code should just be the 'test' artifact parts of the kubernetes module, no?
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe e-mail: [hidden email]
>> >>>>
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>



--
Stavros Kontopoulos
Senior Software Engineer
Lightbend, Inc.
<a href="tel:%2B1%20650%20678%200020" value="+16506780020" target="_blank">p:  +30 6977967274
Reply | Threaded
Open this post in threaded view
|

Re: What if anything to fix about k8s for the 2.4.0 RC5?

Sean Owen-2
Yep, we're going to merge a change to separate the k8s tests into a separate profile, and fix up the Scala 2.12 thing. While non-critical those are pretty nice to have for 2.4. I think that's doable within the next 12 hours even.

@skonto I think there's one last minor thing needed on this PR?

On Thu, Oct 25, 2018 at 6:42 PM Wenchen Fan <[hidden email]> wrote:
Any updates on this topic? https://github.com/apache/spark/pull/22827 is merged and 2.4 is unblocked.

I'll cut RC5 shortly after the weekend, and it will be great to include the change proposed here.

Thanks,
Wenchen

On Fri, Oct 26, 2018 at 12:55 AM Stavros Kontopoulos <[hidden email]> wrote:
I think it's worth getting in a change to just not enable this module,
which ought to be entirely safe, and avoid two of the issues we
identified.

Besides disabling it, when someone wants to run the tests with 2.12 he should be able to do so. So propagating the Scala profile still makes sense
but it is not related to the release other than making sure things work fine. 

On Thu, Oct 25, 2018 at 7:02 PM, Sean Owen <[hidden email]> wrote:
I think it's worth getting in a change to just not enable this module,
which ought to be entirely safe, and avoid two of the issues we
identified.
that said it didn't block RC4 so need not block RC5.
But should happen today if we're doing it.
On Thu, Oct 25, 2018 at 10:47 AM Xiao Li <[hidden email]> wrote:
>
> Hopefully, this will not delay RC5. Since this is not a blocker ticket, RC5 will start if all the blocker tickets are resolved.
>
> Thanks,
>
> Xiao
>
> Sean Owen <[hidden email]> 于2018年10月25日周四 上午8:44写道:
>>
>> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :)
>>
>> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos
>> <[hidden email]> wrote:
>> >
>> > I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right?
>> >
>> > For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 .
>> >
>> >
>> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos <[hidden email]> wrote:
>> >>
>> >> I will open a jira for the profile propagation issue and have a look to fix it.
>> >>
>> >> Stavros
>> >>
>> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson <[hidden email]> wrote:
>> >>>
>> >>>
>> >>> I would be comfortable making the integration testing manual for now.  A JIRA for ironing out how to make it reliable for automatic as a goal for 3.0 seems like a good idea.
>> >>>
>> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen <[hidden email]> wrote:
>> >>>>
>> >>>> Forking this thread.
>> >>>>
>> >>>> Because we'll have another RC, we could possibly address these two
>> >>>> issues. Only if we have a reliable change of course.
>> >>>>
>> >>>> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt.
>> >>>>
>> >>>> And is it reasonable to essentially 'disable'
>> >>>> kubernetes/integration-tests by removing it from the kubernetes
>> >>>> profile? it doesn't mean it goes away, just means it's run manually,
>> >>>> not automatically. Is that actually how it's meant to be used anyway?
>> >>>> in the short term? given the discussion around its requirements and
>> >>>> minikube and all that?
>> >>>>
>> >>>> (Actually, this would also 'solve' the Scala 2.12 build problem too)
>> >>>>
>> >>>> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen <[hidden email]> wrote:
>> >>>> >
>> >>>> > To be clear I'm currently +1 on this release, with much commentary.
>> >>>> >
>> >>>> > OK, the explanation for kubernetes tests makes sense. Yes I think we need to propagate the scala-2.12 build profile to make it work. Go for it, if you have a lead on what the change is.
>> >>>> > This doesn't block the release as it's an issue for tests, and only affects 2.12. However if we had a clean fix for this and there were another RC, I'd include it.
>> >>>> >
>> >>>> > Dongjoon has a good point about the spark-kubernetes-integration-tests artifact. That doesn't sound like it should be published in this way, though, of course, we publish the test artifacts from every module already. This is only a bit odd in being a non-test artifact meant for testing. But it's special testing! So I also don't think that needs to block a release.
>> >>>> >
>> >>>> > This happens because the integration tests module is enabled with the 'kubernetes' profile too, and also this output is copied into the release tarball at kubernetes/integration-tests/tests. Do we need that in a binary release?
>> >>>> >
>> >>>> > If these integration tests are meant to be run ad hoc, manually, not part of a normal test cycle, then I think we can just not enable it with -Pkubernetes. If it is meant to run every time, then it sounds like we need a little extra work shown in recent PRs to make that easier, but then, this test code should just be the 'test' artifact parts of the kubernetes module, no?
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe e-mail: [hidden email]
>> >>>>
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>



--
Stavros Kontopoulos
Senior Software Engineer
Lightbend, Inc.
<a href="tel:%2B1%20650%20678%200020" value="+16506780020" target="_blank">p:  +30 6977967274
Reply | Threaded
Open this post in threaded view
|

Re: What if anything to fix about k8s for the 2.4.0 RC5?

rxin
I also think we should get this in: https://github.com/apache/spark/pull/22841

It's to deprecate a confusing & broken window function API, so we can remove them in 3.0 and redesign a better one. See https://issues.apache.org/jira/browse/SPARK-25841 for more information.


On Thu, Oct 25, 2018 at 4:55 PM Sean Owen <[hidden email]> wrote:
Yep, we're going to merge a change to separate the k8s tests into a separate profile, and fix up the Scala 2.12 thing. While non-critical those are pretty nice to have for 2.4. I think that's doable within the next 12 hours even.

@skonto I think there's one last minor thing needed on this PR?

On Thu, Oct 25, 2018 at 6:42 PM Wenchen Fan <[hidden email]> wrote:
Any updates on this topic? https://github.com/apache/spark/pull/22827 is merged and 2.4 is unblocked.

I'll cut RC5 shortly after the weekend, and it will be great to include the change proposed here.

Thanks,
Wenchen

On Fri, Oct 26, 2018 at 12:55 AM Stavros Kontopoulos <[hidden email]> wrote:
I think it's worth getting in a change to just not enable this module,
which ought to be entirely safe, and avoid two of the issues we
identified.

Besides disabling it, when someone wants to run the tests with 2.12 he should be able to do so. So propagating the Scala profile still makes sense
but it is not related to the release other than making sure things work fine. 

On Thu, Oct 25, 2018 at 7:02 PM, Sean Owen <[hidden email]> wrote:
I think it's worth getting in a change to just not enable this module,
which ought to be entirely safe, and avoid two of the issues we
identified.
that said it didn't block RC4 so need not block RC5.
But should happen today if we're doing it.
On Thu, Oct 25, 2018 at 10:47 AM Xiao Li <[hidden email]> wrote:
>
> Hopefully, this will not delay RC5. Since this is not a blocker ticket, RC5 will start if all the blocker tickets are resolved.
>
> Thanks,
>
> Xiao
>
> Sean Owen <[hidden email]> 于2018年10月25日周四 上午8:44写道:
>>
>> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :)
>>
>> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos
>> <[hidden email]> wrote:
>> >
>> > I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right?
>> >
>> > For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 .
>> >
>> >
>> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos <[hidden email]> wrote:
>> >>
>> >> I will open a jira for the profile propagation issue and have a look to fix it.
>> >>
>> >> Stavros
>> >>
>> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson <[hidden email]> wrote:
>> >>>
>> >>>
>> >>> I would be comfortable making the integration testing manual for now.  A JIRA for ironing out how to make it reliable for automatic as a goal for 3.0 seems like a good idea.
>> >>>
>> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen <[hidden email]> wrote:
>> >>>>
>> >>>> Forking this thread.
>> >>>>
>> >>>> Because we'll have another RC, we could possibly address these two
>> >>>> issues. Only if we have a reliable change of course.
>> >>>>
>> >>>> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt.
>> >>>>
>> >>>> And is it reasonable to essentially 'disable'
>> >>>> kubernetes/integration-tests by removing it from the kubernetes
>> >>>> profile? it doesn't mean it goes away, just means it's run manually,
>> >>>> not automatically. Is that actually how it's meant to be used anyway?
>> >>>> in the short term? given the discussion around its requirements and
>> >>>> minikube and all that?
>> >>>>
>> >>>> (Actually, this would also 'solve' the Scala 2.12 build problem too)
>> >>>>
>> >>>> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen <[hidden email]> wrote:
>> >>>> >
>> >>>> > To be clear I'm currently +1 on this release, with much commentary.
>> >>>> >
>> >>>> > OK, the explanation for kubernetes tests makes sense. Yes I think we need to propagate the scala-2.12 build profile to make it work. Go for it, if you have a lead on what the change is.
>> >>>> > This doesn't block the release as it's an issue for tests, and only affects 2.12. However if we had a clean fix for this and there were another RC, I'd include it.
>> >>>> >
>> >>>> > Dongjoon has a good point about the spark-kubernetes-integration-tests artifact. That doesn't sound like it should be published in this way, though, of course, we publish the test artifacts from every module already. This is only a bit odd in being a non-test artifact meant for testing. But it's special testing! So I also don't think that needs to block a release.
>> >>>> >
>> >>>> > This happens because the integration tests module is enabled with the 'kubernetes' profile too, and also this output is copied into the release tarball at kubernetes/integration-tests/tests. Do we need that in a binary release?
>> >>>> >
>> >>>> > If these integration tests are meant to be run ad hoc, manually, not part of a normal test cycle, then I think we can just not enable it with -Pkubernetes. If it is meant to run every time, then it sounds like we need a little extra work shown in recent PRs to make that easier, but then, this test code should just be the 'test' artifact parts of the kubernetes module, no?
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe e-mail: [hidden email]
>> >>>>
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>



--
Stavros Kontopoulos
Senior Software Engineer
Lightbend, Inc.
<a href="tel:%2B1%20650%20678%200020" value="+16506780020" target="_blank">p:  +30 6977967274
Reply | Threaded
Open this post in threaded view
|

Re: What if anything to fix about k8s for the 2.4.0 RC5?

Stavros Kontopoulos-3
In reply to this post by Sean Owen-2
Sean,

Yes, I updated the PR and re-run it.

On Fri, Oct 26, 2018 at 2:54 AM, Sean Owen <[hidden email]> wrote:
Yep, we're going to merge a change to separate the k8s tests into a separate profile, and fix up the Scala 2.12 thing. While non-critical those are pretty nice to have for 2.4. I think that's doable within the next 12 hours even.

@skonto I think there's one last minor thing needed on this PR?

On Thu, Oct 25, 2018 at 6:42 PM Wenchen Fan <[hidden email]> wrote:
Any updates on this topic? https://github.com/apache/spark/pull/22827 is merged and 2.4 is unblocked.

I'll cut RC5 shortly after the weekend, and it will be great to include the change proposed here.

Thanks,
Wenchen

On Fri, Oct 26, 2018 at 12:55 AM Stavros Kontopoulos <[hidden email]> wrote:
I think it's worth getting in a change to just not enable this module,
which ought to be entirely safe, and avoid two of the issues we
identified.

Besides disabling it, when someone wants to run the tests with 2.12 he should be able to do so. So propagating the Scala profile still makes sense
but it is not related to the release other than making sure things work fine. 

On Thu, Oct 25, 2018 at 7:02 PM, Sean Owen <[hidden email]> wrote:
I think it's worth getting in a change to just not enable this module,
which ought to be entirely safe, and avoid two of the issues we
identified.
that said it didn't block RC4 so need not block RC5.
But should happen today if we're doing it.
On Thu, Oct 25, 2018 at 10:47 AM Xiao Li <[hidden email]> wrote:
>
> Hopefully, this will not delay RC5. Since this is not a blocker ticket, RC5 will start if all the blocker tickets are resolved.
>
> Thanks,
>
> Xiao
>
> Sean Owen <[hidden email]> 于2018年10月25日周四 上午8:44写道:
>>
>> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 :)
>>
>> On Thu, Oct 25, 2018 at 10:41 AM Stavros Kontopoulos
>> <[hidden email]> wrote:
>> >
>> > I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right?
>> >
>> > For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 .
>> >
>> >
>> > On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos <[hidden email]> wrote:
>> >>
>> >> I will open a jira for the profile propagation issue and have a look to fix it.
>> >>
>> >> Stavros
>> >>
>> >> On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson <[hidden email]> wrote:
>> >>>
>> >>>
>> >>> I would be comfortable making the integration testing manual for now.  A JIRA for ironing out how to make it reliable for automatic as a goal for 3.0 seems like a good idea.
>> >>>
>> >>> On Thu, Oct 25, 2018 at 8:11 AM Sean Owen <[hidden email]> wrote:
>> >>>>
>> >>>> Forking this thread.
>> >>>>
>> >>>> Because we'll have another RC, we could possibly address these two
>> >>>> issues. Only if we have a reliable change of course.
>> >>>>
>> >>>> Is it easy enough to propagate the -Pscala-2.12 profile? can't hurt.
>> >>>>
>> >>>> And is it reasonable to essentially 'disable'
>> >>>> kubernetes/integration-tests by removing it from the kubernetes
>> >>>> profile? it doesn't mean it goes away, just means it's run manually,
>> >>>> not automatically. Is that actually how it's meant to be used anyway?
>> >>>> in the short term? given the discussion around its requirements and
>> >>>> minikube and all that?
>> >>>>
>> >>>> (Actually, this would also 'solve' the Scala 2.12 build problem too)
>> >>>>
>> >>>> On Tue, Oct 23, 2018 at 2:45 PM Sean Owen <[hidden email]> wrote:
>> >>>> >
>> >>>> > To be clear I'm currently +1 on this release, with much commentary.
>> >>>> >
>> >>>> > OK, the explanation for kubernetes tests makes sense. Yes I think we need to propagate the scala-2.12 build profile to make it work. Go for it, if you have a lead on what the change is.
>> >>>> > This doesn't block the release as it's an issue for tests, and only affects 2.12. However if we had a clean fix for this and there were another RC, I'd include it.
>> >>>> >
>> >>>> > Dongjoon has a good point about the spark-kubernetes-integration-tests artifact. That doesn't sound like it should be published in this way, though, of course, we publish the test artifacts from every module already. This is only a bit odd in being a non-test artifact meant for testing. But it's special testing! So I also don't think that needs to block a release.
>> >>>> >
>> >>>> > This happens because the integration tests module is enabled with the 'kubernetes' profile too, and also this output is copied into the release tarball at kubernetes/integration-tests/tests. Do we need that in a binary release?
>> >>>> >
>> >>>> > If these integration tests are meant to be run ad hoc, manually, not part of a normal test cycle, then I think we can just not enable it with -Pkubernetes. If it is meant to run every time, then it sounds like we need a little extra work shown in recent PRs to make that easier, but then, this test code should just be the 'test' artifact parts of the kubernetes module, no?
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe e-mail: [hidden email]
>> >>>>
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>



Reply | Threaded
Open this post in threaded view
|

Re: What if anything to fix about k8s for the 2.4.0 RC5?

Sean Owen-2
In reply to this post by Sean Owen-2
This is all merged to master/2.4. AFAIK there aren't any items I'm monitoring that are needed for 2.4. 

On Thu, Oct 25, 2018 at 6:54 PM Sean Owen <[hidden email]> wrote:
Yep, we're going to merge a change to separate the k8s tests into a separate profile, and fix up the Scala 2.12 thing. While non-critical those are pretty nice to have for 2.4. I think that's doable within the next 12 hours even.

@skonto I think there's one last minor thing needed on this PR?

On Thu, Oct 25, 2018 at 6:42 PM Wenchen Fan <[hidden email]> wrote:
Any updates on this topic? https://github.com/apache/spark/pull/22827 is merged and 2.4 is unblocked.

I'll cut RC5 shortly after the weekend, and it will be great to include the change proposed here.

Thanks,
Wenchen
123