Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Sean Owen-2
Yesterday I cleaned out old Spark releases from the mirror system -- we're supposed to only keep the latest release from active branches out on mirrors. (All releases are available from the Apache archive site.)

Having done so I realized quickly that the HiveExternalCatalogVersionsSuite relies on the versions it downloads being available from mirrors. It has been flaky, as sometimes mirrors are unreliable. I think now it will not work for any versions except 2.3.1, 2.2.2, 2.1.3.

Because we do need to clean those releases out of the mirrors soon anyway, and because they're flaky sometimes, I propose adding logic to the test to fall back on downloading from the Apache archive site.

... and I'll do that right away to unblock HiveExternalCatalogVersionsSuite runs. I think it needs to be backported to other branches as they will still be testing against potentially non-current Spark releases.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

rxin
Makes sense. Thanks for looking into this. 

On Sun, Jul 15, 2018 at 1:51 PM Sean Owen <[hidden email]> wrote:
Yesterday I cleaned out old Spark releases from the mirror system -- we're supposed to only keep the latest release from active branches out on mirrors. (All releases are available from the Apache archive site.)

Having done so I realized quickly that the HiveExternalCatalogVersionsSuite relies on the versions it downloads being available from mirrors. It has been flaky, as sometimes mirrors are unreliable. I think now it will not work for any versions except 2.3.1, 2.2.2, 2.1.3.

Because we do need to clean those releases out of the mirrors soon anyway, and because they're flaky sometimes, I propose adding logic to the test to fall back on downloading from the Apache archive site.

... and I'll do that right away to unblock HiveExternalCatalogVersionsSuite runs. I think it needs to be backported to other branches as they will still be testing against potentially non-current Spark releases.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Marco Gaido
+1, this was indeed a problem in the past.

On Sun, 15 Jul 2018, 22:56 Reynold Xin, <[hidden email]> wrote:
Makes sense. Thanks for looking into this. 

On Sun, Jul 15, 2018 at 1:51 PM Sean Owen <[hidden email]> wrote:
Yesterday I cleaned out old Spark releases from the mirror system -- we're supposed to only keep the latest release from active branches out on mirrors. (All releases are available from the Apache archive site.)

Having done so I realized quickly that the HiveExternalCatalogVersionsSuite relies on the versions it downloads being available from mirrors. It has been flaky, as sometimes mirrors are unreliable. I think now it will not work for any versions except 2.3.1, 2.2.2, 2.1.3.

Because we do need to clean those releases out of the mirrors soon anyway, and because they're flaky sometimes, I propose adding logic to the test to fall back on downloading from the Apache archive site.

... and I'll do that right away to unblock HiveExternalCatalogVersionsSuite runs. I think it needs to be backported to other branches as they will still be testing against potentially non-current Spark releases.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Sean Owen-3
In reply to this post by Sean Owen-2
Fix is committed to branches back through 2.2.x, where this test was added.

There is still some issue; I'm seeing that archive.apache.org is rate-limiting downloads and frequently returning 503 errors.

We can help, I guess, by avoiding testing against non-current releases. Right now we should be testing against 2.3.1, 2.2.2, 2.1.3, right? 2.0.x is now effectively EOL right?

I can make that quick change too if everyone's amenable, in order to prevent more failures in this test from master.

On Sun, Jul 15, 2018 at 3:51 PM Sean Owen <[hidden email]> wrote:
Yesterday I cleaned out old Spark releases from the mirror system -- we're supposed to only keep the latest release from active branches out on mirrors. (All releases are available from the Apache archive site.)

Having done so I realized quickly that the HiveExternalCatalogVersionsSuite relies on the versions it downloads being available from mirrors. It has been flaky, as sometimes mirrors are unreliable. I think now it will not work for any versions except 2.3.1, 2.2.2, 2.1.3.

Because we do need to clean those releases out of the mirrors soon anyway, and because they're flaky sometimes, I propose adding logic to the test to fall back on downloading from the Apache archive site.

... and I'll do that right away to unblock HiveExternalCatalogVersionsSuite runs. I think it needs to be backported to other branches as they will still be testing against potentially non-current Spark releases.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Hyukjin Kwon
+1

2018년 7월 17일 (화) 오전 7:34, Sean Owen <[hidden email]>님이 작성:
Fix is committed to branches back through 2.2.x, where this test was added.

There is still some issue; I'm seeing that archive.apache.org is rate-limiting downloads and frequently returning 503 errors.

We can help, I guess, by avoiding testing against non-current releases. Right now we should be testing against 2.3.1, 2.2.2, 2.1.3, right? 2.0.x is now effectively EOL right?

I can make that quick change too if everyone's amenable, in order to prevent more failures in this test from master.

On Sun, Jul 15, 2018 at 3:51 PM Sean Owen <[hidden email]> wrote:
Yesterday I cleaned out old Spark releases from the mirror system -- we're supposed to only keep the latest release from active branches out on mirrors. (All releases are available from the Apache archive site.)

Having done so I realized quickly that the HiveExternalCatalogVersionsSuite relies on the versions it downloads being available from mirrors. It has been flaky, as sometimes mirrors are unreliable. I think now it will not work for any versions except 2.3.1, 2.2.2, 2.1.3.

Because we do need to clean those releases out of the mirrors soon anyway, and because they're flaky sometimes, I propose adding logic to the test to fall back on downloading from the Apache archive site.

... and I'll do that right away to unblock HiveExternalCatalogVersionsSuite runs. I think it needs to be backported to other branches as they will still be testing against potentially non-current Spark releases.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Marco Gaido
+1 too

On Tue, 17 Jul 2018, 05:38 Hyukjin Kwon, <[hidden email]> wrote:
+1

2018년 7월 17일 (화) 오전 7:34, Sean Owen <[hidden email]>님이 작성:
Fix is committed to branches back through 2.2.x, where this test was added.

There is still some issue; I'm seeing that archive.apache.org is rate-limiting downloads and frequently returning 503 errors.

We can help, I guess, by avoiding testing against non-current releases. Right now we should be testing against 2.3.1, 2.2.2, 2.1.3, right? 2.0.x is now effectively EOL right?

I can make that quick change too if everyone's amenable, in order to prevent more failures in this test from master.

On Sun, Jul 15, 2018 at 3:51 PM Sean Owen <[hidden email]> wrote:
Yesterday I cleaned out old Spark releases from the mirror system -- we're supposed to only keep the latest release from active branches out on mirrors. (All releases are available from the Apache archive site.)

Having done so I realized quickly that the HiveExternalCatalogVersionsSuite relies on the versions it downloads being available from mirrors. It has been flaky, as sometimes mirrors are unreliable. I think now it will not work for any versions except 2.3.1, 2.2.2, 2.1.3.

Because we do need to clean those releases out of the mirrors soon anyway, and because they're flaky sometimes, I propose adding logic to the test to fall back on downloading from the Apache archive site.

... and I'll do that right away to unblock HiveExternalCatalogVersionsSuite runs. I think it needs to be backported to other branches as they will still be testing against potentially non-current Spark releases.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Felix Cheung
+1 this has been problematic.

Also, this list needs to be updated every time we make a new release?

Plus can we cache them on Jenkins, maybe we can avoid downloading the same thing from Apache archive every test run.

 

From: Marco Gaido <[hidden email]>
Sent: Monday, July 16, 2018 11:12 PM
To: Hyukjin Kwon
Cc: Sean Owen; dev
Subject: Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite
 
+1 too

On Tue, 17 Jul 2018, 05:38 Hyukjin Kwon, <[hidden email]> wrote:
+1

2018년 7월 17일 (화) 오전 7:34, Sean Owen <[hidden email]>님이 작성:
Fix is committed to branches back through 2.2.x, where this test was added.

There is still some issue; I'm seeing that archive.apache.org is rate-limiting downloads and frequently returning 503 errors.

We can help, I guess, by avoiding testing against non-current releases. Right now we should be testing against 2.3.1, 2.2.2, 2.1.3, right? 2.0.x is now effectively EOL right?

I can make that quick change too if everyone's amenable, in order to prevent more failures in this test from master.

On Sun, Jul 15, 2018 at 3:51 PM Sean Owen <[hidden email]> wrote:
Yesterday I cleaned out old Spark releases from the mirror system -- we're supposed to only keep the latest release from active branches out on mirrors. (All releases are available from the Apache archive site.)

Having done so I realized quickly that the HiveExternalCatalogVersionsSuite relies on the versions it downloads being available from mirrors. It has been flaky, as sometimes mirrors are unreliable. I think now it will not work for any versions except 2.3.1, 2.2.2, 2.1.3.

Because we do need to clean those releases out of the mirrors soon anyway, and because they're flaky sometimes, I propose adding logic to the test to fall back on downloading from the Apache archive site.

... and I'll do that right away to unblock HiveExternalCatalogVersionsSuite runs. I think it needs to be backported to other branches as they will still be testing against potentially non-current Spark releases.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Sean Owen-3
Ideally, that list is updated with each release, yes. Non-current releases will now always download from archive.apache.org though. But we run into rate-limiting problems if that gets pinged too much. So yes good to keep the list only to current branches.

It looks like the download is cached in /tmp/test-spark, for what it's worth.

On Thu, Jul 19, 2018 at 11:06 AM Felix Cheung <[hidden email]> wrote:
+1 this has been problematic.

Also, this list needs to be updated every time we make a new release?

Plus can we cache them on Jenkins, maybe we can avoid downloading the same thing from Apache archive every test run.

 

From: Marco Gaido <[hidden email]>
Sent: Monday, July 16, 2018 11:12 PM
To: Hyukjin Kwon
Cc: Sean Owen; dev
Subject: Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite
 
+1 too

On Tue, 17 Jul 2018, 05:38 Hyukjin Kwon, <[hidden email]> wrote:
+1

2018년 7월 17일 (화) 오전 7:34, Sean Owen <[hidden email]>님이 작성:
Fix is committed to branches back through 2.2.x, where this test was added.

There is still some issue; I'm seeing that archive.apache.org is rate-limiting downloads and frequently returning 503 errors.

We can help, I guess, by avoiding testing against non-current releases. Right now we should be testing against 2.3.1, 2.2.2, 2.1.3, right? 2.0.x is now effectively EOL right?

I can make that quick change too if everyone's amenable, in order to prevent more failures in this test from master.

On Sun, Jul 15, 2018 at 3:51 PM Sean Owen <[hidden email]> wrote:
Yesterday I cleaned out old Spark releases from the mirror system -- we're supposed to only keep the latest release from active branches out on mirrors. (All releases are available from the Apache archive site.)

Having done so I realized quickly that the HiveExternalCatalogVersionsSuite relies on the versions it downloads being available from mirrors. It has been flaky, as sometimes mirrors are unreliable. I think now it will not work for any versions except 2.3.1, 2.2.2, 2.1.3.

Because we do need to clean those releases out of the mirrors soon anyway, and because they're flaky sometimes, I propose adding logic to the test to fall back on downloading from the Apache archive site.

... and I'll do that right away to unblock HiveExternalCatalogVersionsSuite runs. I think it needs to be backported to other branches as they will still be testing against potentially non-current Spark releases.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Mark Hamstra
Is there or should there be some checking of digests just to make sure that we are really testing against the same thing in /tmp/test-spark that we are distributing from the archive?

On Thu, Jul 19, 2018 at 11:15 AM Sean Owen <[hidden email]> wrote:
Ideally, that list is updated with each release, yes. Non-current releases will now always download from archive.apache.org though. But we run into rate-limiting problems if that gets pinged too much. So yes good to keep the list only to current branches.

It looks like the download is cached in /tmp/test-spark, for what it's worth.

On Thu, Jul 19, 2018 at 11:06 AM Felix Cheung <[hidden email]> wrote:
+1 this has been problematic.

Also, this list needs to be updated every time we make a new release?

Plus can we cache them on Jenkins, maybe we can avoid downloading the same thing from Apache archive every test run.

 

From: Marco Gaido <[hidden email]>
Sent: Monday, July 16, 2018 11:12 PM
To: Hyukjin Kwon
Cc: Sean Owen; dev
Subject: Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite
 
+1 too

On Tue, 17 Jul 2018, 05:38 Hyukjin Kwon, <[hidden email]> wrote:
+1

2018년 7월 17일 (화) 오전 7:34, Sean Owen <[hidden email]>님이 작성:
Fix is committed to branches back through 2.2.x, where this test was added.

There is still some issue; I'm seeing that archive.apache.org is rate-limiting downloads and frequently returning 503 errors.

We can help, I guess, by avoiding testing against non-current releases. Right now we should be testing against 2.3.1, 2.2.2, 2.1.3, right? 2.0.x is now effectively EOL right?

I can make that quick change too if everyone's amenable, in order to prevent more failures in this test from master.

On Sun, Jul 15, 2018 at 3:51 PM Sean Owen <[hidden email]> wrote:
Yesterday I cleaned out old Spark releases from the mirror system -- we're supposed to only keep the latest release from active branches out on mirrors. (All releases are available from the Apache archive site.)

Having done so I realized quickly that the HiveExternalCatalogVersionsSuite relies on the versions it downloads being available from mirrors. It has been flaky, as sometimes mirrors are unreliable. I think now it will not work for any versions except 2.3.1, 2.2.2, 2.1.3.

Because we do need to clean those releases out of the mirrors soon anyway, and because they're flaky sometimes, I propose adding logic to the test to fall back on downloading from the Apache archive site.

... and I'll do that right away to unblock HiveExternalCatalogVersionsSuite runs. I think it needs to be backported to other branches as they will still be testing against potentially non-current Spark releases.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Sean Owen-3
Yeah if the test code keeps around the archive and/or digest of what it unpacked. A release should never be modified though, so highly rare. 

If the worry is hacked mirrors then we might have bigger problems, but there the issue is verifying the download sigs in the first place. Those would have to come from archive.apache.org.

If you're up for it, yes that could be a fine security precaution. 

On Thu, Jul 19, 2018, 2:11 PM Mark Hamstra <[hidden email]> wrote:
Is there or should there be some checking of digests just to make sure that we are really testing against the same thing in /tmp/test-spark that we are distributing from the archive?

On Thu, Jul 19, 2018 at 11:15 AM Sean Owen <[hidden email]> wrote:
Ideally, that list is updated with each release, yes. Non-current releases will now always download from archive.apache.org though. But we run into rate-limiting problems if that gets pinged too much. So yes good to keep the list only to current branches.

It looks like the download is cached in /tmp/test-spark, for what it's worth.

On Thu, Jul 19, 2018 at 11:06 AM Felix Cheung <[hidden email]> wrote:
+1 this has been problematic.

Also, this list needs to be updated every time we make a new release?

Plus can we cache them on Jenkins, maybe we can avoid downloading the same thing from Apache archive every test run.

 

From: Marco Gaido <[hidden email]>
Sent: Monday, July 16, 2018 11:12 PM
To: Hyukjin Kwon
Cc: Sean Owen; dev
Subject: Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite
 
+1 too

On Tue, 17 Jul 2018, 05:38 Hyukjin Kwon, <[hidden email]> wrote:
+1

2018년 7월 17일 (화) 오전 7:34, Sean Owen <[hidden email]>님이 작성:
Fix is committed to branches back through 2.2.x, where this test was added.

There is still some issue; I'm seeing that archive.apache.org is rate-limiting downloads and frequently returning 503 errors.

We can help, I guess, by avoiding testing against non-current releases. Right now we should be testing against 2.3.1, 2.2.2, 2.1.3, right? 2.0.x is now effectively EOL right?

I can make that quick change too if everyone's amenable, in order to prevent more failures in this test from master.

On Sun, Jul 15, 2018 at 3:51 PM Sean Owen <[hidden email]> wrote:
Yesterday I cleaned out old Spark releases from the mirror system -- we're supposed to only keep the latest release from active branches out on mirrors. (All releases are available from the Apache archive site.)

Having done so I realized quickly that the HiveExternalCatalogVersionsSuite relies on the versions it downloads being available from mirrors. It has been flaky, as sometimes mirrors are unreliable. I think now it will not work for any versions except 2.3.1, 2.2.2, 2.1.3.

Because we do need to clean those releases out of the mirrors soon anyway, and because they're flaky sometimes, I propose adding logic to the test to fall back on downloading from the Apache archive site.

... and I'll do that right away to unblock HiveExternalCatalogVersionsSuite runs. I think it needs to be backported to other branches as they will still be testing against potentially non-current Spark releases.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Mark Hamstra
Yeah, I was mostly thinking that, if the normal Spark PR tests were setup to check the sigs (every time? some of the time?), then this could serve as an automatic check that nothing funny has been done to the archives. There shouldn't be any difference between the cache and the archive; but if there ever is, then we may well have a serious security problem.

On Thu, Jul 19, 2018 at 12:41 PM Sean Owen <[hidden email]> wrote:
Yeah if the test code keeps around the archive and/or digest of what it unpacked. A release should never be modified though, so highly rare. 

If the worry is hacked mirrors then we might have bigger problems, but there the issue is verifying the download sigs in the first place. Those would have to come from archive.apache.org.

If you're up for it, yes that could be a fine security precaution. 

On Thu, Jul 19, 2018, 2:11 PM Mark Hamstra <[hidden email]> wrote:
Is there or should there be some checking of digests just to make sure that we are really testing against the same thing in /tmp/test-spark that we are distributing from the archive?

On Thu, Jul 19, 2018 at 11:15 AM Sean Owen <[hidden email]> wrote:
Ideally, that list is updated with each release, yes. Non-current releases will now always download from archive.apache.org though. But we run into rate-limiting problems if that gets pinged too much. So yes good to keep the list only to current branches.

It looks like the download is cached in /tmp/test-spark, for what it's worth.

On Thu, Jul 19, 2018 at 11:06 AM Felix Cheung <[hidden email]> wrote:
+1 this has been problematic.

Also, this list needs to be updated every time we make a new release?

Plus can we cache them on Jenkins, maybe we can avoid downloading the same thing from Apache archive every test run.

 

From: Marco Gaido <[hidden email]>
Sent: Monday, July 16, 2018 11:12 PM
To: Hyukjin Kwon
Cc: Sean Owen; dev
Subject: Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite
 
+1 too

On Tue, 17 Jul 2018, 05:38 Hyukjin Kwon, <[hidden email]> wrote:
+1

2018년 7월 17일 (화) 오전 7:34, Sean Owen <[hidden email]>님이 작성:
Fix is committed to branches back through 2.2.x, where this test was added.

There is still some issue; I'm seeing that archive.apache.org is rate-limiting downloads and frequently returning 503 errors.

We can help, I guess, by avoiding testing against non-current releases. Right now we should be testing against 2.3.1, 2.2.2, 2.1.3, right? 2.0.x is now effectively EOL right?

I can make that quick change too if everyone's amenable, in order to prevent more failures in this test from master.

On Sun, Jul 15, 2018 at 3:51 PM Sean Owen <[hidden email]> wrote:
Yesterday I cleaned out old Spark releases from the mirror system -- we're supposed to only keep the latest release from active branches out on mirrors. (All releases are available from the Apache archive site.)

Having done so I realized quickly that the HiveExternalCatalogVersionsSuite relies on the versions it downloads being available from mirrors. It has been flaky, as sometimes mirrors are unreliable. I think now it will not work for any versions except 2.3.1, 2.2.2, 2.1.3.

Because we do need to clean those releases out of the mirrors soon anyway, and because they're flaky sometimes, I propose adding logic to the test to fall back on downloading from the Apache archive site.

... and I'll do that right away to unblock HiveExternalCatalogVersionsSuite runs. I think it needs to be backported to other branches as they will still be testing against potentially non-current Spark releases.

Sean
Reply | Threaded
Open this post in threaded view
|

Re: Cleaning Spark releases from mirrors, and the flakiness of HiveExternalCatalogVersionsSuite

Marcelo Vanzin-2
In reply to this post by Sean Owen-2
On this topic... when I worked on 2.3.1 and caused this breakage by
deleting and old release, I tried to write some code to make this more
automatic:

https://github.com/vanzin/spark/tree/SPARK-24532

I just found that the code was a little too large and hacky for what
it does (find out the latest releases on each branch). But maybe it
would be worth to do that?

In any case, agree with Mark that checking signatures would be good, eventually.


On Sun, Jul 15, 2018 at 1:51 PM, Sean Owen <[hidden email]> wrote:

> Yesterday I cleaned out old Spark releases from the mirror system -- we're
> supposed to only keep the latest release from active branches out on
> mirrors. (All releases are available from the Apache archive site.)
>
> Having done so I realized quickly that the HiveExternalCatalogVersionsSuite
> relies on the versions it downloads being available from mirrors. It has
> been flaky, as sometimes mirrors are unreliable. I think now it will not
> work for any versions except 2.3.1, 2.2.2, 2.1.3.
>
> Because we do need to clean those releases out of the mirrors soon anyway,
> and because they're flaky sometimes, I propose adding logic to the test to
> fall back on downloading from the Apache archive site.
>
> ... and I'll do that right away to unblock HiveExternalCatalogVersionsSuite
> runs. I think it needs to be backported to other branches as they will still
> be testing against potentially non-current Spark releases.
>
> Sean



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]