[build system] important: potential upgrading of the python build environment

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[build system] important: potential upgrading of the python build environment

shane knapp
(big CC list so the people involved have visibility)

we're currently using a (very old) installation of anaconda python to
manage the python3 build deps, but it turns out that our (very old)
versions of numpy and pandas are starting to hold us back.

reference:
https://github.com/apache/spark/pull/18459

i'm proposing an upgrade to bring numpy and pandas up to 1.11.3 and
0.19.2 respectively...  however, as with all things "dependency hell"
related, a lot more will be upgraded in the process:
https://github.com/apache/spark/pull/18459#issuecomment-313468679

the upgrade is easy to do, and to facilitate "holy shit i broke the
world" rollbacks, the original anaconda installation has been backed
up and i can revert in moments.

i'm reasonably sure that this upgrade won't horribly break things, but
it's hard to know for sure w/o a dedicated staging environment (also
on the books for the end-of-summer new jenkins rollout).

if people seem ok w/this plan (update, watch builds for failures,
rollback immediately if necessary), i will do this first thing
tomorrow morning (~7am PDT).

thoughts?  comments?  concerns?

shane

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [build system] important: potential upgrading of the python build environment

shane knapp
doing this now.

On Thu, Jul 6, 2017 at 1:34 PM, shane knapp <[hidden email]> wrote:

> (big CC list so the people involved have visibility)
>
> we're currently using a (very old) installation of anaconda python to
> manage the python3 build deps, but it turns out that our (very old)
> versions of numpy and pandas are starting to hold us back.
>
> reference:
> https://github.com/apache/spark/pull/18459
>
> i'm proposing an upgrade to bring numpy and pandas up to 1.11.3 and
> 0.19.2 respectively...  however, as with all things "dependency hell"
> related, a lot more will be upgraded in the process:
> https://github.com/apache/spark/pull/18459#issuecomment-313468679
>
> the upgrade is easy to do, and to facilitate "holy shit i broke the
> world" rollbacks, the original anaconda installation has been backed
> up and i can revert in moments.
>
> i'm reasonably sure that this upgrade won't horribly break things, but
> it's hard to know for sure w/o a dedicated staging environment (also
> on the books for the end-of-summer new jenkins rollout).
>
> if people seem ok w/this plan (update, watch builds for failures,
> rollback immediately if necessary), i will do this first thing
> tomorrow morning (~7am PDT).
>
> thoughts?  comments?  concerns?
>
> shane

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [build system] important: potential upgrading of the python build environment

shane knapp
this is done.  i'll babysit builds today.

On Fri, Jul 7, 2017 at 8:40 AM, shane knapp <[hidden email]> wrote:

> doing this now.
>
> On Thu, Jul 6, 2017 at 1:34 PM, shane knapp <[hidden email]> wrote:
>> (big CC list so the people involved have visibility)
>>
>> we're currently using a (very old) installation of anaconda python to
>> manage the python3 build deps, but it turns out that our (very old)
>> versions of numpy and pandas are starting to hold us back.
>>
>> reference:
>> https://github.com/apache/spark/pull/18459
>>
>> i'm proposing an upgrade to bring numpy and pandas up to 1.11.3 and
>> 0.19.2 respectively...  however, as with all things "dependency hell"
>> related, a lot more will be upgraded in the process:
>> https://github.com/apache/spark/pull/18459#issuecomment-313468679
>>
>> the upgrade is easy to do, and to facilitate "holy shit i broke the
>> world" rollbacks, the original anaconda installation has been backed
>> up and i can revert in moments.
>>
>> i'm reasonably sure that this upgrade won't horribly break things, but
>> it's hard to know for sure w/o a dedicated staging environment (also
>> on the books for the end-of-summer new jenkins rollout).
>>
>> if people seem ok w/this plan (update, watch builds for failures,
>> rollback immediately if necessary), i will do this first thing
>> tomorrow morning (~7am PDT).
>>
>> thoughts?  comments?  concerns?
>>
>> shane

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Loading...