Re: moving the spark jenkins job builder repo from dbricks --> spark

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: moving the spark jenkins job builder repo from dbricks --> spark

shane knapp
hey everyone!

just for visibility, after some lengthy conversations w/some PMC members (mostly sean and josh) about the location of the jenkins job builder temples being in a private, databricks repo, we've decided to move them in to the main apache spark repo.


On Tue, Oct 9, 2018 at 10:22 PM Sean Owen <[hidden email]> wrote:
Some responses inline -- this discussion can do to dev@ though.

dev@ added.
 
On Tue, Oct 9, 2018 at 3:28 PM shane knapp <[hidden email]> wrote:
> JBB templates in spark repo:
> * code path is currently undecided, maybe build/?  i honestly don't have any strong opinions

How about a subfolder of dev/? that's where many items like the
release scripts and build style checkers live.

works for me.
 
> * this stuff will only in the master branch

Sure, it'll get versioned with branches anyway, but only the master
branch will matter.

> * the current JJB templates include release and packaging jobs, which aren't run via jenkins anymore.  this means we can remove the job builder configs for these two, as well as the encrypted secrets.

Sure, if it's not used, remove it. I suppose the concerns below lessen
if none of the jobs in question here create release artifacts.

exactly.
 
> * the JJB templates are able to be run by anyone w/jenkins login access without the need to commit changes to the repo.  this means there's a non-zero potential for bad actors to change the build configs.  since we will only be managing test and compile jobs through this, the chances for Real Bad Stuff[tm] is minimized.  i will also have a local server, not on the jenkins network, run a nightly cron job that grabs the latest configs from github and syncs them to jenkins.

You mean anyone with access to amplab Jenkins? I think this is an
acceptable risk, especially as it's never been an issue to date. The
worst case is deleting or sabotaging CI jobs, right? not great, but,
nobody would be able to commit code or anything. That's the worst
case, right?

re: access to jenkins -- correct.
re: worst case, deleting a job -- correct (but a cron sync of the jobs from the tip of master will repair and damage done by nefarious folks).
re: committing code -- correct
 
This might be a good time to ask whether we want to use a different CI
system. I don't see a need to. I don't see any problem that's surfaced
by publishing the configs as part of the project. If riselab is OK
continuing to subsidize the build infra, and it continues to work for
the project, it seems fine. As long as the PMC is meaningfully in
control of it, I can't see any issue.

i have confirmation from RISELab's PI (ion stoica) that we are committed to continuing to host the apache spark build system here.

if that situation changes (which i cannot foresee), it will of course be brought up w/the community and the PMC immediately.  i would also like some heads up from the PMC if the situation changes on their end.  ;)

btw, work will not begin on this until next week.  once i get a jira and PR opened, i'll respond to this thread w/those links.

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: moving the spark jenkins job builder repo from dbricks --> spark

Marcelo Vanzin-2
Thanks for doing this. The more things we have accessible to the
project members in general the better!

(Now there's that hive fork repo somewhere, but let's not talk about that.)

On Wed, Oct 10, 2018 at 9:30 AM shane knapp <[hidden email]> wrote:
>> > * the JJB templates are able to be run by anyone w/jenkins login access without the need to commit changes to the repo.  this means there's a non-zero potential for bad actors to change the build configs.  since we will only be managing test and compile jobs through this, the chances for Real Bad Stuff[tm] is minimized.  i will also have a local server, not on the jenkins network, run a nightly cron job that grabs the latest configs from github and syncs them to jenkins.

Not sure if that's what you meant; but it should be ok for the jenkins
servers to manually sync with master after you (or someone else) have
verified the changes. That should prevent inadvertent breakages since
I don't expect it to be easy to test those scripts without access to
some test jenkins server.

--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: moving the spark jenkins job builder repo from dbricks --> spark

shane knapp
Not sure if that's what you meant; but it should be ok for the jenkins
servers to manually sync with master after you (or someone else) have
verified the changes. That should prevent inadvertent breakages since
I don't expect it to be easy to test those scripts without access to
some test jenkins server.

JJB has some built-in lint and testing, so that'll be the first step in verifying the build configs.

i still have a dream where i have a fully functioning jenkins staging deployment...  one day i will make that happen.  :)

shane

--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead