SQL test failures in PR builder?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

SQL test failures in PR builder?

Sean Owen-2
I'm seeing consistent failures in the PR builder when touching SQL code:

https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/

 org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's own GetSchemasOperation(SparkGetSchemasOperation)14 ms2
 org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It is not a test it is a sbt.testing.SuiteSelector)

Looks like this has failed about 6 builds in the past few days. Has anyone seen this / has a clue what's causing it? errors are like ...

java.sql.SQLException: No suitable driver found for jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue

Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not org.apache.hadoop.hive.metastore.MetaStoreFilterHook
Reply | Threaded
Open this post in threaded view
|

Re: SQL test failures in PR builder?

Dongjoon Hyun-2
Hi, Sean.

It seems that there is no failure on your other SQL PR.

    https://github.com/apache/spark/pull/26748

Does the sequential failure happen only at `NewSparkPullRequestBuilder`?
Since `NewSparkPullRequestBuilder` is not the same with `SparkPullRequestBuilder`,
there might be a root cause inside it if it happens only at `NewSparkPullRequestBuilder`.

For `org.apache.hive.service.ServiceException: Failed to Start HiveServer2`,
I've observed them before, but the root cause might be different from this one.

BTW, to reduce the scope of investigation, could you try with `[hive-1.2]` tag in your PR?

Bests,
Dongjoon.


On Wed, Dec 4, 2019 at 6:29 AM Sean Owen <[hidden email]> wrote:
I'm seeing consistent failures in the PR builder when touching SQL code:

https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/

 org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's own GetSchemasOperation(SparkGetSchemasOperation)14 ms2
 org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It is not a test it is a sbt.testing.SuiteSelector)

Looks like this has failed about 6 builds in the past few days. Has anyone seen this / has a clue what's causing it? errors are like ...

java.sql.SQLException: No suitable driver found for jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue

Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not org.apache.hadoop.hive.metastore.MetaStoreFilterHook
Reply | Threaded
Open this post in threaded view
|

Re: SQL test failures in PR builder?

Sean Owen-2
BTW does anyone know why there are two PR builder jobs? I'm confused
about why different ones would execute.

Yes I see NewSparkPullRequestBuilder failing on a variety of PRs.
I don't think it has anything to do with Hive; these PRs touch
different parts of code but all not related to this failure.

On Wed, Dec 4, 2019 at 12:40 PM Dongjoon Hyun <[hidden email]> wrote:

>
> Hi, Sean.
>
> It seems that there is no failure on your other SQL PR.
>
>     https://github.com/apache/spark/pull/26748
>
> Does the sequential failure happen only at `NewSparkPullRequestBuilder`?
> Since `NewSparkPullRequestBuilder` is not the same with `SparkPullRequestBuilder`,
> there might be a root cause inside it if it happens only at `NewSparkPullRequestBuilder`.
>
> For `org.apache.hive.service.ServiceException: Failed to Start HiveServer2`,
> I've observed them before, but the root cause might be different from this one.
>
> BTW, to reduce the scope of investigation, could you try with `[hive-1.2]` tag in your PR?
>
> Bests,
> Dongjoon.
>
>
> On Wed, Dec 4, 2019 at 6:29 AM Sean Owen <[hidden email]> wrote:
>>
>> I'm seeing consistent failures in the PR builder when touching SQL code:
>>
>> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/
>>
>>  org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's own GetSchemasOperation(SparkGetSchemasOperation)14 ms2
>>  org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It is not a test it is a sbt.testing.SuiteSelector)
>>
>> Looks like this has failed about 6 builds in the past few days. Has anyone seen this / has a clue what's causing it? errors are like ...
>>
>> java.sql.SQLException: No suitable driver found for jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue
>>
>>
>> Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not org.apache.hadoop.hive.metastore.MetaStoreFilterHook

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: SQL test failures in PR builder?

shane knapp
++yin huai for more insight in to the NewSparkPullRequestBuilder job...

tbh, i never (or still) really understand the exact use for that job,
except that it's triggered by https://spark-prs.appspot.com/

shane


On Wed, Dec 4, 2019 at 3:34 PM Sean Owen <[hidden email]> wrote:

>
> BTW does anyone know why there are two PR builder jobs? I'm confused
> about why different ones would execute.
>
> Yes I see NewSparkPullRequestBuilder failing on a variety of PRs.
> I don't think it has anything to do with Hive; these PRs touch
> different parts of code but all not related to this failure.
>
> On Wed, Dec 4, 2019 at 12:40 PM Dongjoon Hyun <[hidden email]> wrote:
> >
> > Hi, Sean.
> >
> > It seems that there is no failure on your other SQL PR.
> >
> >     https://github.com/apache/spark/pull/26748
> >
> > Does the sequential failure happen only at `NewSparkPullRequestBuilder`?
> > Since `NewSparkPullRequestBuilder` is not the same with `SparkPullRequestBuilder`,
> > there might be a root cause inside it if it happens only at `NewSparkPullRequestBuilder`.
> >
> > For `org.apache.hive.service.ServiceException: Failed to Start HiveServer2`,
> > I've observed them before, but the root cause might be different from this one.
> >
> > BTW, to reduce the scope of investigation, could you try with `[hive-1.2]` tag in your PR?
> >
> > Bests,
> > Dongjoon.
> >
> >
> > On Wed, Dec 4, 2019 at 6:29 AM Sean Owen <[hidden email]> wrote:
> >>
> >> I'm seeing consistent failures in the PR builder when touching SQL code:
> >>
> >> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/
> >>
> >>  org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's own GetSchemasOperation(SparkGetSchemasOperation)14 ms2
> >>  org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It is not a test it is a sbt.testing.SuiteSelector)
> >>
> >> Looks like this has failed about 6 builds in the past few days. Has anyone seen this / has a clue what's causing it? errors are like ...
> >>
> >> java.sql.SQLException: No suitable driver found for jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue
> >>
> >>
> >> Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not org.apache.hadoop.hive.metastore.MetaStoreFilterHook



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: SQL test failures in PR builder?

Sean Owen-2
Hm, so they look pretty similar except for minor differences in the
actual script run. Is there any reason this should be different? Would
it be reasonable to try making the 'new' one work like the 'old' one
if the former isn't working?

But I still can't figure out why it causes the same odd error every
time on this one PR, which is a minor change to tooltips in the UI. I
haven't seen other manually-triggered PR builds fail this way. Really
mysterious so far!

https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4964/testReport/


Old:

#!/bin/bash

set -e  # fail on any non-zero exit code
set -x

export AMPLAB_JENKINS=1
export PATH="$PATH:/home/anaconda/envs/py3k/bin"

# Prepend JAVA_HOME/bin to fix issue where Zinc's embedded SBT
incremental compiler seems to
# ignore our JAVA_HOME and use the system javac instead.
export PATH="$JAVA_HOME/bin:$PATH"

# Add a pre-downloaded version of Maven to the path so that we avoid
the flaky download step.
export PATH="/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/:$PATH"

echo "fixing target dir permissions"
chmod -R +w target/* || true  # stupid hack by sknapp to ensure that
the chmod always exits w/0 and doesn't bork the script

echo "running git clean -fdx"
git clean -fdx

# Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention
export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER"
mkdir -p "$HOME"
export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2"
export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2"


./dev/run-tests-jenkins


# Hack to ensure that at least one JVM suite always runs in order to
prevent spurious errors from the
# Jenkins JUnit test reporter plugin
./build/sbt unsafe/test > /dev/null 2>&1



New:

#!/bin/bash

set -e
export AMPLAB_JENKINS=1
export PATH="$PATH:/home/anaconda/envs/py3k/bin"
git clean -fdx

# Prepend JAVA_HOME/bin to fix issue where Zinc's embedded SBT
incremental compiler seems to
# ignore our JAVA_HOME and use the system javac instead.
export PATH="$JAVA_HOME/bin:$PATH"

# Add a pre-downloaded version of Maven to the path so that we avoid
the flaky download step.
export PATH="/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/:$PATH"

# Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention
export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER"
mkdir -p "$HOME"
export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2"
export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2"

# This is required for tests of backport patches.
# We need to download the run-tests-codes.sh file because it's
imported by run-tests-jenkins.
# When running tests on branch-1.0 (and earlier), the older version of
run-tests won't set CURRENT_BLOCK, so
# the Jenkins scripts will report all failures as "some tests failed"
rather than a more specific
# error message.
if [ ! -f "dev/run-tests-jenkins" ]; then
  wget https://raw.githubusercontent.com/apache/spark/master/dev/run-tests-jenkins
  wget https://raw.githubusercontent.com/apache/spark/master/dev/run-tests-codes.sh
  mv run-tests-jenkins dev/
  mv run-tests-codes.sh dev/
  chmod 755 dev/run-tests-jenkins
  chmod 755 dev/run-tests-codes.sh
fi

./dev/run-tests-jenkins


On Wed, Dec 4, 2019 at 5:53 PM Shane Knapp <[hidden email]> wrote:

>
> ++yin huai for more insight in to the NewSparkPullRequestBuilder job...
>
> tbh, i never (or still) really understand the exact use for that job,
> except that it's triggered by https://spark-prs.appspot.com/
>
> shane
>
>
> On Wed, Dec 4, 2019 at 3:34 PM Sean Owen <[hidden email]> wrote:
> >
> > BTW does anyone know why there are two PR builder jobs? I'm confused
> > about why different ones would execute.
> >
> > Yes I see NewSparkPullRequestBuilder failing on a variety of PRs.
> > I don't think it has anything to do with Hive; these PRs touch
> > different parts of code but all not related to this failure.
> >
> > On Wed, Dec 4, 2019 at 12:40 PM Dongjoon Hyun <[hidden email]> wrote:
> > >
> > > Hi, Sean.
> > >
> > > It seems that there is no failure on your other SQL PR.
> > >
> > >     https://github.com/apache/spark/pull/26748
> > >
> > > Does the sequential failure happen only at `NewSparkPullRequestBuilder`?
> > > Since `NewSparkPullRequestBuilder` is not the same with `SparkPullRequestBuilder`,
> > > there might be a root cause inside it if it happens only at `NewSparkPullRequestBuilder`.
> > >
> > > For `org.apache.hive.service.ServiceException: Failed to Start HiveServer2`,
> > > I've observed them before, but the root cause might be different from this one.
> > >
> > > BTW, to reduce the scope of investigation, could you try with `[hive-1.2]` tag in your PR?
> > >
> > > Bests,
> > > Dongjoon.
> > >
> > >
> > > On Wed, Dec 4, 2019 at 6:29 AM Sean Owen <[hidden email]> wrote:
> > >>
> > >> I'm seeing consistent failures in the PR builder when touching SQL code:
> > >>
> > >> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/
> > >>
> > >>  org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's own GetSchemasOperation(SparkGetSchemasOperation)14 ms2
> > >>  org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It is not a test it is a sbt.testing.SuiteSelector)
> > >>
> > >> Looks like this has failed about 6 builds in the past few days. Has anyone seen this / has a clue what's causing it? errors are like ...
> > >>
> > >> java.sql.SQLException: No suitable driver found for jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue
> > >>
> > >>
> > >> Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not org.apache.hadoop.hive.metastore.MetaStoreFilterHook
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: SQL test failures in PR builder?

shane knapp
yeah, totally weird.

i'm actually going to take this moment and clean up the build scripts
for both of these jobs.  there's a lot of years-old cruft that i'll
delete and make things more readable.

On Sun, Dec 8, 2019 at 7:50 PM Sean Owen <[hidden email]> wrote:

>
> Hm, so they look pretty similar except for minor differences in the
> actual script run. Is there any reason this should be different? Would
> it be reasonable to try making the 'new' one work like the 'old' one
> if the former isn't working?
>
> But I still can't figure out why it causes the same odd error every
> time on this one PR, which is a minor change to tooltips in the UI. I
> haven't seen other manually-triggered PR builds fail this way. Really
> mysterious so far!
>
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4964/testReport/
>
>
> Old:
>
> #!/bin/bash
>
> set -e  # fail on any non-zero exit code
> set -x
>
> export AMPLAB_JENKINS=1
> export PATH="$PATH:/home/anaconda/envs/py3k/bin"
>
> # Prepend JAVA_HOME/bin to fix issue where Zinc's embedded SBT
> incremental compiler seems to
> # ignore our JAVA_HOME and use the system javac instead.
> export PATH="$JAVA_HOME/bin:$PATH"
>
> # Add a pre-downloaded version of Maven to the path so that we avoid
> the flaky download step.
> export PATH="/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/:$PATH"
>
> echo "fixing target dir permissions"
> chmod -R +w target/* || true  # stupid hack by sknapp to ensure that
> the chmod always exits w/0 and doesn't bork the script
>
> echo "running git clean -fdx"
> git clean -fdx
>
> # Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention
> export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER"
> mkdir -p "$HOME"
> export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2"
> export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2"
>
>
> ./dev/run-tests-jenkins
>
>
> # Hack to ensure that at least one JVM suite always runs in order to
> prevent spurious errors from the
> # Jenkins JUnit test reporter plugin
> ./build/sbt unsafe/test > /dev/null 2>&1
>
>
>
> New:
>
> #!/bin/bash
>
> set -e
> export AMPLAB_JENKINS=1
> export PATH="$PATH:/home/anaconda/envs/py3k/bin"
> git clean -fdx
>
> # Prepend JAVA_HOME/bin to fix issue where Zinc's embedded SBT
> incremental compiler seems to
> # ignore our JAVA_HOME and use the system javac instead.
> export PATH="$JAVA_HOME/bin:$PATH"
>
> # Add a pre-downloaded version of Maven to the path so that we avoid
> the flaky download step.
> export PATH="/home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.3.9/bin/:$PATH"
>
> # Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention
> export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER"
> mkdir -p "$HOME"
> export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2"
> export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2"
>
> # This is required for tests of backport patches.
> # We need to download the run-tests-codes.sh file because it's
> imported by run-tests-jenkins.
> # When running tests on branch-1.0 (and earlier), the older version of
> run-tests won't set CURRENT_BLOCK, so
> # the Jenkins scripts will report all failures as "some tests failed"
> rather than a more specific
> # error message.
> if [ ! -f "dev/run-tests-jenkins" ]; then
>   wget https://raw.githubusercontent.com/apache/spark/master/dev/run-tests-jenkins
>   wget https://raw.githubusercontent.com/apache/spark/master/dev/run-tests-codes.sh
>   mv run-tests-jenkins dev/
>   mv run-tests-codes.sh dev/
>   chmod 755 dev/run-tests-jenkins
>   chmod 755 dev/run-tests-codes.sh
> fi
>
> ./dev/run-tests-jenkins
>
>
> On Wed, Dec 4, 2019 at 5:53 PM Shane Knapp <[hidden email]> wrote:
> >
> > ++yin huai for more insight in to the NewSparkPullRequestBuilder job...
> >
> > tbh, i never (or still) really understand the exact use for that job,
> > except that it's triggered by https://spark-prs.appspot.com/
> >
> > shane
> >
> >
> > On Wed, Dec 4, 2019 at 3:34 PM Sean Owen <[hidden email]> wrote:
> > >
> > > BTW does anyone know why there are two PR builder jobs? I'm confused
> > > about why different ones would execute.
> > >
> > > Yes I see NewSparkPullRequestBuilder failing on a variety of PRs.
> > > I don't think it has anything to do with Hive; these PRs touch
> > > different parts of code but all not related to this failure.
> > >
> > > On Wed, Dec 4, 2019 at 12:40 PM Dongjoon Hyun <[hidden email]> wrote:
> > > >
> > > > Hi, Sean.
> > > >
> > > > It seems that there is no failure on your other SQL PR.
> > > >
> > > >     https://github.com/apache/spark/pull/26748
> > > >
> > > > Does the sequential failure happen only at `NewSparkPullRequestBuilder`?
> > > > Since `NewSparkPullRequestBuilder` is not the same with `SparkPullRequestBuilder`,
> > > > there might be a root cause inside it if it happens only at `NewSparkPullRequestBuilder`.
> > > >
> > > > For `org.apache.hive.service.ServiceException: Failed to Start HiveServer2`,
> > > > I've observed them before, but the root cause might be different from this one.
> > > >
> > > > BTW, to reduce the scope of investigation, could you try with `[hive-1.2]` tag in your PR?
> > > >
> > > > Bests,
> > > > Dongjoon.
> > > >
> > > >
> > > > On Wed, Dec 4, 2019 at 6:29 AM Sean Owen <[hidden email]> wrote:
> > > >>
> > > >> I'm seeing consistent failures in the PR builder when touching SQL code:
> > > >>
> > > >> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/
> > > >>
> > > >>  org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's own GetSchemasOperation(SparkGetSchemasOperation)14 ms2
> > > >>  org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It is not a test it is a sbt.testing.SuiteSelector)
> > > >>
> > > >> Looks like this has failed about 6 builds in the past few days. Has anyone seen this / has a clue what's causing it? errors are like ...
> > > >>
> > > >> java.sql.SQLException: No suitable driver found for jdbc:hive2://localhost:13694/?a=avalue;b=bvalue#c=cvalue;d=dvalue
> > > >>
> > > >>
> > > >> Caused by: sbt.ForkMain$ForkError: java.lang.RuntimeException: class org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl not org.apache.hadoop.hive.metastore.MetaStoreFilterHook
> >
> >
> >
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]