Spark master build hangs using parallel build option in maven

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark master build hangs using parallel build option in maven

Saurabh Chawla
Hi All,

Spark master build hangs using parallel build option in maven. On running build the sequentially on spark master using maven, build did not hang. This issue occurs on giving hadoop-provided (-Phadoop-provided -Dhadoop.version=2.8.5) option. Same command works fine to build spark-2.4.3 parallelly

Command to build spark master sequentially - Spark build works fine
build/mvn  -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true  clean package

Command to build spark master parallel - spark build hangs
build/mvn -X -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true -T 4 clean package

This is the trace which keeps on repeating in maven logs

[DEBUG] building maven31 dependency graph for org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT with Maven31DependencyGraphBuilder
[DEBUG] Dependency collection stats: {ConflictMarker.analyzeTime=60583, ConflictMarker.markTime=23750, ConflictMarker.nodeCount=419, ConflictIdSorter.graphTime=41262, ConflictIdSorter.topsortTime=9704, ConflictIdSorter.conflictIdCount=105, ConflictIdSorter.conflictIdCycleCount=0, ConflictResolver.totalTime=632542, ConflictResolver.conflictItemCount=193, DefaultDependencyCollector.collectTime=1020759, DefaultDependencyCollector.transformTime=775495}
[DEBUG] org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT
[DEBUG]    org.apache.spark:spark-network-shuffle_2.12:jar:3.0.0-SNAPSHOT:compile
[DEBUG]       org.apache.spark:spark-network-common_2.12:jar:3.0.0-SNAPSHOT:compile
[DEBUG]          io.netty:netty-all:jar:4.1.42.Final:compile (version managed from 4.1.42.Final)
[DEBUG]          org.apache.commons:commons-lang3:jar:3.9:compile (version managed from 3.9)
[DEBUG]          org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile (version managed from 1.8)
[DEBUG]          com.fasterxml.jackson.core:jackson-databind:jar:2.10.0:compile (version managed from 2.10.0)
[DEBUG]             com.fasterxml.jackson.core:jackson-core:jar:2.10.0:compile (version managed from 2.10.0)
[DEBUG]          com.fasterxml.jackson.core:jackson-annotations:jar:2.10.0:compile (version managed from 2.10.0)
[DEBUG]          com.google.code.findbugs:jsr305:jar:3.0.0:compile (version managed from 3.0.0)
[DEBUG]          com.google.guava:guava:jar:14.0.1:provided (scope managed from compile) (version managed from 14.0.1)
[DEBUG]          org.apache.commons:commons-crypto:jar:1.0.0:compile (version managed from 1.0.0) (exclusions managed from [net.java.dev.jna:jna:*:*])
[DEBUG]       io.dropwizard.metrics:metrics-core:jar:4.1.1:compile (version managed from 4.1.1)
[DEBUG]    org.apache.spark:spark-tags_2.12:jar:3.0.0-SNAPSHOT:test
[DEBUG]       org.scala-lang:scala-library:jar:2.12.10:compile (version managed from 2.12.10)
[DEBUG]    org.apache.spark:spark-tags_2.12:jar:tests:3.0.0-SNAPSHOT:test
[DEBUG]    org.apache.hadoop:hadoop-client:jar:2.8.5:provided (exclusions managed from [org.fusesource.leveldbjni:leveldbjni-all:*:*, asm:asm:*:*, org.codehaus.jackson:jackson-mapper-asl:*:*, org.ow2.asm:asm:*:*, org.jboss.netty:netty:*:*, io.netty:netty:*:*, commons-beanutils:commons-beanutils-core:*:*, commons-logging:commons-logging:*:*, org.mockito:mockito-all:*:*, org.mortbay.jetty:servlet-api-2.5:*:*, javax.servlet:servlet-api:*:*, junit:junit:*:*, com.sun.jersey:*:*:*, com.sun.jersey.jersey-test-framework:*:*:*, com.sun.jersey.contribs:*:*:*, net.java.dev.jets3t:jets3t:*:*, javax.ws.rs:jsr311-api:*:*, org.eclipse.jetty:jetty-webapp:*:*])
[DEBUG]       org.apache.hadoop:hadoop-common:jar:2.8.5:provided
[DEBUG]          com.hadoop.gplcompression:hadoop-lzo:jar:0.4.19:provided
[DEBUG]          commons-cli:commons-cli:jar:1.2:provided
[DEBUG]          org.apache.commons:commons-math3:jar:3.4.1:provided (version managed from 3.1.1)
[DEBUG]          org.apache.httpcomponents:httpclient:jar:4.5.6:provided (version managed from 4.5.2)
[DEBUG]             org.apache.httpcomponents:httpcore:jar:4.4.12:provided (version managed from 4.4.10)
[DEBUG]          commons-codec:commons-codec:jar:1.10:provided (version managed from 1.11)
[DEBUG]          commons-io:commons-io:jar:2.4:provided (version managed from 2.5)
[DEBUG]          commons-net:commons-net:jar:3.1:provided (version managed from 3.6)
[DEBUG]          commons-collections:commons-collections:jar:3.2.2:provided (version managed from 3.2.2)
[DEBUG]          org.eclipse.jetty:jetty-servlet:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.3.19.v20170502)
[DEBUG]             org.eclipse.jetty:jetty-security:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.4.18.v20190429)
[DEBUG]          javax.servlet.jsp:jsp-api:jar:2.1:provided
[DEBUG]          log4j:log4j:jar:1.2.17:provided (scope managed from compile) (version managed from 1.2.17)
[DEBUG]          commons-lang:commons-lang:jar:2.6:provided (version managed from 2.6)
[DEBUG]          commons-beanutils:commons-beanutils:jar:1.9.4:provided (version managed from 1.9.3)
[DEBUG]          org.apache.commons:commons-configuration2:jar:2.1.1:provided
[DEBUG]          org.apache.avro:avro:jar:1.8.2:provided (version managed from 1.7.7)
[DEBUG]             org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided (scope managed from compile) (version managed from 1.9.13)
[DEBUG]             com.thoughtworks.paranamer:paranamer:jar:2.8:runtime (scope managed from compile) (version managed from 2.7)
[DEBUG]             org.xerial.snappy:snappy-java:jar:1.1.7.3:provided (version managed from 1.1.1.3)
[DEBUG]             org.tukaani:xz:jar:1.5:provided
[DEBUG]          com.google.re2j:re2j:jar:1.1:provided
[DEBUG]          com.google.protobuf:protobuf-java:jar:2.5.0:provided (scope managed from compile) (version managed from 2.5.0)
[DEBUG]          com.google.code.gson:gson:jar:2.2.4:provided
[DEBUG]          org.apache.hadoop:hadoop-auth:jar:2.8.5:provided
[DEBUG]             com.nimbusds:nimbus-jose-jwt:jar:4.41.1:provided
[DEBUG]                com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:provided
[DEBUG]             net.minidev:json-smart:jar:2.3:provided
[DEBUG]                net.minidev:accessors-smart:jar:1.2:provided
[DEBUG]             org.apache.curator:curator-framework:jar:2.7.1:provided (version managed from 2.12.0)
[DEBUG]          org.apache.curator:curator-client:jar:2.7.1:provided (version managed from 2.12.0)
[DEBUG]          org.apache.curator:curator-recipes:jar:2.7.1:provided (scope managed from compile) (version managed from 2.12.0) (exclusions managed from [])
[DEBUG]          org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
[DEBUG]          org.apache.commons:commons-compress:jar:1.4.1:provided
[DEBUG]          org.apache.kerby:kerb-simplekdc:jar:1.0.1:provided
[DEBUG]             org.apache.kerby:kerb-client:jar:1.0.1:provided
[DEBUG]                org.apache.kerby:kerby-config:jar:1.0.1:provided
[DEBUG]                org.apache.kerby:kerb-core:jar:1.0.1:provided
[DEBUG]                   org.apache.kerby:kerby-pkix:jar:1.0.1:provided
[DEBUG]                      org.apache.kerby:kerby-asn1:jar:1.0.1:provided
[DEBUG]                      org.apache.kerby:kerby-util:jar:1.0.1:provided
[DEBUG]                org.apache.kerby:kerb-common:jar:1.0.1:provided
[DEBUG]                   org.apache.kerby:kerb-crypto:jar:1.0.1:provided
[DEBUG]                org.apache.kerby:kerb-util:jar:1.0.1:provided
[DEBUG]                org.apache.kerby:token-provider:jar:1.0.1:provided
[DEBUG]             org.apache.kerby:kerb-admin:jar:1.0.1:provided
[DEBUG]                org.apache.kerby:kerb-server:jar:1.0.1:provided
[DEBUG]                   org.apache.kerby:kerb-identity:jar:1.0.1:provided
[DEBUG]                org.apache.kerby:kerby-xdr:jar:1.0.1:provided
[DEBUG]          org.codehaus.woodstox:stax2-api:jar:3.1.4:provided
[DEBUG]          com.fasterxml.woodstox:woodstox-core:jar:5.0.3:provided
[DEBUG]       org.apache.hadoop:hadoop-hdfs-client:jar:2.8.5:provided
[DEBUG]          com.squareup.okhttp:okhttp:jar:2.7.5:provided
[DEBUG]             com.squareup.okio:okio:jar:1.6.0:provided
[DEBUG]       org.apache.hadoop:hadoop-yarn-api:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [org.apache.hadoop:hadoop-annotations:*:*, com.google.inject:guice:*:*, com.sun.jersey.jersey-test-framework:jersey-test-framework-grizzly2:*:*, com.sun.jersey:jersey-server:*:*, com.sun.jersey.contribs:jersey-guice:*:*, com.google.inject.extensions:guice-servlet:*:*, org.apache.avro:avro:*:*, com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-json:*:*, io.netty:netty:*:*])
[DEBUG]          javax.xml.bind:jaxb-api:jar:2.2.11:provided
[DEBUG]          com.googlecode.json-simple:json-simple:jar:1.1.1:provided
[DEBUG]       org.apache.hadoop:hadoop-yarn-client:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [org.apache.hadoop:hadoop-yarn-api:*:*, org.apache.hadoop:hadoop-yarn-common:*:*, org.apache.hadoop:hadoop-annotations:*:*, com.google.guava:guava:*:*, commons-cli:commons-cli:*:*, log4j:log4j:*:*, com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-server:*:*, com.sun.jersey:jersey-json:*:*, com.sun.jersey:jersey-servlet:*:*, io.netty:netty:*:*, com.google.inject.extensions:guice-servlet:*:*])
[DEBUG]       org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.8.5:provided
[DEBUG]          org.apache.hadoop:hadoop-yarn-common:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [])
[DEBUG]             javax.servlet:javax.servlet-api:jar:3.1.0:provided
[DEBUG]             org.eclipse.jetty:jetty-util:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.3.19.v20170502)
[DEBUG]             com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.10.0:provided (version managed from 2.7.8)
[DEBUG]                jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:provided
[DEBUG]                jakarta.activation:jakarta.activation-api:jar:1.2.1:provided
[DEBUG]             com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.7.8:provided
[DEBUG]                com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.7.8:provided
[DEBUG]       org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.8.5:provided
[DEBUG]          org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.8.5:provided
[DEBUG]       org.apache.hadoop:hadoop-annotations:jar:2.8.5:provided
[DEBUG]    org.slf4j:slf4j-api:jar:1.7.16:provided
[DEBUG]    org.spark-project.spark:unused:jar:1.0.0:compile
[DEBUG]    org.scalatest:scalatest_2.12:jar:3.0.8:test
[DEBUG]       org.scalactic:scalactic_2.12:jar:3.0.8:test
[DEBUG]       org.scala-lang:scala-reflect:jar:2.12.10:test (version managed from 2.12.8)
[DEBUG]       org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:test (version managed from 1.2.0)
[DEBUG]    junit:junit:jar:4.12:test
[DEBUG]       org.hamcrest:hamcrest-core:jar:1.3:test (scope managed from compile) (version managed from 1.3)
[DEBUG]    com.novocode:junit-interface:jar:0.11:test
[DEBUG]       org.scala-sbt:test-interface:jar:1.0:test
[DEBUG] updateExcludesInDeps()

I'll be very grateful if someone could help.

Regards
Saurabh Chawla

Reply | Threaded
Open this post in threaded view
|

Re: Spark master build hangs using parallel build option in maven

Sean Owen-2
I don't believe you can use a parallel build indeed. Some things
collide with each other. Some of the suites are run in parallel inside
the build though already.

On Fri, Jan 17, 2020 at 1:23 PM Saurabh Chawla <[hidden email]> wrote:

>
> Hi All,
>
> Spark master build hangs using parallel build option in maven. On running build the sequentially on spark master using maven, build did not hang. This issue occurs on giving hadoop-provided (-Phadoop-provided -Dhadoop.version=2.8.5) option. Same command works fine to build spark-2.4.3 parallelly
>
> Command to build spark master sequentially - Spark build works fine
> build/mvn  -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true  clean package
>
> Command to build spark master parallel - spark build hangs
> build/mvn -X -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true -T 4 clean package
>
> This is the trace which keeps on repeating in maven logs
>
> [DEBUG] building maven31 dependency graph for org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT with Maven31DependencyGraphBuilder
> [DEBUG] Dependency collection stats: {ConflictMarker.analyzeTime=60583, ConflictMarker.markTime=23750, ConflictMarker.nodeCount=419, ConflictIdSorter.graphTime=41262, ConflictIdSorter.topsortTime=9704, ConflictIdSorter.conflictIdCount=105, ConflictIdSorter.conflictIdCycleCount=0, ConflictResolver.totalTime=632542, ConflictResolver.conflictItemCount=193, DefaultDependencyCollector.collectTime=1020759, DefaultDependencyCollector.transformTime=775495}
> [DEBUG] org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT
> [DEBUG]    org.apache.spark:spark-network-shuffle_2.12:jar:3.0.0-SNAPSHOT:compile
> [DEBUG]       org.apache.spark:spark-network-common_2.12:jar:3.0.0-SNAPSHOT:compile
> [DEBUG]          io.netty:netty-all:jar:4.1.42.Final:compile (version managed from 4.1.42.Final)
> [DEBUG]          org.apache.commons:commons-lang3:jar:3.9:compile (version managed from 3.9)
> [DEBUG]          org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile (version managed from 1.8)
> [DEBUG]          com.fasterxml.jackson.core:jackson-databind:jar:2.10.0:compile (version managed from 2.10.0)
> [DEBUG]             com.fasterxml.jackson.core:jackson-core:jar:2.10.0:compile (version managed from 2.10.0)
> [DEBUG]          com.fasterxml.jackson.core:jackson-annotations:jar:2.10.0:compile (version managed from 2.10.0)
> [DEBUG]          com.google.code.findbugs:jsr305:jar:3.0.0:compile (version managed from 3.0.0)
> [DEBUG]          com.google.guava:guava:jar:14.0.1:provided (scope managed from compile) (version managed from 14.0.1)
> [DEBUG]          org.apache.commons:commons-crypto:jar:1.0.0:compile (version managed from 1.0.0) (exclusions managed from [net.java.dev.jna:jna:*:*])
> [DEBUG]       io.dropwizard.metrics:metrics-core:jar:4.1.1:compile (version managed from 4.1.1)
> [DEBUG]    org.apache.spark:spark-tags_2.12:jar:3.0.0-SNAPSHOT:test
> [DEBUG]       org.scala-lang:scala-library:jar:2.12.10:compile (version managed from 2.12.10)
> [DEBUG]    org.apache.spark:spark-tags_2.12:jar:tests:3.0.0-SNAPSHOT:test
> [DEBUG]    org.apache.hadoop:hadoop-client:jar:2.8.5:provided (exclusions managed from [org.fusesource.leveldbjni:leveldbjni-all:*:*, asm:asm:*:*, org.codehaus.jackson:jackson-mapper-asl:*:*, org.ow2.asm:asm:*:*, org.jboss.netty:netty:*:*, io.netty:netty:*:*, commons-beanutils:commons-beanutils-core:*:*, commons-logging:commons-logging:*:*, org.mockito:mockito-all:*:*, org.mortbay.jetty:servlet-api-2.5:*:*, javax.servlet:servlet-api:*:*, junit:junit:*:*, com.sun.jersey:*:*:*, com.sun.jersey.jersey-test-framework:*:*:*, com.sun.jersey.contribs:*:*:*, net.java.dev.jets3t:jets3t:*:*, javax.ws.rs:jsr311-api:*:*, org.eclipse.jetty:jetty-webapp:*:*])
> [DEBUG]       org.apache.hadoop:hadoop-common:jar:2.8.5:provided
> [DEBUG]          com.hadoop.gplcompression:hadoop-lzo:jar:0.4.19:provided
> [DEBUG]          commons-cli:commons-cli:jar:1.2:provided
> [DEBUG]          org.apache.commons:commons-math3:jar:3.4.1:provided (version managed from 3.1.1)
> [DEBUG]          org.apache.httpcomponents:httpclient:jar:4.5.6:provided (version managed from 4.5.2)
> [DEBUG]             org.apache.httpcomponents:httpcore:jar:4.4.12:provided (version managed from 4.4.10)
> [DEBUG]          commons-codec:commons-codec:jar:1.10:provided (version managed from 1.11)
> [DEBUG]          commons-io:commons-io:jar:2.4:provided (version managed from 2.5)
> [DEBUG]          commons-net:commons-net:jar:3.1:provided (version managed from 3.6)
> [DEBUG]          commons-collections:commons-collections:jar:3.2.2:provided (version managed from 3.2.2)
> [DEBUG]          org.eclipse.jetty:jetty-servlet:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.3.19.v20170502)
> [DEBUG]             org.eclipse.jetty:jetty-security:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.4.18.v20190429)
> [DEBUG]          javax.servlet.jsp:jsp-api:jar:2.1:provided
> [DEBUG]          log4j:log4j:jar:1.2.17:provided (scope managed from compile) (version managed from 1.2.17)
> [DEBUG]          commons-lang:commons-lang:jar:2.6:provided (version managed from 2.6)
> [DEBUG]          commons-beanutils:commons-beanutils:jar:1.9.4:provided (version managed from 1.9.3)
> [DEBUG]          org.apache.commons:commons-configuration2:jar:2.1.1:provided
> [DEBUG]          org.apache.avro:avro:jar:1.8.2:provided (version managed from 1.7.7)
> [DEBUG]             org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided (scope managed from compile) (version managed from 1.9.13)
> [DEBUG]             com.thoughtworks.paranamer:paranamer:jar:2.8:runtime (scope managed from compile) (version managed from 2.7)
> [DEBUG]             org.xerial.snappy:snappy-java:jar:1.1.7.3:provided (version managed from 1.1.1.3)
> [DEBUG]             org.tukaani:xz:jar:1.5:provided
> [DEBUG]          com.google.re2j:re2j:jar:1.1:provided
> [DEBUG]          com.google.protobuf:protobuf-java:jar:2.5.0:provided (scope managed from compile) (version managed from 2.5.0)
> [DEBUG]          com.google.code.gson:gson:jar:2.2.4:provided
> [DEBUG]          org.apache.hadoop:hadoop-auth:jar:2.8.5:provided
> [DEBUG]             com.nimbusds:nimbus-jose-jwt:jar:4.41.1:provided
> [DEBUG]                com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:provided
> [DEBUG]             net.minidev:json-smart:jar:2.3:provided
> [DEBUG]                net.minidev:accessors-smart:jar:1.2:provided
> [DEBUG]             org.apache.curator:curator-framework:jar:2.7.1:provided (version managed from 2.12.0)
> [DEBUG]          org.apache.curator:curator-client:jar:2.7.1:provided (version managed from 2.12.0)
> [DEBUG]          org.apache.curator:curator-recipes:jar:2.7.1:provided (scope managed from compile) (version managed from 2.12.0) (exclusions managed from [])
> [DEBUG]          org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
> [DEBUG]          org.apache.commons:commons-compress:jar:1.4.1:provided
> [DEBUG]          org.apache.kerby:kerb-simplekdc:jar:1.0.1:provided
> [DEBUG]             org.apache.kerby:kerb-client:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerby-config:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-core:jar:1.0.1:provided
> [DEBUG]                   org.apache.kerby:kerby-pkix:jar:1.0.1:provided
> [DEBUG]                      org.apache.kerby:kerby-asn1:jar:1.0.1:provided
> [DEBUG]                      org.apache.kerby:kerby-util:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-common:jar:1.0.1:provided
> [DEBUG]                   org.apache.kerby:kerb-crypto:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-util:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:token-provider:jar:1.0.1:provided
> [DEBUG]             org.apache.kerby:kerb-admin:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-server:jar:1.0.1:provided
> [DEBUG]                   org.apache.kerby:kerb-identity:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerby-xdr:jar:1.0.1:provided
> [DEBUG]          org.codehaus.woodstox:stax2-api:jar:3.1.4:provided
> [DEBUG]          com.fasterxml.woodstox:woodstox-core:jar:5.0.3:provided
> [DEBUG]       org.apache.hadoop:hadoop-hdfs-client:jar:2.8.5:provided
> [DEBUG]          com.squareup.okhttp:okhttp:jar:2.7.5:provided
> [DEBUG]             com.squareup.okio:okio:jar:1.6.0:provided
> [DEBUG]       org.apache.hadoop:hadoop-yarn-api:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [org.apache.hadoop:hadoop-annotations:*:*, com.google.inject:guice:*:*, com.sun.jersey.jersey-test-framework:jersey-test-framework-grizzly2:*:*, com.sun.jersey:jersey-server:*:*, com.sun.jersey.contribs:jersey-guice:*:*, com.google.inject.extensions:guice-servlet:*:*, org.apache.avro:avro:*:*, com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-json:*:*, io.netty:netty:*:*])
> [DEBUG]          javax.xml.bind:jaxb-api:jar:2.2.11:provided
> [DEBUG]          com.googlecode.json-simple:json-simple:jar:1.1.1:provided
> [DEBUG]       org.apache.hadoop:hadoop-yarn-client:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [org.apache.hadoop:hadoop-yarn-api:*:*, org.apache.hadoop:hadoop-yarn-common:*:*, org.apache.hadoop:hadoop-annotations:*:*, com.google.guava:guava:*:*, commons-cli:commons-cli:*:*, log4j:log4j:*:*, com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-server:*:*, com.sun.jersey:jersey-json:*:*, com.sun.jersey:jersey-servlet:*:*, io.netty:netty:*:*, com.google.inject.extensions:guice-servlet:*:*])
> [DEBUG]       org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.8.5:provided
> [DEBUG]          org.apache.hadoop:hadoop-yarn-common:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [])
> [DEBUG]             javax.servlet:javax.servlet-api:jar:3.1.0:provided
> [DEBUG]             org.eclipse.jetty:jetty-util:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.3.19.v20170502)
> [DEBUG]             com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.10.0:provided (version managed from 2.7.8)
> [DEBUG]                jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:provided
> [DEBUG]                jakarta.activation:jakarta.activation-api:jar:1.2.1:provided
> [DEBUG]             com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.7.8:provided
> [DEBUG]                com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.7.8:provided
> [DEBUG]       org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.8.5:provided
> [DEBUG]          org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.8.5:provided
> [DEBUG]       org.apache.hadoop:hadoop-annotations:jar:2.8.5:provided
> [DEBUG]    org.slf4j:slf4j-api:jar:1.7.16:provided
> [DEBUG]    org.spark-project.spark:unused:jar:1.0.0:compile
> [DEBUG]    org.scalatest:scalatest_2.12:jar:3.0.8:test
> [DEBUG]       org.scalactic:scalactic_2.12:jar:3.0.8:test
> [DEBUG]       org.scala-lang:scala-reflect:jar:2.12.10:test (version managed from 2.12.8)
> [DEBUG]       org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:test (version managed from 1.2.0)
> [DEBUG]    junit:junit:jar:4.12:test
> [DEBUG]       org.hamcrest:hamcrest-core:jar:1.3:test (scope managed from compile) (version managed from 1.3)
> [DEBUG]    com.novocode:junit-interface:jar:0.11:test
> [DEBUG]       org.scala-sbt:test-interface:jar:1.0:test
> [DEBUG] updateExcludesInDeps()
>
> I'll be very grateful if someone could help.
>
> Regards
> Saurabh Chawla
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Spark master build hangs using parallel build option in maven

Saurabh Chawla
Hi Sean,

Thanks for checking this.

I am able to see parallel build info in the readme file https://github.com/apache/spark#building-spark

"
You can build Spark using more than one thread by using the -T option with Maven, see "Parallel builds in Maven 3". More detailed documentation is available from the project site, at "Building Spark".
"

This used to work while building older version of spark(2.4.3, 2.3.2 etc).
build/mvn -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true -T 4 clean package

Also I have seen the maven version is changed from 3.5.4 to 3.6.3 in master branch compared to spark 2.4.3.
Not sure if it's due to some bug in maven version used in master or some new change added in the master branch that prevent the parallel build.

Regards
Saurabh Chawla
 

On Sat, Jan 18, 2020 at 2:19 AM Sean Owen <[hidden email]> wrote:
I don't believe you can use a parallel build indeed. Some things
collide with each other. Some of the suites are run in parallel inside
the build though already.

On Fri, Jan 17, 2020 at 1:23 PM Saurabh Chawla <[hidden email]> wrote:
>
> Hi All,
>
> Spark master build hangs using parallel build option in maven. On running build the sequentially on spark master using maven, build did not hang. This issue occurs on giving hadoop-provided (-Phadoop-provided -Dhadoop.version=2.8.5) option. Same command works fine to build spark-2.4.3 parallelly
>
> Command to build spark master sequentially - Spark build works fine
> build/mvn  -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true  clean package
>
> Command to build spark master parallel - spark build hangs
> build/mvn -X -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true -T 4 clean package
>
> This is the trace which keeps on repeating in maven logs
>
> [DEBUG] building maven31 dependency graph for org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT with Maven31DependencyGraphBuilder
> [DEBUG] Dependency collection stats: {ConflictMarker.analyzeTime=60583, ConflictMarker.markTime=23750, ConflictMarker.nodeCount=419, ConflictIdSorter.graphTime=41262, ConflictIdSorter.topsortTime=9704, ConflictIdSorter.conflictIdCount=105, ConflictIdSorter.conflictIdCycleCount=0, ConflictResolver.totalTime=632542, ConflictResolver.conflictItemCount=193, DefaultDependencyCollector.collectTime=1020759, DefaultDependencyCollector.transformTime=775495}
> [DEBUG] org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT
> [DEBUG]    org.apache.spark:spark-network-shuffle_2.12:jar:3.0.0-SNAPSHOT:compile
> [DEBUG]       org.apache.spark:spark-network-common_2.12:jar:3.0.0-SNAPSHOT:compile
> [DEBUG]          io.netty:netty-all:jar:4.1.42.Final:compile (version managed from 4.1.42.Final)
> [DEBUG]          org.apache.commons:commons-lang3:jar:3.9:compile (version managed from 3.9)
> [DEBUG]          org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile (version managed from 1.8)
> [DEBUG]          com.fasterxml.jackson.core:jackson-databind:jar:2.10.0:compile (version managed from 2.10.0)
> [DEBUG]             com.fasterxml.jackson.core:jackson-core:jar:2.10.0:compile (version managed from 2.10.0)
> [DEBUG]          com.fasterxml.jackson.core:jackson-annotations:jar:2.10.0:compile (version managed from 2.10.0)
> [DEBUG]          com.google.code.findbugs:jsr305:jar:3.0.0:compile (version managed from 3.0.0)
> [DEBUG]          com.google.guava:guava:jar:14.0.1:provided (scope managed from compile) (version managed from 14.0.1)
> [DEBUG]          org.apache.commons:commons-crypto:jar:1.0.0:compile (version managed from 1.0.0) (exclusions managed from [net.java.dev.jna:jna:*:*])
> [DEBUG]       io.dropwizard.metrics:metrics-core:jar:4.1.1:compile (version managed from 4.1.1)
> [DEBUG]    org.apache.spark:spark-tags_2.12:jar:3.0.0-SNAPSHOT:test
> [DEBUG]       org.scala-lang:scala-library:jar:2.12.10:compile (version managed from 2.12.10)
> [DEBUG]    org.apache.spark:spark-tags_2.12:jar:tests:3.0.0-SNAPSHOT:test
> [DEBUG]    org.apache.hadoop:hadoop-client:jar:2.8.5:provided (exclusions managed from [org.fusesource.leveldbjni:leveldbjni-all:*:*, asm:asm:*:*, org.codehaus.jackson:jackson-mapper-asl:*:*, org.ow2.asm:asm:*:*, org.jboss.netty:netty:*:*, io.netty:netty:*:*, commons-beanutils:commons-beanutils-core:*:*, commons-logging:commons-logging:*:*, org.mockito:mockito-all:*:*, org.mortbay.jetty:servlet-api-2.5:*:*, javax.servlet:servlet-api:*:*, junit:junit:*:*, com.sun.jersey:*:*:*, com.sun.jersey.jersey-test-framework:*:*:*, com.sun.jersey.contribs:*:*:*, net.java.dev.jets3t:jets3t:*:*, javax.ws.rs:jsr311-api:*:*, org.eclipse.jetty:jetty-webapp:*:*])
> [DEBUG]       org.apache.hadoop:hadoop-common:jar:2.8.5:provided
> [DEBUG]          com.hadoop.gplcompression:hadoop-lzo:jar:0.4.19:provided
> [DEBUG]          commons-cli:commons-cli:jar:1.2:provided
> [DEBUG]          org.apache.commons:commons-math3:jar:3.4.1:provided (version managed from 3.1.1)
> [DEBUG]          org.apache.httpcomponents:httpclient:jar:4.5.6:provided (version managed from 4.5.2)
> [DEBUG]             org.apache.httpcomponents:httpcore:jar:4.4.12:provided (version managed from 4.4.10)
> [DEBUG]          commons-codec:commons-codec:jar:1.10:provided (version managed from 1.11)
> [DEBUG]          commons-io:commons-io:jar:2.4:provided (version managed from 2.5)
> [DEBUG]          commons-net:commons-net:jar:3.1:provided (version managed from 3.6)
> [DEBUG]          commons-collections:commons-collections:jar:3.2.2:provided (version managed from 3.2.2)
> [DEBUG]          org.eclipse.jetty:jetty-servlet:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.3.19.v20170502)
> [DEBUG]             org.eclipse.jetty:jetty-security:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.4.18.v20190429)
> [DEBUG]          javax.servlet.jsp:jsp-api:jar:2.1:provided
> [DEBUG]          log4j:log4j:jar:1.2.17:provided (scope managed from compile) (version managed from 1.2.17)
> [DEBUG]          commons-lang:commons-lang:jar:2.6:provided (version managed from 2.6)
> [DEBUG]          commons-beanutils:commons-beanutils:jar:1.9.4:provided (version managed from 1.9.3)
> [DEBUG]          org.apache.commons:commons-configuration2:jar:2.1.1:provided
> [DEBUG]          org.apache.avro:avro:jar:1.8.2:provided (version managed from 1.7.7)
> [DEBUG]             org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided (scope managed from compile) (version managed from 1.9.13)
> [DEBUG]             com.thoughtworks.paranamer:paranamer:jar:2.8:runtime (scope managed from compile) (version managed from 2.7)
> [DEBUG]             org.xerial.snappy:snappy-java:jar:1.1.7.3:provided (version managed from 1.1.1.3)
> [DEBUG]             org.tukaani:xz:jar:1.5:provided
> [DEBUG]          com.google.re2j:re2j:jar:1.1:provided
> [DEBUG]          com.google.protobuf:protobuf-java:jar:2.5.0:provided (scope managed from compile) (version managed from 2.5.0)
> [DEBUG]          com.google.code.gson:gson:jar:2.2.4:provided
> [DEBUG]          org.apache.hadoop:hadoop-auth:jar:2.8.5:provided
> [DEBUG]             com.nimbusds:nimbus-jose-jwt:jar:4.41.1:provided
> [DEBUG]                com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:provided
> [DEBUG]             net.minidev:json-smart:jar:2.3:provided
> [DEBUG]                net.minidev:accessors-smart:jar:1.2:provided
> [DEBUG]             org.apache.curator:curator-framework:jar:2.7.1:provided (version managed from 2.12.0)
> [DEBUG]          org.apache.curator:curator-client:jar:2.7.1:provided (version managed from 2.12.0)
> [DEBUG]          org.apache.curator:curator-recipes:jar:2.7.1:provided (scope managed from compile) (version managed from 2.12.0) (exclusions managed from [])
> [DEBUG]          org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
> [DEBUG]          org.apache.commons:commons-compress:jar:1.4.1:provided
> [DEBUG]          org.apache.kerby:kerb-simplekdc:jar:1.0.1:provided
> [DEBUG]             org.apache.kerby:kerb-client:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerby-config:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-core:jar:1.0.1:provided
> [DEBUG]                   org.apache.kerby:kerby-pkix:jar:1.0.1:provided
> [DEBUG]                      org.apache.kerby:kerby-asn1:jar:1.0.1:provided
> [DEBUG]                      org.apache.kerby:kerby-util:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-common:jar:1.0.1:provided
> [DEBUG]                   org.apache.kerby:kerb-crypto:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-util:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:token-provider:jar:1.0.1:provided
> [DEBUG]             org.apache.kerby:kerb-admin:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-server:jar:1.0.1:provided
> [DEBUG]                   org.apache.kerby:kerb-identity:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerby-xdr:jar:1.0.1:provided
> [DEBUG]          org.codehaus.woodstox:stax2-api:jar:3.1.4:provided
> [DEBUG]          com.fasterxml.woodstox:woodstox-core:jar:5.0.3:provided
> [DEBUG]       org.apache.hadoop:hadoop-hdfs-client:jar:2.8.5:provided
> [DEBUG]          com.squareup.okhttp:okhttp:jar:2.7.5:provided
> [DEBUG]             com.squareup.okio:okio:jar:1.6.0:provided
> [DEBUG]       org.apache.hadoop:hadoop-yarn-api:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [org.apache.hadoop:hadoop-annotations:*:*, com.google.inject:guice:*:*, com.sun.jersey.jersey-test-framework:jersey-test-framework-grizzly2:*:*, com.sun.jersey:jersey-server:*:*, com.sun.jersey.contribs:jersey-guice:*:*, com.google.inject.extensions:guice-servlet:*:*, org.apache.avro:avro:*:*, com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-json:*:*, io.netty:netty:*:*])
> [DEBUG]          javax.xml.bind:jaxb-api:jar:2.2.11:provided
> [DEBUG]          com.googlecode.json-simple:json-simple:jar:1.1.1:provided
> [DEBUG]       org.apache.hadoop:hadoop-yarn-client:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [org.apache.hadoop:hadoop-yarn-api:*:*, org.apache.hadoop:hadoop-yarn-common:*:*, org.apache.hadoop:hadoop-annotations:*:*, com.google.guava:guava:*:*, commons-cli:commons-cli:*:*, log4j:log4j:*:*, com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-server:*:*, com.sun.jersey:jersey-json:*:*, com.sun.jersey:jersey-servlet:*:*, io.netty:netty:*:*, com.google.inject.extensions:guice-servlet:*:*])
> [DEBUG]       org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.8.5:provided
> [DEBUG]          org.apache.hadoop:hadoop-yarn-common:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [])
> [DEBUG]             javax.servlet:javax.servlet-api:jar:3.1.0:provided
> [DEBUG]             org.eclipse.jetty:jetty-util:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.3.19.v20170502)
> [DEBUG]             com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.10.0:provided (version managed from 2.7.8)
> [DEBUG]                jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:provided
> [DEBUG]                jakarta.activation:jakarta.activation-api:jar:1.2.1:provided
> [DEBUG]             com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.7.8:provided
> [DEBUG]                com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.7.8:provided
> [DEBUG]       org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.8.5:provided
> [DEBUG]          org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.8.5:provided
> [DEBUG]       org.apache.hadoop:hadoop-annotations:jar:2.8.5:provided
> [DEBUG]    org.slf4j:slf4j-api:jar:1.7.16:provided
> [DEBUG]    org.spark-project.spark:unused:jar:1.0.0:compile
> [DEBUG]    org.scalatest:scalatest_2.12:jar:3.0.8:test
> [DEBUG]       org.scalactic:scalactic_2.12:jar:3.0.8:test
> [DEBUG]       org.scala-lang:scala-reflect:jar:2.12.10:test (version managed from 2.12.8)
> [DEBUG]       org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:test (version managed from 1.2.0)
> [DEBUG]    junit:junit:jar:4.12:test
> [DEBUG]       org.hamcrest:hamcrest-core:jar:1.3:test (scope managed from compile) (version managed from 1.3)
> [DEBUG]    com.novocode:junit-interface:jar:0.11:test
> [DEBUG]       org.scala-sbt:test-interface:jar:1.0:test
> [DEBUG] updateExcludesInDeps()
>
> I'll be very grateful if someone could help.
>
> Regards
> Saurabh Chawla
>
Reply | Threaded
Open this post in threaded view
|

Re: Spark master build hangs using parallel build option in maven

Dongjoon Hyun-2
Hi, Saurabh.

It seems that you are hitting https://issues.apache.org/jira/browse/SPARK-26095 .

And, we disabled the parallel build via https://github.com/apache/spark/pull/23061 at 3.0.0.

According to the stack trace in JIRA and PR description, `maven-shade-plugin` seems to be the root cause.

For now, I'd like to recommend you to disable it because `Maven` itself warns you already. (You know that, right?)

[INFO] --------------------------------[ pom ]---------------------------------
[WARNING] *****************************************************************
[WARNING] * Your build is requesting parallel execution, but project      *
[WARNING] * contains the following plugin(s) that have goals not marked   *
[WARNING] * as @threadSafe to support parallel building.                  *
[WARNING] * While this /may/ work fine, please look for plugin updates    *
[WARNING] * and/or request plugins be made thread-safe.                   *
[WARNING] * If reporting an issue, report it against the plugin in        *
[WARNING] * question, not against maven-core                              *
[WARNING] *****************************************************************
[WARNING] The following plugins are not marked @threadSafe in Spark Project Parent POM:
[WARNING] org.scalatest:scalatest-maven-plugin:2.0.0
[WARNING] Enable debug to see more precisely which goals are not marked @threadSafe.
[WARNING] *****************************************************************

I respect `Maven` warnings.

Bests,
Dongjoon.


On Fri, Jan 17, 2020 at 9:22 PM Saurabh Chawla <[hidden email]> wrote:
Hi Sean,

Thanks for checking this.

I am able to see parallel build info in the readme file https://github.com/apache/spark#building-spark

"
You can build Spark using more than one thread by using the -T option with Maven, see "Parallel builds in Maven 3". More detailed documentation is available from the project site, at "Building Spark".
"

This used to work while building older version of spark(2.4.3, 2.3.2 etc).
build/mvn -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true -T 4 clean package

Also I have seen the maven version is changed from 3.5.4 to 3.6.3 in master branch compared to spark 2.4.3.
Not sure if it's due to some bug in maven version used in master or some new change added in the master branch that prevent the parallel build.

Regards
Saurabh Chawla
 

On Sat, Jan 18, 2020 at 2:19 AM Sean Owen <[hidden email]> wrote:
I don't believe you can use a parallel build indeed. Some things
collide with each other. Some of the suites are run in parallel inside
the build though already.

On Fri, Jan 17, 2020 at 1:23 PM Saurabh Chawla <[hidden email]> wrote:
>
> Hi All,
>
> Spark master build hangs using parallel build option in maven. On running build the sequentially on spark master using maven, build did not hang. This issue occurs on giving hadoop-provided (-Phadoop-provided -Dhadoop.version=2.8.5) option. Same command works fine to build spark-2.4.3 parallelly
>
> Command to build spark master sequentially - Spark build works fine
> build/mvn  -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true  clean package
>
> Command to build spark master parallel - spark build hangs
> build/mvn -X -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true -T 4 clean package
>
> This is the trace which keeps on repeating in maven logs
>
> [DEBUG] building maven31 dependency graph for org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT with Maven31DependencyGraphBuilder
> [DEBUG] Dependency collection stats: {ConflictMarker.analyzeTime=60583, ConflictMarker.markTime=23750, ConflictMarker.nodeCount=419, ConflictIdSorter.graphTime=41262, ConflictIdSorter.topsortTime=9704, ConflictIdSorter.conflictIdCount=105, ConflictIdSorter.conflictIdCycleCount=0, ConflictResolver.totalTime=632542, ConflictResolver.conflictItemCount=193, DefaultDependencyCollector.collectTime=1020759, DefaultDependencyCollector.transformTime=775495}
> [DEBUG] org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT
> [DEBUG]    org.apache.spark:spark-network-shuffle_2.12:jar:3.0.0-SNAPSHOT:compile
> [DEBUG]       org.apache.spark:spark-network-common_2.12:jar:3.0.0-SNAPSHOT:compile
> [DEBUG]          io.netty:netty-all:jar:4.1.42.Final:compile (version managed from 4.1.42.Final)
> [DEBUG]          org.apache.commons:commons-lang3:jar:3.9:compile (version managed from 3.9)
> [DEBUG]          org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile (version managed from 1.8)
> [DEBUG]          com.fasterxml.jackson.core:jackson-databind:jar:2.10.0:compile (version managed from 2.10.0)
> [DEBUG]             com.fasterxml.jackson.core:jackson-core:jar:2.10.0:compile (version managed from 2.10.0)
> [DEBUG]          com.fasterxml.jackson.core:jackson-annotations:jar:2.10.0:compile (version managed from 2.10.0)
> [DEBUG]          com.google.code.findbugs:jsr305:jar:3.0.0:compile (version managed from 3.0.0)
> [DEBUG]          com.google.guava:guava:jar:14.0.1:provided (scope managed from compile) (version managed from 14.0.1)
> [DEBUG]          org.apache.commons:commons-crypto:jar:1.0.0:compile (version managed from 1.0.0) (exclusions managed from [net.java.dev.jna:jna:*:*])
> [DEBUG]       io.dropwizard.metrics:metrics-core:jar:4.1.1:compile (version managed from 4.1.1)
> [DEBUG]    org.apache.spark:spark-tags_2.12:jar:3.0.0-SNAPSHOT:test
> [DEBUG]       org.scala-lang:scala-library:jar:2.12.10:compile (version managed from 2.12.10)
> [DEBUG]    org.apache.spark:spark-tags_2.12:jar:tests:3.0.0-SNAPSHOT:test
> [DEBUG]    org.apache.hadoop:hadoop-client:jar:2.8.5:provided (exclusions managed from [org.fusesource.leveldbjni:leveldbjni-all:*:*, asm:asm:*:*, org.codehaus.jackson:jackson-mapper-asl:*:*, org.ow2.asm:asm:*:*, org.jboss.netty:netty:*:*, io.netty:netty:*:*, commons-beanutils:commons-beanutils-core:*:*, commons-logging:commons-logging:*:*, org.mockito:mockito-all:*:*, org.mortbay.jetty:servlet-api-2.5:*:*, javax.servlet:servlet-api:*:*, junit:junit:*:*, com.sun.jersey:*:*:*, com.sun.jersey.jersey-test-framework:*:*:*, com.sun.jersey.contribs:*:*:*, net.java.dev.jets3t:jets3t:*:*, javax.ws.rs:jsr311-api:*:*, org.eclipse.jetty:jetty-webapp:*:*])
> [DEBUG]       org.apache.hadoop:hadoop-common:jar:2.8.5:provided
> [DEBUG]          com.hadoop.gplcompression:hadoop-lzo:jar:0.4.19:provided
> [DEBUG]          commons-cli:commons-cli:jar:1.2:provided
> [DEBUG]          org.apache.commons:commons-math3:jar:3.4.1:provided (version managed from 3.1.1)
> [DEBUG]          org.apache.httpcomponents:httpclient:jar:4.5.6:provided (version managed from 4.5.2)
> [DEBUG]             org.apache.httpcomponents:httpcore:jar:4.4.12:provided (version managed from 4.4.10)
> [DEBUG]          commons-codec:commons-codec:jar:1.10:provided (version managed from 1.11)
> [DEBUG]          commons-io:commons-io:jar:2.4:provided (version managed from 2.5)
> [DEBUG]          commons-net:commons-net:jar:3.1:provided (version managed from 3.6)
> [DEBUG]          commons-collections:commons-collections:jar:3.2.2:provided (version managed from 3.2.2)
> [DEBUG]          org.eclipse.jetty:jetty-servlet:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.3.19.v20170502)
> [DEBUG]             org.eclipse.jetty:jetty-security:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.4.18.v20190429)
> [DEBUG]          javax.servlet.jsp:jsp-api:jar:2.1:provided
> [DEBUG]          log4j:log4j:jar:1.2.17:provided (scope managed from compile) (version managed from 1.2.17)
> [DEBUG]          commons-lang:commons-lang:jar:2.6:provided (version managed from 2.6)
> [DEBUG]          commons-beanutils:commons-beanutils:jar:1.9.4:provided (version managed from 1.9.3)
> [DEBUG]          org.apache.commons:commons-configuration2:jar:2.1.1:provided
> [DEBUG]          org.apache.avro:avro:jar:1.8.2:provided (version managed from 1.7.7)
> [DEBUG]             org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided (scope managed from compile) (version managed from 1.9.13)
> [DEBUG]             com.thoughtworks.paranamer:paranamer:jar:2.8:runtime (scope managed from compile) (version managed from 2.7)
> [DEBUG]             org.xerial.snappy:snappy-java:jar:1.1.7.3:provided (version managed from 1.1.1.3)
> [DEBUG]             org.tukaani:xz:jar:1.5:provided
> [DEBUG]          com.google.re2j:re2j:jar:1.1:provided
> [DEBUG]          com.google.protobuf:protobuf-java:jar:2.5.0:provided (scope managed from compile) (version managed from 2.5.0)
> [DEBUG]          com.google.code.gson:gson:jar:2.2.4:provided
> [DEBUG]          org.apache.hadoop:hadoop-auth:jar:2.8.5:provided
> [DEBUG]             com.nimbusds:nimbus-jose-jwt:jar:4.41.1:provided
> [DEBUG]                com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:provided
> [DEBUG]             net.minidev:json-smart:jar:2.3:provided
> [DEBUG]                net.minidev:accessors-smart:jar:1.2:provided
> [DEBUG]             org.apache.curator:curator-framework:jar:2.7.1:provided (version managed from 2.12.0)
> [DEBUG]          org.apache.curator:curator-client:jar:2.7.1:provided (version managed from 2.12.0)
> [DEBUG]          org.apache.curator:curator-recipes:jar:2.7.1:provided (scope managed from compile) (version managed from 2.12.0) (exclusions managed from [])
> [DEBUG]          org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
> [DEBUG]          org.apache.commons:commons-compress:jar:1.4.1:provided
> [DEBUG]          org.apache.kerby:kerb-simplekdc:jar:1.0.1:provided
> [DEBUG]             org.apache.kerby:kerb-client:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerby-config:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-core:jar:1.0.1:provided
> [DEBUG]                   org.apache.kerby:kerby-pkix:jar:1.0.1:provided
> [DEBUG]                      org.apache.kerby:kerby-asn1:jar:1.0.1:provided
> [DEBUG]                      org.apache.kerby:kerby-util:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-common:jar:1.0.1:provided
> [DEBUG]                   org.apache.kerby:kerb-crypto:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-util:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:token-provider:jar:1.0.1:provided
> [DEBUG]             org.apache.kerby:kerb-admin:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-server:jar:1.0.1:provided
> [DEBUG]                   org.apache.kerby:kerb-identity:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerby-xdr:jar:1.0.1:provided
> [DEBUG]          org.codehaus.woodstox:stax2-api:jar:3.1.4:provided
> [DEBUG]          com.fasterxml.woodstox:woodstox-core:jar:5.0.3:provided
> [DEBUG]       org.apache.hadoop:hadoop-hdfs-client:jar:2.8.5:provided
> [DEBUG]          com.squareup.okhttp:okhttp:jar:2.7.5:provided
> [DEBUG]             com.squareup.okio:okio:jar:1.6.0:provided
> [DEBUG]       org.apache.hadoop:hadoop-yarn-api:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [org.apache.hadoop:hadoop-annotations:*:*, com.google.inject:guice:*:*, com.sun.jersey.jersey-test-framework:jersey-test-framework-grizzly2:*:*, com.sun.jersey:jersey-server:*:*, com.sun.jersey.contribs:jersey-guice:*:*, com.google.inject.extensions:guice-servlet:*:*, org.apache.avro:avro:*:*, com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-json:*:*, io.netty:netty:*:*])
> [DEBUG]          javax.xml.bind:jaxb-api:jar:2.2.11:provided
> [DEBUG]          com.googlecode.json-simple:json-simple:jar:1.1.1:provided
> [DEBUG]       org.apache.hadoop:hadoop-yarn-client:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [org.apache.hadoop:hadoop-yarn-api:*:*, org.apache.hadoop:hadoop-yarn-common:*:*, org.apache.hadoop:hadoop-annotations:*:*, com.google.guava:guava:*:*, commons-cli:commons-cli:*:*, log4j:log4j:*:*, com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-server:*:*, com.sun.jersey:jersey-json:*:*, com.sun.jersey:jersey-servlet:*:*, io.netty:netty:*:*, com.google.inject.extensions:guice-servlet:*:*])
> [DEBUG]       org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.8.5:provided
> [DEBUG]          org.apache.hadoop:hadoop-yarn-common:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [])
> [DEBUG]             javax.servlet:javax.servlet-api:jar:3.1.0:provided
> [DEBUG]             org.eclipse.jetty:jetty-util:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.3.19.v20170502)
> [DEBUG]             com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.10.0:provided (version managed from 2.7.8)
> [DEBUG]                jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:provided
> [DEBUG]                jakarta.activation:jakarta.activation-api:jar:1.2.1:provided
> [DEBUG]             com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.7.8:provided
> [DEBUG]                com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.7.8:provided
> [DEBUG]       org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.8.5:provided
> [DEBUG]          org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.8.5:provided
> [DEBUG]       org.apache.hadoop:hadoop-annotations:jar:2.8.5:provided
> [DEBUG]    org.slf4j:slf4j-api:jar:1.7.16:provided
> [DEBUG]    org.spark-project.spark:unused:jar:1.0.0:compile
> [DEBUG]    org.scalatest:scalatest_2.12:jar:3.0.8:test
> [DEBUG]       org.scalactic:scalactic_2.12:jar:3.0.8:test
> [DEBUG]       org.scala-lang:scala-reflect:jar:2.12.10:test (version managed from 2.12.8)
> [DEBUG]       org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:test (version managed from 1.2.0)
> [DEBUG]    junit:junit:jar:4.12:test
> [DEBUG]       org.hamcrest:hamcrest-core:jar:1.3:test (scope managed from compile) (version managed from 1.3)
> [DEBUG]    com.novocode:junit-interface:jar:0.11:test
> [DEBUG]       org.scala-sbt:test-interface:jar:1.0:test
> [DEBUG] updateExcludesInDeps()
>
> I'll be very grateful if someone could help.
>
> Regards
> Saurabh Chawla
>
Reply | Threaded
Open this post in threaded view
|

Re: Spark master build hangs using parallel build option in maven

Saurabh Chawla
Hi Dongjoon,

Thanks for sharing the information related to parallel build issue in spark.

Sure will disable parallel build.

Regards
Saurabh Chawla

On Sat, Jan 18, 2020 at 1:08 PM Dongjoon Hyun <[hidden email]> wrote:
Hi, Saurabh.

It seems that you are hitting https://issues.apache.org/jira/browse/SPARK-26095 .

And, we disabled the parallel build via https://github.com/apache/spark/pull/23061 at 3.0.0.

According to the stack trace in JIRA and PR description, `maven-shade-plugin` seems to be the root cause.

For now, I'd like to recommend you to disable it because `Maven` itself warns you already. (You know that, right?)

[INFO] --------------------------------[ pom ]---------------------------------
[WARNING] *****************************************************************
[WARNING] * Your build is requesting parallel execution, but project      *
[WARNING] * contains the following plugin(s) that have goals not marked   *
[WARNING] * as @threadSafe to support parallel building.                  *
[WARNING] * While this /may/ work fine, please look for plugin updates    *
[WARNING] * and/or request plugins be made thread-safe.                   *
[WARNING] * If reporting an issue, report it against the plugin in        *
[WARNING] * question, not against maven-core                              *
[WARNING] *****************************************************************
[WARNING] The following plugins are not marked @threadSafe in Spark Project Parent POM:
[WARNING] org.scalatest:scalatest-maven-plugin:2.0.0
[WARNING] Enable debug to see more precisely which goals are not marked @threadSafe.
[WARNING] *****************************************************************

I respect `Maven` warnings.

Bests,
Dongjoon.


On Fri, Jan 17, 2020 at 9:22 PM Saurabh Chawla <[hidden email]> wrote:
Hi Sean,

Thanks for checking this.

I am able to see parallel build info in the readme file https://github.com/apache/spark#building-spark

"
You can build Spark using more than one thread by using the -T option with Maven, see "Parallel builds in Maven 3". More detailed documentation is available from the project site, at "Building Spark".
"

This used to work while building older version of spark(2.4.3, 2.3.2 etc).
build/mvn -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true -T 4 clean package

Also I have seen the maven version is changed from 3.5.4 to 3.6.3 in master branch compared to spark 2.4.3.
Not sure if it's due to some bug in maven version used in master or some new change added in the master branch that prevent the parallel build.

Regards
Saurabh Chawla
 

On Sat, Jan 18, 2020 at 2:19 AM Sean Owen <[hidden email]> wrote:
I don't believe you can use a parallel build indeed. Some things
collide with each other. Some of the suites are run in parallel inside
the build though already.

On Fri, Jan 17, 2020 at 1:23 PM Saurabh Chawla <[hidden email]> wrote:
>
> Hi All,
>
> Spark master build hangs using parallel build option in maven. On running build the sequentially on spark master using maven, build did not hang. This issue occurs on giving hadoop-provided (-Phadoop-provided -Dhadoop.version=2.8.5) option. Same command works fine to build spark-2.4.3 parallelly
>
> Command to build spark master sequentially - Spark build works fine
> build/mvn  -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true  clean package
>
> Command to build spark master parallel - spark build hangs
> build/mvn -X -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true -T 4 clean package
>
> This is the trace which keeps on repeating in maven logs
>
> [DEBUG] building maven31 dependency graph for org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT with Maven31DependencyGraphBuilder
> [DEBUG] Dependency collection stats: {ConflictMarker.analyzeTime=60583, ConflictMarker.markTime=23750, ConflictMarker.nodeCount=419, ConflictIdSorter.graphTime=41262, ConflictIdSorter.topsortTime=9704, ConflictIdSorter.conflictIdCount=105, ConflictIdSorter.conflictIdCycleCount=0, ConflictResolver.totalTime=632542, ConflictResolver.conflictItemCount=193, DefaultDependencyCollector.collectTime=1020759, DefaultDependencyCollector.transformTime=775495}
> [DEBUG] org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT
> [DEBUG]    org.apache.spark:spark-network-shuffle_2.12:jar:3.0.0-SNAPSHOT:compile
> [DEBUG]       org.apache.spark:spark-network-common_2.12:jar:3.0.0-SNAPSHOT:compile
> [DEBUG]          io.netty:netty-all:jar:4.1.42.Final:compile (version managed from 4.1.42.Final)
> [DEBUG]          org.apache.commons:commons-lang3:jar:3.9:compile (version managed from 3.9)
> [DEBUG]          org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile (version managed from 1.8)
> [DEBUG]          com.fasterxml.jackson.core:jackson-databind:jar:2.10.0:compile (version managed from 2.10.0)
> [DEBUG]             com.fasterxml.jackson.core:jackson-core:jar:2.10.0:compile (version managed from 2.10.0)
> [DEBUG]          com.fasterxml.jackson.core:jackson-annotations:jar:2.10.0:compile (version managed from 2.10.0)
> [DEBUG]          com.google.code.findbugs:jsr305:jar:3.0.0:compile (version managed from 3.0.0)
> [DEBUG]          com.google.guava:guava:jar:14.0.1:provided (scope managed from compile) (version managed from 14.0.1)
> [DEBUG]          org.apache.commons:commons-crypto:jar:1.0.0:compile (version managed from 1.0.0) (exclusions managed from [net.java.dev.jna:jna:*:*])
> [DEBUG]       io.dropwizard.metrics:metrics-core:jar:4.1.1:compile (version managed from 4.1.1)
> [DEBUG]    org.apache.spark:spark-tags_2.12:jar:3.0.0-SNAPSHOT:test
> [DEBUG]       org.scala-lang:scala-library:jar:2.12.10:compile (version managed from 2.12.10)
> [DEBUG]    org.apache.spark:spark-tags_2.12:jar:tests:3.0.0-SNAPSHOT:test
> [DEBUG]    org.apache.hadoop:hadoop-client:jar:2.8.5:provided (exclusions managed from [org.fusesource.leveldbjni:leveldbjni-all:*:*, asm:asm:*:*, org.codehaus.jackson:jackson-mapper-asl:*:*, org.ow2.asm:asm:*:*, org.jboss.netty:netty:*:*, io.netty:netty:*:*, commons-beanutils:commons-beanutils-core:*:*, commons-logging:commons-logging:*:*, org.mockito:mockito-all:*:*, org.mortbay.jetty:servlet-api-2.5:*:*, javax.servlet:servlet-api:*:*, junit:junit:*:*, com.sun.jersey:*:*:*, com.sun.jersey.jersey-test-framework:*:*:*, com.sun.jersey.contribs:*:*:*, net.java.dev.jets3t:jets3t:*:*, javax.ws.rs:jsr311-api:*:*, org.eclipse.jetty:jetty-webapp:*:*])
> [DEBUG]       org.apache.hadoop:hadoop-common:jar:2.8.5:provided
> [DEBUG]          com.hadoop.gplcompression:hadoop-lzo:jar:0.4.19:provided
> [DEBUG]          commons-cli:commons-cli:jar:1.2:provided
> [DEBUG]          org.apache.commons:commons-math3:jar:3.4.1:provided (version managed from 3.1.1)
> [DEBUG]          org.apache.httpcomponents:httpclient:jar:4.5.6:provided (version managed from 4.5.2)
> [DEBUG]             org.apache.httpcomponents:httpcore:jar:4.4.12:provided (version managed from 4.4.10)
> [DEBUG]          commons-codec:commons-codec:jar:1.10:provided (version managed from 1.11)
> [DEBUG]          commons-io:commons-io:jar:2.4:provided (version managed from 2.5)
> [DEBUG]          commons-net:commons-net:jar:3.1:provided (version managed from 3.6)
> [DEBUG]          commons-collections:commons-collections:jar:3.2.2:provided (version managed from 3.2.2)
> [DEBUG]          org.eclipse.jetty:jetty-servlet:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.3.19.v20170502)
> [DEBUG]             org.eclipse.jetty:jetty-security:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.4.18.v20190429)
> [DEBUG]          javax.servlet.jsp:jsp-api:jar:2.1:provided
> [DEBUG]          log4j:log4j:jar:1.2.17:provided (scope managed from compile) (version managed from 1.2.17)
> [DEBUG]          commons-lang:commons-lang:jar:2.6:provided (version managed from 2.6)
> [DEBUG]          commons-beanutils:commons-beanutils:jar:1.9.4:provided (version managed from 1.9.3)
> [DEBUG]          org.apache.commons:commons-configuration2:jar:2.1.1:provided
> [DEBUG]          org.apache.avro:avro:jar:1.8.2:provided (version managed from 1.7.7)
> [DEBUG]             org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided (scope managed from compile) (version managed from 1.9.13)
> [DEBUG]             com.thoughtworks.paranamer:paranamer:jar:2.8:runtime (scope managed from compile) (version managed from 2.7)
> [DEBUG]             org.xerial.snappy:snappy-java:jar:1.1.7.3:provided (version managed from 1.1.1.3)
> [DEBUG]             org.tukaani:xz:jar:1.5:provided
> [DEBUG]          com.google.re2j:re2j:jar:1.1:provided
> [DEBUG]          com.google.protobuf:protobuf-java:jar:2.5.0:provided (scope managed from compile) (version managed from 2.5.0)
> [DEBUG]          com.google.code.gson:gson:jar:2.2.4:provided
> [DEBUG]          org.apache.hadoop:hadoop-auth:jar:2.8.5:provided
> [DEBUG]             com.nimbusds:nimbus-jose-jwt:jar:4.41.1:provided
> [DEBUG]                com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:provided
> [DEBUG]             net.minidev:json-smart:jar:2.3:provided
> [DEBUG]                net.minidev:accessors-smart:jar:1.2:provided
> [DEBUG]             org.apache.curator:curator-framework:jar:2.7.1:provided (version managed from 2.12.0)
> [DEBUG]          org.apache.curator:curator-client:jar:2.7.1:provided (version managed from 2.12.0)
> [DEBUG]          org.apache.curator:curator-recipes:jar:2.7.1:provided (scope managed from compile) (version managed from 2.12.0) (exclusions managed from [])
> [DEBUG]          org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
> [DEBUG]          org.apache.commons:commons-compress:jar:1.4.1:provided
> [DEBUG]          org.apache.kerby:kerb-simplekdc:jar:1.0.1:provided
> [DEBUG]             org.apache.kerby:kerb-client:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerby-config:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-core:jar:1.0.1:provided
> [DEBUG]                   org.apache.kerby:kerby-pkix:jar:1.0.1:provided
> [DEBUG]                      org.apache.kerby:kerby-asn1:jar:1.0.1:provided
> [DEBUG]                      org.apache.kerby:kerby-util:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-common:jar:1.0.1:provided
> [DEBUG]                   org.apache.kerby:kerb-crypto:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-util:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:token-provider:jar:1.0.1:provided
> [DEBUG]             org.apache.kerby:kerb-admin:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerb-server:jar:1.0.1:provided
> [DEBUG]                   org.apache.kerby:kerb-identity:jar:1.0.1:provided
> [DEBUG]                org.apache.kerby:kerby-xdr:jar:1.0.1:provided
> [DEBUG]          org.codehaus.woodstox:stax2-api:jar:3.1.4:provided
> [DEBUG]          com.fasterxml.woodstox:woodstox-core:jar:5.0.3:provided
> [DEBUG]       org.apache.hadoop:hadoop-hdfs-client:jar:2.8.5:provided
> [DEBUG]          com.squareup.okhttp:okhttp:jar:2.7.5:provided
> [DEBUG]             com.squareup.okio:okio:jar:1.6.0:provided
> [DEBUG]       org.apache.hadoop:hadoop-yarn-api:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [org.apache.hadoop:hadoop-annotations:*:*, com.google.inject:guice:*:*, com.sun.jersey.jersey-test-framework:jersey-test-framework-grizzly2:*:*, com.sun.jersey:jersey-server:*:*, com.sun.jersey.contribs:jersey-guice:*:*, com.google.inject.extensions:guice-servlet:*:*, org.apache.avro:avro:*:*, com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-json:*:*, io.netty:netty:*:*])
> [DEBUG]          javax.xml.bind:jaxb-api:jar:2.2.11:provided
> [DEBUG]          com.googlecode.json-simple:json-simple:jar:1.1.1:provided
> [DEBUG]       org.apache.hadoop:hadoop-yarn-client:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [org.apache.hadoop:hadoop-yarn-api:*:*, org.apache.hadoop:hadoop-yarn-common:*:*, org.apache.hadoop:hadoop-annotations:*:*, com.google.guava:guava:*:*, commons-cli:commons-cli:*:*, log4j:log4j:*:*, com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-server:*:*, com.sun.jersey:jersey-json:*:*, com.sun.jersey:jersey-servlet:*:*, io.netty:netty:*:*, com.google.inject.extensions:guice-servlet:*:*])
> [DEBUG]       org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.8.5:provided
> [DEBUG]          org.apache.hadoop:hadoop-yarn-common:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [])
> [DEBUG]             javax.servlet:javax.servlet-api:jar:3.1.0:provided
> [DEBUG]             org.eclipse.jetty:jetty-util:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.3.19.v20170502)
> [DEBUG]             com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.10.0:provided (version managed from 2.7.8)
> [DEBUG]                jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:provided
> [DEBUG]                jakarta.activation:jakarta.activation-api:jar:1.2.1:provided
> [DEBUG]             com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.7.8:provided
> [DEBUG]                com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.7.8:provided
> [DEBUG]       org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.8.5:provided
> [DEBUG]          org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.8.5:provided
> [DEBUG]       org.apache.hadoop:hadoop-annotations:jar:2.8.5:provided
> [DEBUG]    org.slf4j:slf4j-api:jar:1.7.16:provided
> [DEBUG]    org.spark-project.spark:unused:jar:1.0.0:compile
> [DEBUG]    org.scalatest:scalatest_2.12:jar:3.0.8:test
> [DEBUG]       org.scalactic:scalactic_2.12:jar:3.0.8:test
> [DEBUG]       org.scala-lang:scala-reflect:jar:2.12.10:test (version managed from 2.12.8)
> [DEBUG]       org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:test (version managed from 1.2.0)
> [DEBUG]    junit:junit:jar:4.12:test
> [DEBUG]       org.hamcrest:hamcrest-core:jar:1.3:test (scope managed from compile) (version managed from 1.3)
> [DEBUG]    com.novocode:junit-interface:jar:0.11:test
> [DEBUG]       org.scala-sbt:test-interface:jar:1.0:test
> [DEBUG] updateExcludesInDeps()
>
> I'll be very grateful if someone could help.
>
> Regards
> Saurabh Chawla
>
Reply | Threaded
Open this post in threaded view
|

Re: Spark master build hangs using parallel build option in maven

Sean Owen-2
In reply to this post by Dongjoon Hyun-2
I think we can remove that note from the README then, I'll do that.

On Sat, Jan 18, 2020 at 1:38 AM Dongjoon Hyun <[hidden email]> wrote:

>
> Hi, Saurabh.
>
> It seems that you are hitting https://issues.apache.org/jira/browse/SPARK-26095 .
>
> And, we disabled the parallel build via https://github.com/apache/spark/pull/23061 at 3.0.0.
>
> According to the stack trace in JIRA and PR description, `maven-shade-plugin` seems to be the root cause.
>
> For now, I'd like to recommend you to disable it because `Maven` itself warns you already. (You know that, right?)
>
>> [INFO] --------------------------------[ pom ]---------------------------------
>> [WARNING] *****************************************************************
>> [WARNING] * Your build is requesting parallel execution, but project      *
>> [WARNING] * contains the following plugin(s) that have goals not marked   *
>> [WARNING] * as @threadSafe to support parallel building.                  *
>> [WARNING] * While this /may/ work fine, please look for plugin updates    *
>> [WARNING] * and/or request plugins be made thread-safe.                   *
>> [WARNING] * If reporting an issue, report it against the plugin in        *
>> [WARNING] * question, not against maven-core                              *
>> [WARNING] *****************************************************************
>> [WARNING] The following plugins are not marked @threadSafe in Spark Project Parent POM:
>> [WARNING] org.scalatest:scalatest-maven-plugin:2.0.0
>> [WARNING] Enable debug to see more precisely which goals are not marked @threadSafe.
>> [WARNING] *****************************************************************
>
>
> I respect `Maven` warnings.
>
> Bests,
> Dongjoon.
>
>
> On Fri, Jan 17, 2020 at 9:22 PM Saurabh Chawla <[hidden email]> wrote:
>>
>> Hi Sean,
>>
>> Thanks for checking this.
>>
>> I am able to see parallel build info in the readme file https://github.com/apache/spark#building-spark
>>
>> "
>> You can build Spark using more than one thread by using the -T option with Maven, see "Parallel builds in Maven 3". More detailed documentation is available from the project site, at "Building Spark".
>> "
>>
>> This used to work while building older version of spark(2.4.3, 2.3.2 etc).
>> build/mvn -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true -T 4 clean package
>>
>> Also I have seen the maven version is changed from 3.5.4 to 3.6.3 in master branch compared to spark 2.4.3.
>> Not sure if it's due to some bug in maven version used in master or some new change added in the master branch that prevent the parallel build.
>>
>> Regards
>> Saurabh Chawla
>>
>>
>> On Sat, Jan 18, 2020 at 2:19 AM Sean Owen <[hidden email]> wrote:
>>>
>>> I don't believe you can use a parallel build indeed. Some things
>>> collide with each other. Some of the suites are run in parallel inside
>>> the build though already.
>>>
>>> On Fri, Jan 17, 2020 at 1:23 PM Saurabh Chawla <[hidden email]> wrote:
>>> >
>>> > Hi All,
>>> >
>>> > Spark master build hangs using parallel build option in maven. On running build the sequentially on spark master using maven, build did not hang. This issue occurs on giving hadoop-provided (-Phadoop-provided -Dhadoop.version=2.8.5) option. Same command works fine to build spark-2.4.3 parallelly
>>> >
>>> > Command to build spark master sequentially - Spark build works fine
>>> > build/mvn  -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true  clean package
>>> >
>>> > Command to build spark master parallel - spark build hangs
>>> > build/mvn -X -Duse.zinc.server=false -DuseZincForJdk8=false -Dmaven.javadoc.skip=true -DskipSource=true -Phive -Phive-thriftserver -Phive-provided -Pyarn -Phadoop-provided -Dhadoop.version=2.8.5 -DskipTests=true -T 4 clean package
>>> >
>>> > This is the trace which keeps on repeating in maven logs
>>> >
>>> > [DEBUG] building maven31 dependency graph for org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT with Maven31DependencyGraphBuilder
>>> > [DEBUG] Dependency collection stats: {ConflictMarker.analyzeTime=60583, ConflictMarker.markTime=23750, ConflictMarker.nodeCount=419, ConflictIdSorter.graphTime=41262, ConflictIdSorter.topsortTime=9704, ConflictIdSorter.conflictIdCount=105, ConflictIdSorter.conflictIdCycleCount=0, ConflictResolver.totalTime=632542, ConflictResolver.conflictItemCount=193, DefaultDependencyCollector.collectTime=1020759, DefaultDependencyCollector.transformTime=775495}
>>> > [DEBUG] org.apache.spark:spark-network-yarn_2.12:jar:3.0.0-SNAPSHOT
>>> > [DEBUG]    org.apache.spark:spark-network-shuffle_2.12:jar:3.0.0-SNAPSHOT:compile
>>> > [DEBUG]       org.apache.spark:spark-network-common_2.12:jar:3.0.0-SNAPSHOT:compile
>>> > [DEBUG]          io.netty:netty-all:jar:4.1.42.Final:compile (version managed from 4.1.42.Final)
>>> > [DEBUG]          org.apache.commons:commons-lang3:jar:3.9:compile (version managed from 3.9)
>>> > [DEBUG]          org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile (version managed from 1.8)
>>> > [DEBUG]          com.fasterxml.jackson.core:jackson-databind:jar:2.10.0:compile (version managed from 2.10.0)
>>> > [DEBUG]             com.fasterxml.jackson.core:jackson-core:jar:2.10.0:compile (version managed from 2.10.0)
>>> > [DEBUG]          com.fasterxml.jackson.core:jackson-annotations:jar:2.10.0:compile (version managed from 2.10.0)
>>> > [DEBUG]          com.google.code.findbugs:jsr305:jar:3.0.0:compile (version managed from 3.0.0)
>>> > [DEBUG]          com.google.guava:guava:jar:14.0.1:provided (scope managed from compile) (version managed from 14.0.1)
>>> > [DEBUG]          org.apache.commons:commons-crypto:jar:1.0.0:compile (version managed from 1.0.0) (exclusions managed from [net.java.dev.jna:jna:*:*])
>>> > [DEBUG]       io.dropwizard.metrics:metrics-core:jar:4.1.1:compile (version managed from 4.1.1)
>>> > [DEBUG]    org.apache.spark:spark-tags_2.12:jar:3.0.0-SNAPSHOT:test
>>> > [DEBUG]       org.scala-lang:scala-library:jar:2.12.10:compile (version managed from 2.12.10)
>>> > [DEBUG]    org.apache.spark:spark-tags_2.12:jar:tests:3.0.0-SNAPSHOT:test
>>> > [DEBUG]    org.apache.hadoop:hadoop-client:jar:2.8.5:provided (exclusions managed from [org.fusesource.leveldbjni:leveldbjni-all:*:*, asm:asm:*:*, org.codehaus.jackson:jackson-mapper-asl:*:*, org.ow2.asm:asm:*:*, org.jboss.netty:netty:*:*, io.netty:netty:*:*, commons-beanutils:commons-beanutils-core:*:*, commons-logging:commons-logging:*:*, org.mockito:mockito-all:*:*, org.mortbay.jetty:servlet-api-2.5:*:*, javax.servlet:servlet-api:*:*, junit:junit:*:*, com.sun.jersey:*:*:*, com.sun.jersey.jersey-test-framework:*:*:*, com.sun.jersey.contribs:*:*:*, net.java.dev.jets3t:jets3t:*:*, javax.ws.rs:jsr311-api:*:*, org.eclipse.jetty:jetty-webapp:*:*])
>>> > [DEBUG]       org.apache.hadoop:hadoop-common:jar:2.8.5:provided
>>> > [DEBUG]          com.hadoop.gplcompression:hadoop-lzo:jar:0.4.19:provided
>>> > [DEBUG]          commons-cli:commons-cli:jar:1.2:provided
>>> > [DEBUG]          org.apache.commons:commons-math3:jar:3.4.1:provided (version managed from 3.1.1)
>>> > [DEBUG]          org.apache.httpcomponents:httpclient:jar:4.5.6:provided (version managed from 4.5.2)
>>> > [DEBUG]             org.apache.httpcomponents:httpcore:jar:4.4.12:provided (version managed from 4.4.10)
>>> > [DEBUG]          commons-codec:commons-codec:jar:1.10:provided (version managed from 1.11)
>>> > [DEBUG]          commons-io:commons-io:jar:2.4:provided (version managed from 2.5)
>>> > [DEBUG]          commons-net:commons-net:jar:3.1:provided (version managed from 3.6)
>>> > [DEBUG]          commons-collections:commons-collections:jar:3.2.2:provided (version managed from 3.2.2)
>>> > [DEBUG]          org.eclipse.jetty:jetty-servlet:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.3.19.v20170502)
>>> > [DEBUG]             org.eclipse.jetty:jetty-security:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.4.18.v20190429)
>>> > [DEBUG]          javax.servlet.jsp:jsp-api:jar:2.1:provided
>>> > [DEBUG]          log4j:log4j:jar:1.2.17:provided (scope managed from compile) (version managed from 1.2.17)
>>> > [DEBUG]          commons-lang:commons-lang:jar:2.6:provided (version managed from 2.6)
>>> > [DEBUG]          commons-beanutils:commons-beanutils:jar:1.9.4:provided (version managed from 1.9.3)
>>> > [DEBUG]          org.apache.commons:commons-configuration2:jar:2.1.1:provided
>>> > [DEBUG]          org.apache.avro:avro:jar:1.8.2:provided (version managed from 1.7.7)
>>> > [DEBUG]             org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided (scope managed from compile) (version managed from 1.9.13)
>>> > [DEBUG]             com.thoughtworks.paranamer:paranamer:jar:2.8:runtime (scope managed from compile) (version managed from 2.7)
>>> > [DEBUG]             org.xerial.snappy:snappy-java:jar:1.1.7.3:provided (version managed from 1.1.1.3)
>>> > [DEBUG]             org.tukaani:xz:jar:1.5:provided
>>> > [DEBUG]          com.google.re2j:re2j:jar:1.1:provided
>>> > [DEBUG]          com.google.protobuf:protobuf-java:jar:2.5.0:provided (scope managed from compile) (version managed from 2.5.0)
>>> > [DEBUG]          com.google.code.gson:gson:jar:2.2.4:provided
>>> > [DEBUG]          org.apache.hadoop:hadoop-auth:jar:2.8.5:provided
>>> > [DEBUG]             com.nimbusds:nimbus-jose-jwt:jar:4.41.1:provided
>>> > [DEBUG]                com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:provided
>>> > [DEBUG]             net.minidev:json-smart:jar:2.3:provided
>>> > [DEBUG]                net.minidev:accessors-smart:jar:1.2:provided
>>> > [DEBUG]             org.apache.curator:curator-framework:jar:2.7.1:provided (version managed from 2.12.0)
>>> > [DEBUG]          org.apache.curator:curator-client:jar:2.7.1:provided (version managed from 2.12.0)
>>> > [DEBUG]          org.apache.curator:curator-recipes:jar:2.7.1:provided (scope managed from compile) (version managed from 2.12.0) (exclusions managed from [])
>>> > [DEBUG]          org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided
>>> > [DEBUG]          org.apache.commons:commons-compress:jar:1.4.1:provided
>>> > [DEBUG]          org.apache.kerby:kerb-simplekdc:jar:1.0.1:provided
>>> > [DEBUG]             org.apache.kerby:kerb-client:jar:1.0.1:provided
>>> > [DEBUG]                org.apache.kerby:kerby-config:jar:1.0.1:provided
>>> > [DEBUG]                org.apache.kerby:kerb-core:jar:1.0.1:provided
>>> > [DEBUG]                   org.apache.kerby:kerby-pkix:jar:1.0.1:provided
>>> > [DEBUG]                      org.apache.kerby:kerby-asn1:jar:1.0.1:provided
>>> > [DEBUG]                      org.apache.kerby:kerby-util:jar:1.0.1:provided
>>> > [DEBUG]                org.apache.kerby:kerb-common:jar:1.0.1:provided
>>> > [DEBUG]                   org.apache.kerby:kerb-crypto:jar:1.0.1:provided
>>> > [DEBUG]                org.apache.kerby:kerb-util:jar:1.0.1:provided
>>> > [DEBUG]                org.apache.kerby:token-provider:jar:1.0.1:provided
>>> > [DEBUG]             org.apache.kerby:kerb-admin:jar:1.0.1:provided
>>> > [DEBUG]                org.apache.kerby:kerb-server:jar:1.0.1:provided
>>> > [DEBUG]                   org.apache.kerby:kerb-identity:jar:1.0.1:provided
>>> > [DEBUG]                org.apache.kerby:kerby-xdr:jar:1.0.1:provided
>>> > [DEBUG]          org.codehaus.woodstox:stax2-api:jar:3.1.4:provided
>>> > [DEBUG]          com.fasterxml.woodstox:woodstox-core:jar:5.0.3:provided
>>> > [DEBUG]       org.apache.hadoop:hadoop-hdfs-client:jar:2.8.5:provided
>>> > [DEBUG]          com.squareup.okhttp:okhttp:jar:2.7.5:provided
>>> > [DEBUG]             com.squareup.okio:okio:jar:1.6.0:provided
>>> > [DEBUG]       org.apache.hadoop:hadoop-yarn-api:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [org.apache.hadoop:hadoop-annotations:*:*, com.google.inject:guice:*:*, com.sun.jersey.jersey-test-framework:jersey-test-framework-grizzly2:*:*, com.sun.jersey:jersey-server:*:*, com.sun.jersey.contribs:jersey-guice:*:*, com.google.inject.extensions:guice-servlet:*:*, org.apache.avro:avro:*:*, com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-json:*:*, io.netty:netty:*:*])
>>> > [DEBUG]          javax.xml.bind:jaxb-api:jar:2.2.11:provided
>>> > [DEBUG]          com.googlecode.json-simple:json-simple:jar:1.1.1:provided
>>> > [DEBUG]       org.apache.hadoop:hadoop-yarn-client:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [org.apache.hadoop:hadoop-yarn-api:*:*, org.apache.hadoop:hadoop-yarn-common:*:*, org.apache.hadoop:hadoop-annotations:*:*, com.google.guava:guava:*:*, commons-cli:commons-cli:*:*, log4j:log4j:*:*, com.sun.jersey:jersey-core:*:*, com.sun.jersey:jersey-server:*:*, com.sun.jersey:jersey-json:*:*, com.sun.jersey:jersey-servlet:*:*, io.netty:netty:*:*, com.google.inject.extensions:guice-servlet:*:*])
>>> > [DEBUG]       org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.8.5:provided
>>> > [DEBUG]          org.apache.hadoop:hadoop-yarn-common:jar:2.8.5:provided (scope managed from compile) (version managed from 2.8.5) (exclusions managed from [])
>>> > [DEBUG]             javax.servlet:javax.servlet-api:jar:3.1.0:provided
>>> > [DEBUG]             org.eclipse.jetty:jetty-util:jar:9.4.18.v20190429:provided (scope managed from compile) (version managed from 9.3.19.v20170502)
>>> > [DEBUG]             com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.10.0:provided (version managed from 2.7.8)
>>> > [DEBUG]                jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:provided
>>> > [DEBUG]                jakarta.activation:jakarta.activation-api:jar:1.2.1:provided
>>> > [DEBUG]             com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.7.8:provided
>>> > [DEBUG]                com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.7.8:provided
>>> > [DEBUG]       org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.8.5:provided
>>> > [DEBUG]          org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.8.5:provided
>>> > [DEBUG]       org.apache.hadoop:hadoop-annotations:jar:2.8.5:provided
>>> > [DEBUG]    org.slf4j:slf4j-api:jar:1.7.16:provided
>>> > [DEBUG]    org.spark-project.spark:unused:jar:1.0.0:compile
>>> > [DEBUG]    org.scalatest:scalatest_2.12:jar:3.0.8:test
>>> > [DEBUG]       org.scalactic:scalactic_2.12:jar:3.0.8:test
>>> > [DEBUG]       org.scala-lang:scala-reflect:jar:2.12.10:test (version managed from 2.12.8)
>>> > [DEBUG]       org.scala-lang.modules:scala-xml_2.12:jar:1.2.0:test (version managed from 1.2.0)
>>> > [DEBUG]    junit:junit:jar:4.12:test
>>> > [DEBUG]       org.hamcrest:hamcrest-core:jar:1.3:test (scope managed from compile) (version managed from 1.3)
>>> > [DEBUG]    com.novocode:junit-interface:jar:0.11:test
>>> > [DEBUG]       org.scala-sbt:test-interface:jar:1.0:test
>>> > [DEBUG] updateExcludesInDeps()
>>> >
>>> > I'll be very grateful if someone could help.
>>> >
>>> > Regards
>>> > Saurabh Chawla
>>> >

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]