Spark 3: creating schema for hive metastore hangs forever

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Spark 3: creating schema for hive metastore hangs forever

Tomas Bartalos
Hello,

I'm using spark-3.0.0-bin-hadoop3.2 with custom hive metastore DB (postgres). I'm setting the "autoCreateAll" flag to true, so hive is creating its relational schema on first use. The problem is there is a deadlock and the query hangs forever:
Tx1 (holds lock on TBLS relation, wait_event: ClientRead): 
INSERT INTO "PARTITION_KEYS" ("TBL_ID","PKEY_COMMENT","PKEY_NAME","PKEY_TYPE","INTEGER_IDX") VALUES ($1,$2,$3,$4,$5
Tx2 (waiting for lock on TBLS relation, wait_event: relation): 
ALTER TABLE "TBL_PRIVS" ADD CONSTRAINT "TBL_PRIVS_FK1" FOREIGN KEY ("TBL_ID") REFERENCES "TBLS" ("TBL_ID") INITIALLY DEFERRED 

So Tx1 waits for client read and Tx2 is waiting for Tx1. I'm attaching the stack trace of active thread reading from postgres.

Steps to reproduce:
#start postgres
docker run --name postgres-hive -d -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres:13

#run shell with hive metastore DB pointing to postgres
./bin/spark-shell \
--packages org.postgresql:postgresql:42.2.14 \
--conf spark.hadoop.javax.jdo.option.ConnectionURL=jdbc:postgresql://localhost/postgres?createDatabaseIfNotExist=true \
--conf spark.hadoop.javax.jdo.option.ConnectionDriverName=org.postgresql.Driver \
--conf spark.hadoop.javax.jdo.option.ConnectionUserName=postgres \
--conf spark.hadoop.javax.jdo.option.ConnectionPassword=postgres \
--conf spark.hadoop.hive.metastore.schema.verification=false \
--conf spark.hadoop.datanucleus.schema.autoCreateAll=true \
--conf spark.hadoop.datanucleus.fixedDatastore=false

#Create any hive table
spark.sql("""
     | CREATE TABLE parquet_test (
     |  id int,
     |  str string,
     |  mp MAP<STRING,STRING>,
     |  lst ARRAY<STRING>,
     |  strct STRUCT<A:STRING,B:STRING>) 
     | PARTITIONED BY (part string)
     | STORED AS PARQUET;
     | """)

BR,
Tomas



---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

hive_pg_read_trace.txt (18K) Download Attachment