It could get some data skew problem but might work for you
From: Burak Yavuz <[hidden email]> Sent: Tuesday, May 7, 2019 9:35:10 AM To: Shubham Chaurasia Cc: dev; [hidden email] Subject: Re: Static partitioning in partitionBy()
It depends on the data source. Delta Lake (https://delta.io) allows you to do it with the .option("replaceWhere", "c = c1"). With other file formats, you can write directly into the partition directory (tablePath/c=c1),
but you lose atomicity.
On Tue, May 7, 2019, 6:36 AM Shubham Chaurasia <[hidden email]> wrote:
Is there a way I can provide static partitions in partitionBy()?