It seems that hive style partition pruning is not working for file based
data sources such as Parquet and ORC.
This causes serious performance degradation for non hive tables.
The reason for that is that the FileScan abstract class is not aware of the partition and data filters.
The method for getting the selectedPartitions calls the FileIndex listFiles
method with empty sequence for both - see here.
In the v1 datasource the FileSourceScanExec class gets the partition and data filters and use them to filter unnecessary
partitions by passing them to the listFiles function - see here.
Are there any ongoing plans to add a support for that?
Also, It seems to me that the dynamic partition pruning capabilities (SPARK-11150) that were added in Spark 3.0 are also not available for DataSourceV2 file based datasources.
The code for dynamically selecting the partitions was added only to the v1 code path - see here.