Metadata columns or function push-down for Kafka v2 source
Ryan: Kafka v1 source has more read columns than write columns. This is to expose metadata like partition, offset, and timestamp. Those are read columns, but not write columns, which is fine in v1. v2 requires a table schema
Ryan: Two main options to fix this in v2: add metadata columns like Presto’s $file or add function push-down similar to Spark’s input_file_name(). Metadata columns require less work (expose additional columns) but functions are more flexible (can call modified_time(col1))
Gabor: That’s a fair summary
Jose: the problem with input_file_name() is that it can be called anywhere, but is only valid in the context of a projection. After a group by, it returns empty string, which is odd.
Ryan: Couldn’t we handle that case using push-down? It is a function defined by a source that can only be run by pushing it down. It doesn’t exist after a group by, so analysis would fail if it were used in that context, just like columns don’t exist in the group by context unless they were in the grouping expression or created by aggregates.
Jose: That would work
Ryan: The metadata column approach takes less work, so I think we should do that unless someone has the time to drive the function push-down option.
USE ... [IN catalog] to set current namespace in a catalog
Matt offered to implement DESCRIBE TABLE
IF NOT EXISTS with INSERT INTO
John: This validates that overwrite does not overwrite partitions and is append only. Should this be supported?
Consensus was “why not?” Will add a mix-in trait in a follow-up for sources that choose to implement it
File source compatibility
Ryan: I tried to use built-in sources like Parquet in SQL tests and hit problems. Not being able to pass a schema or table partitioning means that these tables won’t behave right. What is the plan to get these sources working with SQL?
No one has time to work on this
Ryan: I’ll write some tests to at least set out the contract so we know when the built-in sources are ready to be used.