Hi,

I’m working on something that requires deterministic randomness, i.e. a row gets the same “random” value no matter the order of the DataFrame. A seeded hash seems to be the perfect way to do this, but the existing hashes have various limitations:

- hash: 32-bit output (only 4 billion possibilities will result in a lot of collisions for many tables: the birthday paradox implies >50% chance of at least one for tables larger than 77000 rows, and likely ~1.6 billion collisions in a table of size 4 billion)

- sha1/sha2/md5: single binary column input, string output

It seems there’s already support for a 64-bit hash function that can work with an arbitrary number of arbitrary-typed columns (XxHash64), and exposing this for DataFrames seems like it’s essentially one line in sql/functions.scala to match `hash` (plus docs, tests, function registry etc.):

def hash64(cols: Column*): Column = withExpr { new XxHash64(cols.map(_.expr)) }

For my use case, this can then be used to get a 64-bit “random” column like

val seed = rng.nextLong()

hash64(lit(seed), col1, col2)

I’ve created a (hopefully) complete patch by mimicking ‘hash’ at

https://github.com/apache/spark/compare/master...huonw:hash64; should I open a JIRA and submit it as a pull request?

Additionally, both hash and the new hash64 already have support for being seeded, but this isn’t exposed directly and instead requires something like the `lit` above. Would it make sense to add overloads like the following?

def hash(seed: Int, cols: Columns*) = …

def hash64(seed: Long, cols: Columns*) = …

Though, it does seem a bit unfortunate to be forced to pass the seed first.

(I sent this email to

[hidden email] a few days ago, but didn't get any discussion about the Spark aspects of this, so I'm resending it here; I apologise in advance if I'm breaking protocol!)

- Huon Wilson

---------------------------------------------------------------------

To unsubscribe e-mail:

[hidden email]