Scala type checking thread-safety issue, and global locks to resolve it

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Scala type checking thread-safety issue, and global locks to resolve it

Sean Owen-2
This is worth a look: https://github.com/apache/spark/pull/24085

Scala has a surprising thread-safety bug in the "<:<" operator that's
used to check subtypes, which can lead to incorrect results in
non-trivial situations.

The fix on the table is to introduce a global lock to protect a lot of
the Scala-related reflection code to resolve it. This may be the best
we can do, but 'global lock' sounds ominous.

Any thoughts on other ways to resolve it? I'm not sure there are.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Scala type checking thread-safety issue, and global locks to resolve it

Shixiong(Ryan) Zhu
Hey Sean,

Sounds good to me. At least, it's not worse than any versions prior to 2.3.0 which has a global ScalaReflectionLock. In addition, if someone hits a performance regression caused by this, they probably are creating too many Encoders. Reusing Encoders is a better solution for this case.

Best regards,
Shixiong

On Thu, Mar 14, 2019 at 2:25 PM Sean Owen <[hidden email]> wrote:
This is worth a look: https://github.com/apache/spark/pull/24085

Scala has a surprising thread-safety bug in the "<:<" operator that's
used to check subtypes, which can lead to incorrect results in
non-trivial situations.

The fix on the table is to introduce a global lock to protect a lot of
the Scala-related reflection code to resolve it. This may be the best
we can do, but 'global lock' sounds ominous.

Any thoughts on other ways to resolve it? I'm not sure there are.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Scala type checking thread-safety issue, and global locks to resolve it

Shixiong(Ryan) Zhu
Forgot to link the ticket that removed the global ScalaReflectionLock: https://issues.apache.org/jira/browse/SPARK-19810

Best Regards,

Ryan


On Fri, Mar 15, 2019 at 10:40 AM Shixiong(Ryan) Zhu <[hidden email]> wrote:
Hey Sean,

Sounds good to me. At least, it's not worse than any versions prior to 2.3.0 which has a global ScalaReflectionLock. In addition, if someone hits a performance regression caused by this, they probably are creating too many Encoders. Reusing Encoders is a better solution for this case.

Best regards,
Shixiong

On Thu, Mar 14, 2019 at 2:25 PM Sean Owen <[hidden email]> wrote:
This is worth a look: https://github.com/apache/spark/pull/24085

Scala has a surprising thread-safety bug in the "<:<" operator that's
used to check subtypes, which can lead to incorrect results in
non-trivial situations.

The fix on the table is to introduce a global lock to protect a lot of
the Scala-related reflection code to resolve it. This may be the best
we can do, but 'global lock' sounds ominous.

Any thoughts on other ways to resolve it? I'm not sure there are.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]