[ANNOUNCE] Announcing Apache Spark 2.4.4

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANNOUNCE] Announcing Apache Spark 2.4.4

Dongjoon Hyun-2
We are happy to announce the availability of Spark 2.4.4!

Spark 2.4.4 is a maintenance release containing stability fixes. This
release is based on the branch-2.4 maintenance branch of Spark. We strongly
recommend all 2.4 users to upgrade to this stable release.

To download Spark 2.4.4, head over to the download page:

Note that you might need to clear your browser cache or
to use `Private`/`Incognito` mode according to your browsers.

To view the release notes:

We would like to acknowledge all community members for contributing to this
release. This release would not have been possible without you.

Dongjoon Hyun
Reply | Threaded
Open this post in threaded view
|

Re: [ANNOUNCE] Announcing Apache Spark 2.4.4

cloud0fan
Great! Thanks!

On Mon, Sep 2, 2019 at 5:55 AM Dongjoon Hyun <[hidden email]> wrote:
We are happy to announce the availability of Spark 2.4.4!

Spark 2.4.4 is a maintenance release containing stability fixes. This
release is based on the branch-2.4 maintenance branch of Spark. We strongly
recommend all 2.4 users to upgrade to this stable release.

To download Spark 2.4.4, head over to the download page:

Note that you might need to clear your browser cache or
to use `Private`/`Incognito` mode according to your browsers.

To view the release notes:

We would like to acknowledge all community members for contributing to this
release. This release would not have been possible without you.

Dongjoon Hyun
Reply | Threaded
Open this post in threaded view
|

RE: read image or binary files / spark 2.3

Ilya Matiach-2
In reply to this post by Dongjoon Hyun-2

Hi Peter,

You can use the spark.readImages API in spark 2.3 for reading images:

 

https://databricks.com/blog/2018/12/10/introducing-built-in-image-data-source-in-apache-spark-2-4.html

https://blogs.technet.microsoft.com/machinelearning/2018/03/05/image-data-support-in-apache-spark/

 

https://spark.apache.org/docs/2.3.0/api/scala/index.html#org.apache.spark.ml.image.ImageSchema$

 

There’s also a spark package for spark versions older than 2.3:

https://github.com/Microsoft/spark-images

 

Thank you, Ilya

 

 

 

 

From: Peter Liu <[hidden email]>
Sent: Thursday, September 5, 2019 2:13 PM
To: dev <[hidden email]>; User <[hidden email]>
Subject: Re: read image or binary files / spark 2.3

 

Hello experts,

 

I have quick question: which API allows me to read images files or binary files (for SparkSession.readStream) from a local/hadoop file system in Spark 2.3?

 

I have been browsing the following documentations and googling for it and didn't find a good example/documentation:

 

 

any hint/help would be very much appreciated!

 

thanks!

 

Peter