spark code formatter?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

spark code formatter?

Michael Allman
Hi,

I've read the spark code style guide for contributors here:

https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide

For scala code, do you have a scalariform configuration that you use to
format your code to these specs?

Cheers,

Michael
Reply | Threaded
Open this post in threaded view
|

Re: spark code formatter?

Matei Zaharia
Administrator
Not that I know of. This would be very useful to add, especially if we can make SBT automatically check the code style (or we can somehow plug this into Jenkins).

Matei

On Jan 8, 2014, at 11:00 AM, Michael Allman <[hidden email]> wrote:

> Hi,
>
> I've read the spark code style guide for contributors here:
>
> https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
>
> For scala code, do you have a scalariform configuration that you use to format your code to these specs?
>
> Cheers,
>
> Michael

Reply | Threaded
Open this post in threaded view
|

Re: spark code formatter?

rxin
We have a Scala style configuration file in Shark:
https://github.com/amplab/shark/blob/master/scalastyle-config.xml

However, the scalastyle project is still pretty primitive and doesn't cover
most of the use cases. It is still great to include it to cover basic
checks such as 100-char wide lines.


On Wed, Jan 8, 2014 at 8:02 PM, Matei Zaharia <[hidden email]>wrote:

> Not that I know of. This would be very useful to add, especially if we can
> make SBT automatically check the code style (or we can somehow plug this
> into Jenkins).
>
> Matei
>
> On Jan 8, 2014, at 11:00 AM, Michael Allman <[hidden email]> wrote:
>
> > Hi,
> >
> > I've read the spark code style guide for contributors here:
> >
> > https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
> >
> > For scala code, do you have a scalariform configuration that you use to
> format your code to these specs?
> >
> > Cheers,
> >
> > Michael
>
>
Reply | Threaded
Open this post in threaded view
|

Re: spark code formatter?

DB Tsai
We use sbt-scalariform in our company, and it can automatically format
the coding style when runs `sbt compile`.

https://github.com/sbt/sbt-scalariform

We ask our developers to run `sbt compile` before commit, and it's
really nice to see everyone has the same spacing and indentation.

Sincerely,

DB Tsai
Machine Learning Engineer
Alpine Data Labs
--------------------------------------
Web: http://alpinenow.com/


On Wed, Jan 8, 2014 at 9:50 PM, Reynold Xin <[hidden email]> wrote:

> We have a Scala style configuration file in Shark:
> https://github.com/amplab/shark/blob/master/scalastyle-config.xml
>
> However, the scalastyle project is still pretty primitive and doesn't cover
> most of the use cases. It is still great to include it to cover basic
> checks such as 100-char wide lines.
>
>
> On Wed, Jan 8, 2014 at 8:02 PM, Matei Zaharia <[hidden email]>wrote:
>
>> Not that I know of. This would be very useful to add, especially if we can
>> make SBT automatically check the code style (or we can somehow plug this
>> into Jenkins).
>>
>> Matei
>>
>> On Jan 8, 2014, at 11:00 AM, Michael Allman <[hidden email]> wrote:
>>
>> > Hi,
>> >
>> > I've read the spark code style guide for contributors here:
>> >
>> > https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
>> >
>> > For scala code, do you have a scalariform configuration that you use to
>> format your code to these specs?
>> >
>> > Cheers,
>> >
>> > Michael
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: spark code formatter?

DB Tsai
A pull request for scalariform.
https://github.com/apache/incubator-spark/pull/365

Sincerely,

DB Tsai
Machine Learning Engineer
Alpine Data Labs
--------------------------------------
Web: http://alpinenow.com/


On Wed, Jan 8, 2014 at 10:09 PM, DB Tsai <[hidden email]> wrote:

> We use sbt-scalariform in our company, and it can automatically format
> the coding style when runs `sbt compile`.
>
> https://github.com/sbt/sbt-scalariform
>
> We ask our developers to run `sbt compile` before commit, and it's
> really nice to see everyone has the same spacing and indentation.
>
> Sincerely,
>
> DB Tsai
> Machine Learning Engineer
> Alpine Data Labs
> --------------------------------------
> Web: http://alpinenow.com/
>
>
> On Wed, Jan 8, 2014 at 9:50 PM, Reynold Xin <[hidden email]> wrote:
>> We have a Scala style configuration file in Shark:
>> https://github.com/amplab/shark/blob/master/scalastyle-config.xml
>>
>> However, the scalastyle project is still pretty primitive and doesn't cover
>> most of the use cases. It is still great to include it to cover basic
>> checks such as 100-char wide lines.
>>
>>
>> On Wed, Jan 8, 2014 at 8:02 PM, Matei Zaharia <[hidden email]>wrote:
>>
>>> Not that I know of. This would be very useful to add, especially if we can
>>> make SBT automatically check the code style (or we can somehow plug this
>>> into Jenkins).
>>>
>>> Matei
>>>
>>> On Jan 8, 2014, at 11:00 AM, Michael Allman <[hidden email]> wrote:
>>>
>>> > Hi,
>>> >
>>> > I've read the spark code style guide for contributors here:
>>> >
>>> > https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
>>> >
>>> > For scala code, do you have a scalariform configuration that you use to
>>> format your code to these specs?
>>> >
>>> > Cheers,
>>> >
>>> > Michael
>>>
>>>
Reply | Threaded
Open this post in threaded view
|

Re: spark code formatter?

rxin
Thanks for doing that, DB. Not sure about others, but I'm actually strongly
against blanket automatic code formatters, given that they can be
disruptive. Often humans would intentionally choose to style things in a
certain way for more clear semantics and better readability. Code
formatters don't capture these nuances. It is pretty dangerous to just auto
format everything.

Maybe it'd be ok if we restrict the code formatters to a very limited set
of things, such as indenting function parameters, etc.


On Wed, Jan 8, 2014 at 10:28 PM, DB Tsai <[hidden email]> wrote:

> A pull request for scalariform.
> https://github.com/apache/incubator-spark/pull/365
>
> Sincerely,
>
> DB Tsai
> Machine Learning Engineer
> Alpine Data Labs
> --------------------------------------
> Web: http://alpinenow.com/
>
>
> On Wed, Jan 8, 2014 at 10:09 PM, DB Tsai <[hidden email]> wrote:
> > We use sbt-scalariform in our company, and it can automatically format
> > the coding style when runs `sbt compile`.
> >
> > https://github.com/sbt/sbt-scalariform
> >
> > We ask our developers to run `sbt compile` before commit, and it's
> > really nice to see everyone has the same spacing and indentation.
> >
> > Sincerely,
> >
> > DB Tsai
> > Machine Learning Engineer
> > Alpine Data Labs
> > --------------------------------------
> > Web: http://alpinenow.com/
> >
> >
> > On Wed, Jan 8, 2014 at 9:50 PM, Reynold Xin <[hidden email]> wrote:
> >> We have a Scala style configuration file in Shark:
> >> https://github.com/amplab/shark/blob/master/scalastyle-config.xml
> >>
> >> However, the scalastyle project is still pretty primitive and doesn't
> cover
> >> most of the use cases. It is still great to include it to cover basic
> >> checks such as 100-char wide lines.
> >>
> >>
> >> On Wed, Jan 8, 2014 at 8:02 PM, Matei Zaharia <[hidden email]
> >wrote:
> >>
> >>> Not that I know of. This would be very useful to add, especially if we
> can
> >>> make SBT automatically check the code style (or we can somehow plug
> this
> >>> into Jenkins).
> >>>
> >>> Matei
> >>>
> >>> On Jan 8, 2014, at 11:00 AM, Michael Allman <[hidden email]> wrote:
> >>>
> >>> > Hi,
> >>> >
> >>> > I've read the spark code style guide for contributors here:
> >>> >
> >>> >
> https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
> >>> >
> >>> > For scala code, do you have a scalariform configuration that you use
> to
> >>> format your code to these specs?
> >>> >
> >>> > Cheers,
> >>> >
> >>> > Michael
> >>>
> >>>
>
Reply | Threaded
Open this post in threaded view
|

Re: spark code formatter?

Patrick Wendell
I'm also very wary of using a code formatter for the reasons already
mentioned by Reynold.

Does scaliform have a mode where it just provides style checks rather
than reformat the code? This is something we really need for, e.g.,
reviewing the many submissions to the project.

- Patrick

On Wed, Jan 8, 2014 at 11:51 PM, Reynold Xin <[hidden email]> wrote:

> Thanks for doing that, DB. Not sure about others, but I'm actually strongly
> against blanket automatic code formatters, given that they can be
> disruptive. Often humans would intentionally choose to style things in a
> certain way for more clear semantics and better readability. Code
> formatters don't capture these nuances. It is pretty dangerous to just auto
> format everything.
>
> Maybe it'd be ok if we restrict the code formatters to a very limited set
> of things, such as indenting function parameters, etc.
>
>
> On Wed, Jan 8, 2014 at 10:28 PM, DB Tsai <[hidden email]> wrote:
>
>> A pull request for scalariform.
>> https://github.com/apache/incubator-spark/pull/365
>>
>> Sincerely,
>>
>> DB Tsai
>> Machine Learning Engineer
>> Alpine Data Labs
>> --------------------------------------
>> Web: http://alpinenow.com/
>>
>>
>> On Wed, Jan 8, 2014 at 10:09 PM, DB Tsai <[hidden email]> wrote:
>> > We use sbt-scalariform in our company, and it can automatically format
>> > the coding style when runs `sbt compile`.
>> >
>> > https://github.com/sbt/sbt-scalariform
>> >
>> > We ask our developers to run `sbt compile` before commit, and it's
>> > really nice to see everyone has the same spacing and indentation.
>> >
>> > Sincerely,
>> >
>> > DB Tsai
>> > Machine Learning Engineer
>> > Alpine Data Labs
>> > --------------------------------------
>> > Web: http://alpinenow.com/
>> >
>> >
>> > On Wed, Jan 8, 2014 at 9:50 PM, Reynold Xin <[hidden email]> wrote:
>> >> We have a Scala style configuration file in Shark:
>> >> https://github.com/amplab/shark/blob/master/scalastyle-config.xml
>> >>
>> >> However, the scalastyle project is still pretty primitive and doesn't
>> cover
>> >> most of the use cases. It is still great to include it to cover basic
>> >> checks such as 100-char wide lines.
>> >>
>> >>
>> >> On Wed, Jan 8, 2014 at 8:02 PM, Matei Zaharia <[hidden email]
>> >wrote:
>> >>
>> >>> Not that I know of. This would be very useful to add, especially if we
>> can
>> >>> make SBT automatically check the code style (or we can somehow plug
>> this
>> >>> into Jenkins).
>> >>>
>> >>> Matei
>> >>>
>> >>> On Jan 8, 2014, at 11:00 AM, Michael Allman <[hidden email]> wrote:
>> >>>
>> >>> > Hi,
>> >>> >
>> >>> > I've read the spark code style guide for contributors here:
>> >>> >
>> >>> >
>> https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
>> >>> >
>> >>> > For scala code, do you have a scalariform configuration that you use
>> to
>> >>> format your code to these specs?
>> >>> >
>> >>> > Cheers,
>> >>> >
>> >>> > Michael
>> >>>
>> >>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: spark code formatter?

DB Tsai
Initially, we also had the same concern, so we started from limited
set of rules. Gradually, we found that it increases the productivity
and readability of our codebase.

PS, Scalariform is compatible with the Scala Style Guide in the sense
that, given the right preference settings, source code that is
initially compiliant with the Style Guide will not become uncompliant
after formatting. In a number of cases, running the formatter will
make uncompliant source more compliant.

I added the configuration option in the latest PR to limit the set of
rules. The options are https://github.com/mdr/scalariform

When developers wants to choose their own style for whatever reasons,
they can source directives to turn it off by `// format: OFF`.

Just quickly run the formatter, and I found that Spark is in general
in good shape; most of the changes are extra space after semicolon.

-  def run[K: Manifest, V <: Vertex : Manifest, M <: Message[K] :
Manifest, C: Manifest](

+  def run[K: Manifest, V <: Vertex: Manifest, M <: Message[K]:
Manifest, C: Manifest](

-  def addFile(file: File) : String = {
+ def addFile(file: File): String = {

Sincerely,

DB Tsai
Machine Learning Engineer
Alpine Data Labs
--------------------------------------
Web: http://alpinenow.com/


On Thu, Jan 9, 2014 at 12:32 AM, Patrick Wendell <[hidden email]> wrote:

> I'm also very wary of using a code formatter for the reasons already
> mentioned by Reynold.
>
> Does scaliform have a mode where it just provides style checks rather
> than reformat the code? This is something we really need for, e.g.,
> reviewing the many submissions to the project.
>
> - Patrick
>
> On Wed, Jan 8, 2014 at 11:51 PM, Reynold Xin <[hidden email]> wrote:
>> Thanks for doing that, DB. Not sure about others, but I'm actually strongly
>> against blanket automatic code formatters, given that they can be
>> disruptive. Often humans would intentionally choose to style things in a
>> certain way for more clear semantics and better readability. Code
>> formatters don't capture these nuances. It is pretty dangerous to just auto
>> format everything.
>>
>> Maybe it'd be ok if we restrict the code formatters to a very limited set
>> of things, such as indenting function parameters, etc.
>>
>>
>> On Wed, Jan 8, 2014 at 10:28 PM, DB Tsai <[hidden email]> wrote:
>>
>>> A pull request for scalariform.
>>> https://github.com/apache/incubator-spark/pull/365
>>>
>>> Sincerely,
>>>
>>> DB Tsai
>>> Machine Learning Engineer
>>> Alpine Data Labs
>>> --------------------------------------
>>> Web: http://alpinenow.com/
>>>
>>>
>>> On Wed, Jan 8, 2014 at 10:09 PM, DB Tsai <[hidden email]> wrote:
>>> > We use sbt-scalariform in our company, and it can automatically format
>>> > the coding style when runs `sbt compile`.
>>> >
>>> > https://github.com/sbt/sbt-scalariform
>>> >
>>> > We ask our developers to run `sbt compile` before commit, and it's
>>> > really nice to see everyone has the same spacing and indentation.
>>> >
>>> > Sincerely,
>>> >
>>> > DB Tsai
>>> > Machine Learning Engineer
>>> > Alpine Data Labs
>>> > --------------------------------------
>>> > Web: http://alpinenow.com/
>>> >
>>> >
>>> > On Wed, Jan 8, 2014 at 9:50 PM, Reynold Xin <[hidden email]> wrote:
>>> >> We have a Scala style configuration file in Shark:
>>> >> https://github.com/amplab/shark/blob/master/scalastyle-config.xml
>>> >>
>>> >> However, the scalastyle project is still pretty primitive and doesn't
>>> cover
>>> >> most of the use cases. It is still great to include it to cover basic
>>> >> checks such as 100-char wide lines.
>>> >>
>>> >>
>>> >> On Wed, Jan 8, 2014 at 8:02 PM, Matei Zaharia <[hidden email]
>>> >wrote:
>>> >>
>>> >>> Not that I know of. This would be very useful to add, especially if we
>>> can
>>> >>> make SBT automatically check the code style (or we can somehow plug
>>> this
>>> >>> into Jenkins).
>>> >>>
>>> >>> Matei
>>> >>>
>>> >>> On Jan 8, 2014, at 11:00 AM, Michael Allman <[hidden email]> wrote:
>>> >>>
>>> >>> > Hi,
>>> >>> >
>>> >>> > I've read the spark code style guide for contributors here:
>>> >>> >
>>> >>> >
>>> https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
>>> >>> >
>>> >>> > For scala code, do you have a scalariform configuration that you use
>>> to
>>> >>> format your code to these specs?
>>> >>> >
>>> >>> > Cheers,
>>> >>> >
>>> >>> > Michael
>>> >>>
>>> >>>
>>>
Reply | Threaded
Open this post in threaded view
|

Re: spark code formatter?

Evan Chan
BTW, we also run Scalariform, but we don't turn it on automatically.  We
find that for the most part it is good, but there are a few places where it
reformats things and doesn't look good, and requires cleanup.  I think
Scalariform requires some more rules to make it more generally useful.

-Evan



On Thu, Jan 9, 2014 at 12:23 PM, DB Tsai <[hidden email]> wrote:

> Initially, we also had the same concern, so we started from limited
> set of rules. Gradually, we found that it increases the productivity
> and readability of our codebase.
>
> PS, Scalariform is compatible with the Scala Style Guide in the sense
> that, given the right preference settings, source code that is
> initially compiliant with the Style Guide will not become uncompliant
> after formatting. In a number of cases, running the formatter will
> make uncompliant source more compliant.
>
> I added the configuration option in the latest PR to limit the set of
> rules. The options are https://github.com/mdr/scalariform
>
> When developers wants to choose their own style for whatever reasons,
> they can source directives to turn it off by `// format: OFF`.
>
> Just quickly run the formatter, and I found that Spark is in general
> in good shape; most of the changes are extra space after semicolon.
>
> -  def run[K: Manifest, V <: Vertex : Manifest, M <: Message[K] :
> Manifest, C: Manifest](
>
> +  def run[K: Manifest, V <: Vertex: Manifest, M <: Message[K]:
> Manifest, C: Manifest](
>
> -  def addFile(file: File) : String = {
> + def addFile(file: File): String = {
>
> Sincerely,
>
> DB Tsai
> Machine Learning Engineer
> Alpine Data Labs
> --------------------------------------
> Web: http://alpinenow.com/
>
>
> On Thu, Jan 9, 2014 at 12:32 AM, Patrick Wendell <[hidden email]>
> wrote:
> > I'm also very wary of using a code formatter for the reasons already
> > mentioned by Reynold.
> >
> > Does scaliform have a mode where it just provides style checks rather
> > than reformat the code? This is something we really need for, e.g.,
> > reviewing the many submissions to the project.
> >
> > - Patrick
> >
> > On Wed, Jan 8, 2014 at 11:51 PM, Reynold Xin <[hidden email]>
> wrote:
> >> Thanks for doing that, DB. Not sure about others, but I'm actually
> strongly
> >> against blanket automatic code formatters, given that they can be
> >> disruptive. Often humans would intentionally choose to style things in a
> >> certain way for more clear semantics and better readability. Code
> >> formatters don't capture these nuances. It is pretty dangerous to just
> auto
> >> format everything.
> >>
> >> Maybe it'd be ok if we restrict the code formatters to a very limited
> set
> >> of things, such as indenting function parameters, etc.
> >>
> >>
> >> On Wed, Jan 8, 2014 at 10:28 PM, DB Tsai <[hidden email]> wrote:
> >>
> >>> A pull request for scalariform.
> >>> https://github.com/apache/incubator-spark/pull/365
> >>>
> >>> Sincerely,
> >>>
> >>> DB Tsai
> >>> Machine Learning Engineer
> >>> Alpine Data Labs
> >>> --------------------------------------
> >>> Web: http://alpinenow.com/
> >>>
> >>>
> >>> On Wed, Jan 8, 2014 at 10:09 PM, DB Tsai <[hidden email]> wrote:
> >>> > We use sbt-scalariform in our company, and it can automatically
> format
> >>> > the coding style when runs `sbt compile`.
> >>> >
> >>> > https://github.com/sbt/sbt-scalariform
> >>> >
> >>> > We ask our developers to run `sbt compile` before commit, and it's
> >>> > really nice to see everyone has the same spacing and indentation.
> >>> >
> >>> > Sincerely,
> >>> >
> >>> > DB Tsai
> >>> > Machine Learning Engineer
> >>> > Alpine Data Labs
> >>> > --------------------------------------
> >>> > Web: http://alpinenow.com/
> >>> >
> >>> >
> >>> > On Wed, Jan 8, 2014 at 9:50 PM, Reynold Xin <[hidden email]>
> wrote:
> >>> >> We have a Scala style configuration file in Shark:
> >>> >> https://github.com/amplab/shark/blob/master/scalastyle-config.xml
> >>> >>
> >>> >> However, the scalastyle project is still pretty primitive and
> doesn't
> >>> cover
> >>> >> most of the use cases. It is still great to include it to cover
> basic
> >>> >> checks such as 100-char wide lines.
> >>> >>
> >>> >>
> >>> >> On Wed, Jan 8, 2014 at 8:02 PM, Matei Zaharia <
> [hidden email]
> >>> >wrote:
> >>> >>
> >>> >>> Not that I know of. This would be very useful to add, especially
> if we
> >>> can
> >>> >>> make SBT automatically check the code style (or we can somehow plug
> >>> this
> >>> >>> into Jenkins).
> >>> >>>
> >>> >>> Matei
> >>> >>>
> >>> >>> On Jan 8, 2014, at 11:00 AM, Michael Allman <[hidden email]> wrote:
> >>> >>>
> >>> >>> > Hi,
> >>> >>> >
> >>> >>> > I've read the spark code style guide for contributors here:
> >>> >>> >
> >>> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
> >>> >>> >
> >>> >>> > For scala code, do you have a scalariform configuration that you
> use
> >>> to
> >>> >>> format your code to these specs?
> >>> >>> >
> >>> >>> > Cheers,
> >>> >>> >
> >>> >>> > Michael
> >>> >>>
> >>> >>>
> >>>
>



--
--
Evan Chan
Staff Engineer
[hidden email]  |

<http://www.ooyala.com/>
<http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>