Running Spark through a debugger

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Running Spark through a debugger

Nicholas Chammas
I normally stick to the Python parts of Spark, but I am interested in walking through the DSv2 code and understanding how it works. I tried following the "IDE Setup" section of the developer tools page, but quickly hit several problems loading the project.

Can anyone point to a tutorial or guide on running Spark through a debugger, for folks like me who are not regular IntelliJ or Scala users?

My goal is just to be able to run something like `spark.read.text(...).show()` and follow it step by step through the codebase.

Nick

Reply | Threaded
Open this post in threaded view
|

Re: Running Spark through a debugger

Sean Owen-2
I just make a new test suite or something, set breakpoints, and
execute it in IJ. That generally works fine. You may need to set the
run configuration to have the right working dir (Spark project root),
and set the right system property to say 'this is running in a test'
in some cases. What are you having trouble with, does it build?

On Mon, Dec 16, 2019 at 11:27 PM Nicholas Chammas
<[hidden email]> wrote:
>
> I normally stick to the Python parts of Spark, but I am interested in walking through the DSv2 code and understanding how it works. I tried following the "IDE Setup" section of the developer tools page, but quickly hit several problems loading the project.
>
> Can anyone point to a tutorial or guide on running Spark through a debugger, for folks like me who are not regular IntelliJ or Scala users?
>
> My goal is just to be able to run something like `spark.read.text(...).show()` and follow it step by step through the codebase.
>
> Nick
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]