Quantcast

Why "Executor" have no NettyRpcEndpointRef?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Why "Executor" have no NettyRpcEndpointRef?

yellowboy_
This post has NOT been accepted by the mailing list yet.
I use spark-shell with spark-2.1.0

but when it begins , executor has no NettyRpcEndpointRef.

I do not know why??????????????If it has no NettyRpcEndpointRef , how it can send or receive message?

The cluster is one master with 2 worker

order: ./spark-shell --master spark://master:7077

result:
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why "Executor" have no NettyRpcEndpointRef?

cane
NettyRpcEndpointRef#toString if RpcAddress is null then will print null,like what you see.And if rpcendpoint is not driver,it will not call startserver,which init RpcAddress
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why "Executor" have no NettyRpcEndpointRef?

Marcelo Vanzin
No sure I fully understand what you're asking, but the executor
doesn't need to listen for incoming connections, so it doesn't need to
start a server and advertise an address.

On Thu, May 4, 2017 at 4:23 AM, cane <[hidden email]> wrote:

> NettyRpcEndpointRef#toString if RpcAddress is null then will print null,like
> what you see.And if rpcendpoint is not driver,it will not call
> startserver,which init RpcAddress
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Why-Executor-have-no-NettyRpcEndpointRef-tp21432p21495.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why "Executor" have no NettyRpcEndpointRef?

cane
Executor will start an actor system for remoting after rpcenv has been created.You can refer to SparkEnv.And why not start a netty server?I think, driver must handle all the executors' connetction but executor no need to.So an actor system server is enough?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why "Executor" have no NettyRpcEndpointRef?

yellowboy_
This post has NOT been accepted by the mailing list yet.
In reply to this post by cane
Well , as you say , executor has no RpcAddress , How Driver send a message to the Executor without RpcAddress ?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why "Executor" have no NettyRpcEndpointRef?

yellowboy_
This post has NOT been accepted by the mailing list yet.
In reply to this post by cane
OK,I got it,probably . I will see the SparkEnv soon.


Thank you!!!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why "Executor" have no NettyRpcEndpointRef?

Marcelo Vanzin
In reply to this post by cane
There's no actor system or akka dependency anymore, since 2.0 IIRC. So
I'm really not sure what you're referring to.

On Thu, May 4, 2017 at 8:29 PM, cane <[hidden email]> wrote:

> Executor will start an actor system for remoting after rpcenv has been
> created.You can refer to SparkEnv.And why not start a netty server?I think,
> driver must handle all the executors' connetction but executor no need to.So
> an actor system server is enough?
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Why-Executor-have-no-NettyRpcEndpointRef-tp21432p21501.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why "Executor" have no NettyRpcEndpointRef?

cane
Yes,i looked into branch 1.6.I will check out branch 2.1
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why "Executor" have no NettyRpcEndpointRef?

cane
In reply to this post by yellowboy_
After looking into code of branch-2.1.
I found out Driver first handle executor connections by NettyRpcHandler.
Once a connection arrive,driver's NettyRpcHandler will call receive,in which NettyRpcHandler#internalReceive will be called.
And then NettyEnv#deserialize will be called,which will update NettyRpcEnv.currentClient used for NettyRpcEndpointRef#client.
So although rpcaddress is null,the executor;s endpoint ref remained by driver will use the client to send message to executor.
Correct me if you found out some other useful message.Thanks!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why "Executor" have no NettyRpcEndpointRef?

yellowboy_
This post has NOT been accepted by the mailing list yet.
Yes!you are right!

First, in "CoarseGrainedSchedulerBackend.scala",there is a "executorRef.send(RegisteredExecutor)".

And then, go to the implemention of .send(),you will find it will call "netty.send(RequestMessage(nettyEnv.address,this,message))",and "this" is the receiver of the message.

And then, go to the "NettyRpcEnv#send", of course,remoteAddr is null(because executorRef.address is null). So go to the "NettyRpcEnv#postToOutbox", although the receiver.address is null, the receiver.client is not null. And Finally, messge will be send in "message.sendWith(receiver.client)"


What do you think of this?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why "Executor" have no NettyRpcEndpointRef?

cane
Yes,that's the true process.
And i think client is initalized when NettyRpcRef of executor is deserialized since NettyRpcEndpointRef#readObject is calling.
Am i right?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why "Executor" have no NettyRpcEndpointRef?

Marcelo Vanzin
I don't understand what it is you're trying to reach at. Are you just
trying to understand the RPC library? Or do you actually have a
question?

On Fri, May 5, 2017 at 3:33 AM, cane <[hidden email]> wrote:

> Yes,that's the true process.
> And i think client is initalized when NettyRpcRef of executor is
> deserialized since NettyRpcEndpointRef#readObject is calling.
> Am i right?
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Why-Executor-have-no-NettyRpcEndpointRef-tp21432p21508.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Why "Executor" have no NettyRpcEndpointRef?

cane

2017-05-06 0:27 GMT+08:00 Marcelo Vanzin <[hidden email]>:
I don't understand what it is you're trying to reach at. Are you just
trying to understand the RPC library? Or do you actually have a
question?

On Fri, May 5, 2017 at 3:33 AM, cane <[hidden email]> wrote:
> Yes,that's the true process.
> And i think client is initalized when NettyRpcRef of executor is
> deserialized since NettyRpcEndpointRef#readObject is calling.
> Am i right?
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Why-Executor-have-no-NettyRpcEndpointRef-tp21432p21508.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>



--
Marcelo

Loading...