the dependence length of RDD, can its size be greater than 1 pleaae?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

the dependence length of RDD, can its size be greater than 1 pleaae?

萝卜丝炒饭
Hi all,

The RDD code keeps a member as below:
dependencies_ : seq[Dependency[_]]

It is a seq, that means it can keep more than one dependency.

I have an issue about this.
Is it possible that its size is greater than one please?
If yes, how to produce it please? Would you like show me some code please?

thanks 
Robin Shao
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: the dependence length of RDD, can its size be greater than 1 pleaae?

rxin
A join?

On Thu, Jun 15, 2017 at 1:11 AM 萝卜丝炒饭 <[hidden email]> wrote:
Hi all,

The RDD code keeps a member as below:
dependencies_ : seq[Dependency[_]]

It is a seq, that means it can keep more than one dependency.

I have an issue about this.
Is it possible that its size is greater than one please?
If yes, how to produce it please? Would you like show me some code please?

thanks 
Robin Shao
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: the dependence length of RDD, can its size be greater than 1 pleaae?

Sean Owen
In reply to this post by 萝卜丝炒饭
Yes. Imagine an RDD that results from a union of other RDDs.

On Thu, Jun 15, 2017, 09:11 萝卜丝炒饭 <[hidden email]> wrote:
Hi all,

The RDD code keeps a member as below:
dependencies_ : seq[Dependency[_]]

It is a seq, that means it can keep more than one dependency.

I have an issue about this.
Is it possible that its size is greater than one please?
If yes, how to produce it please? Would you like show me some code please?

thanks 
Robin Shao
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: the dependence length of RDD, can its size be greater than 1 pleaae?

萝卜丝炒饭
Hi Owen,

More issues about this topic.

Is two the up limit od dependency please?
In the code,  firstParent RDD is used to get partitions.Why is firstparent RDD used please? 
If first parent RDD has 5 partitions and sevond has 6, is it reasonable to use first please?

thanks

---Original---
From: "Sean Owen"<[hidden email]>
Date: 2017/6/15 16:13:11
To: "user"<[hidden email]>;"dev"<[hidden email]>;"萝卜丝炒饭"<[hidden email]>;
Subject: Re: the dependence length of RDD, can its size be greater than 1 pleaae?

Yes. Imagine an RDD that results from a union of other RDDs.

On Thu, Jun 15, 2017, 09:11 萝卜丝炒饭 <[hidden email]> wrote:
Hi all,

The RDD code keeps a member as below:
dependencies_ : seq[Dependency[_]]

It is a seq, that means it can keep more than one dependency.

I have an issue about this.
Is it possible that its size is greater than one please?
If yes, how to produce it please? Would you like show me some code please?

thanks 
Robin Shao
Loading...