[discuss] SparkR CRAN feasibility check server problem

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[discuss] SparkR CRAN feasibility check server problem

Hyukjin Kwon
Hi all, 

I want to raise the CRAN failure issue because it started to block Spark PRs time to time. Since the number 
of PRs grows hugely in Spark community, this is critical to not block other PRs.

There has been a problem at CRAN (See https://github.com/apache/spark/pull/20005 for analysis). 
To cut it short, the root cause is malformed package info from https://cran.r-project.org/src/contrib/PACKAGES 
from server side, and this had to be fixed by requesting it to CRAN sysaadmin's help. 

https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am pretty sure it's the same issue
https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2 times

This happened 5 times for roughly about 10 months, causing blocking almost all PRs in Apache Spark.
Historically, it blocked whole PRs for few days once, and whole Spark community had to stop working.

I assume this has been not a super big big issue so far for other projects or other people because apparently
higher version of R has some logics to handle this malformed documents (at least I verified R 3.4.0 works fine).

For our side, Jenkins has low R version (R 3.1.1 if that's not updated from what I have seen before), 
which is unable to parse the malformed server's response.

So, I want to talk about how we are going to handle this. Possible solutions are:

1. We should start a talk with CRAN sysadmin to permanently prevent this issue
2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test low R versions)
3. ... 

If if we fine, I would like to suggest to forward this email to CRAN sysadmin to discuss further about this.

Adding Liang-Chi Felix and Shivaram who I already talked about this few times before.

Thanks all.

 

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] SparkR CRAN feasibility check server problem

Felix Cheung-2
Thanks for being this up and much appreciate with keeping on top of this at all times.

Can upgrading R able to fix the issue. Is this perhaps  not necessarily malform but some new format for new versions perhaps? Anyway we should consider upgrading R version if that fixes the problem.

As an option we could also disable the repo check in Jenkins but I can see that could also be problematic.


On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <[hidden email]> wrote:
Hi all, 

I want to raise the CRAN failure issue because it started to block Spark PRs time to time. Since the number 
of PRs grows hugely in Spark community, this is critical to not block other PRs.

There has been a problem at CRAN (See https://github.com/apache/spark/pull/20005 for analysis). 
To cut it short, the root cause is malformed package info from https://cran.r-project.org/src/contrib/PACKAGES 
from server side, and this had to be fixed by requesting it to CRAN sysaadmin's help. 

https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am pretty sure it's the same issue
https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2 times

This happened 5 times for roughly about 10 months, causing blocking almost all PRs in Apache Spark.
Historically, it blocked whole PRs for few days once, and whole Spark community had to stop working.

I assume this has been not a super big big issue so far for other projects or other people because apparently
higher version of R has some logics to handle this malformed documents (at least I verified R 3.4.0 works fine).

For our side, Jenkins has low R version (R 3.1.1 if that's not updated from what I have seen before), 
which is unable to parse the malformed server's response.

So, I want to talk about how we are going to handle this. Possible solutions are:

1. We should start a talk with CRAN sysadmin to permanently prevent this issue
2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test low R versions)
3. ... 

If if we fine, I would like to suggest to forward this email to CRAN sysadmin to discuss further about this.

Adding Liang-Chi Felix and Shivaram who I already talked about this few times before.

Thanks all.

 

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] SparkR CRAN feasibility check server problem

Hyukjin Kwon
> Can upgrading R able to fix the issue. Is this perhaps  not necessarily malform but some new format for new versions perhaps? 
That's my guess. I am not totally sure about it tho. 

> Anyway we should consider upgrading R version if that fixes the problem.
Yea, we should. If we should, it should be more them R 3.4. Maybe it's good time to start to talk about minimum R version. 3.1.x is too old. It's released 4.5 years ago.
R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0, deprecating lower versions, bumping up R to 3.4 might be reasonable option.

Adding Shane as well.

If we ended up with not upgrading it, I will forward this email to CRAN sysadmin to discuss further anyway. 



2018년 11월 2일 (금) 오후 12:51, Felix Cheung <[hidden email]>님이 작성:
Thanks for being this up and much appreciate with keeping on top of this at all times.

Can upgrading R able to fix the issue. Is this perhaps  not necessarily malform but some new format for new versions perhaps? Anyway we should consider upgrading R version if that fixes the problem.

As an option we could also disable the repo check in Jenkins but I can see that could also be problematic.


On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <[hidden email]> wrote:
Hi all, 

I want to raise the CRAN failure issue because it started to block Spark PRs time to time. Since the number 
of PRs grows hugely in Spark community, this is critical to not block other PRs.

There has been a problem at CRAN (See https://github.com/apache/spark/pull/20005 for analysis). 
To cut it short, the root cause is malformed package info from https://cran.r-project.org/src/contrib/PACKAGES 
from server side, and this had to be fixed by requesting it to CRAN sysaadmin's help. 

https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am pretty sure it's the same issue
https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2 times

This happened 5 times for roughly about 10 months, causing blocking almost all PRs in Apache Spark.
Historically, it blocked whole PRs for few days once, and whole Spark community had to stop working.

I assume this has been not a super big big issue so far for other projects or other people because apparently
higher version of R has some logics to handle this malformed documents (at least I verified R 3.4.0 works fine).

For our side, Jenkins has low R version (R 3.1.1 if that's not updated from what I have seen before), 
which is unable to parse the malformed server's response.

So, I want to talk about how we are going to handle this. Possible solutions are:

1. We should start a talk with CRAN sysadmin to permanently prevent this issue
2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test low R versions)
3. ... 

If if we fine, I would like to suggest to forward this email to CRAN sysadmin to discuss further about this.

Adding Liang-Chi Felix and Shivaram who I already talked about this few times before.

Thanks all.

 

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] SparkR CRAN feasibility check server problem

Liang-Chi Hsieh

Yeah, thanks Hyukjin Kwon for bringing this up for discussion.

I don't know how higher versions of R are widely used across R community. If
R version 3.1.x was not very commonly used, I think we can discuss to
upgrade minimum R version in next Spark version.

If we ended up with not upgrading, we can discuss with CRAN sysadmin to fix
it by the service side automatically that prevents malformed R packages
info. So we don't need to fix it manually every time.



Hyukjin Kwon wrote

>> Can upgrading R able to fix the issue. Is this perhaps  not necessarily
> malform but some new format for new versions perhaps?
> That's my guess. I am not totally sure about it tho.
>
>> Anyway we should consider upgrading R version if that fixes the problem.
> Yea, we should. If we should, it should be more them R 3.4. Maybe it's
> good
> time to start to talk about minimum R version. 3.1.x is too old. It's
> released 4.5 years ago.
> R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0,
> deprecating lower versions, bumping up R to 3.4 might be reasonable
> option.
>
> Adding Shane as well.
>
> If we ended up with not upgrading it, I will forward this email to CRAN
> sysadmin to discuss further anyway.
>
>
>
> 2018년 11월 2일 (금) 오후 12:51, Felix Cheung &lt;

> felixcheung@

> &gt;님이 작성:
>
>> Thanks for being this up and much appreciate with keeping on top of this
>> at all times.
>>
>> Can upgrading R able to fix the issue. Is this perhaps  not necessarily
>> malform but some new format for new versions perhaps? Anyway we should
>> consider upgrading R version if that fixes the problem.
>>
>> As an option we could also disable the repo check in Jenkins but I can
>> see
>> that could also be problematic.
>>
>>
>> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon &lt;

> gurwls223@

> &gt; wrote:
>>
>>> Hi all,
>>>
>>> I want to raise the CRAN failure issue because it started to block Spark
>>> PRs time to time. Since the number
>>> of PRs grows hugely in Spark community, this is critical to not block
>>> other PRs.
>>>
>>> There has been a problem at CRAN (See
>>> https://github.com/apache/spark/pull/20005 for analysis).
>>> To cut it short, the root cause is malformed package info from
>>> https://cran.r-project.org/src/contrib/PACKAGES
>>> from server side, and this had to be fixed by requesting it to CRAN
>>> sysaadmin's help.
>>>
>>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am
>>> pretty sure it's the same issue
>>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2
>>> times
>>> https://issues.apache.org/jira/browse/SPARK-22812
>>>
>>> This happened 5 times for roughly about 10 months, causing blocking
>>> almost all PRs in Apache Spark.
>>> Historically, it blocked whole PRs for few days once, and whole Spark
>>> community had to stop working.
>>>
>>> I assume this has been not a super big big issue so far for other
>>> projects or other people because apparently
>>> higher version of R has some logics to handle this malformed documents
>>> (at least I verified R 3.4.0 works fine).
>>>
>>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated
>>> from what I have seen before),
>>> which is unable to parse the malformed server's response.
>>>
>>> So, I want to talk about how we are going to handle this. Possible
>>> solutions are:
>>>
>>> 1. We should start a talk with CRAN sysadmin to permanently prevent this
>>> issue
>>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test
>>> low R versions)
>>> 3. ...
>>>
>>> If if we fine, I would like to suggest to forward this email to CRAN
>>> sysadmin to discuss further about this.
>>>
>>> Adding Liang-Chi Felix and Shivaram who I already talked about this few
>>> times before.
>>>
>>> Thanks all.
>>>
>>>
>>>
>>>





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] SparkR CRAN feasibility check server problem

Felix Cheung
It’s a great point about min R version. From what I see, mostly because of fixes and packages support, most users of R are fairly up to date? So perhaps 3.4 as min version is reasonable esp. for Spark 3.

Are we getting traction with CRAN sysadmin? It seems like this has been broken a few times.



From: Liang-Chi Hsieh <[hidden email]>
Sent: Saturday, November 10, 2018 2:32 AM
To: [hidden email]
Subject: Re: [discuss] SparkR CRAN feasibility check server problem
 

Yeah, thanks Hyukjin Kwon for bringing this up for discussion.

I don't know how higher versions of R are widely used across R community. If
R version 3.1.x was not very commonly used, I think we can discuss to
upgrade minimum R version in next Spark version.

If we ended up with not upgrading, we can discuss with CRAN sysadmin to fix
it by the service side automatically that prevents malformed R packages
info. So we don't need to fix it manually every time.



Hyukjin Kwon wrote
>> Can upgrading R able to fix the issue. Is this perhaps not necessarily
> malform but some new format for new versions perhaps?
> That's my guess. I am not totally sure about it tho.
>
>> Anyway we should consider upgrading R version if that fixes the problem.
> Yea, we should. If we should, it should be more them R 3.4. Maybe it's
> good
> time to start to talk about minimum R version. 3.1.x is too old. It's
> released 4.5 years ago.
> R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0,
> deprecating lower versions, bumping up R to 3.4 might be reasonable
> option.
>
> Adding Shane as well.
>
> If we ended up with not upgrading it, I will forward this email to CRAN
> sysadmin to discuss further anyway.
>
>
>
> 2018년 11월 2일 (금) 오후 12:51, Felix Cheung &lt;

> felixcheung@

> &gt;님이 작성:
>
>> Thanks for being this up and much appreciate with keeping on top of this
>> at all times.
>>
>> Can upgrading R able to fix the issue. Is this perhaps not necessarily
>> malform but some new format for new versions perhaps? Anyway we should
>> consider upgrading R version if that fixes the problem.
>>
>> As an option we could also disable the repo check in Jenkins but I can
>> see
>> that could also be problematic.
>>
>>
>> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon &lt;

> gurwls223@

> &gt; wrote:
>>
>>> Hi all,
>>>
>>> I want to raise the CRAN failure issue because it started to block Spark
>>> PRs time to time. Since the number
>>> of PRs grows hugely in Spark community, this is critical to not block
>>> other PRs.
>>>
>>> There has been a problem at CRAN (See
>>> https://github.com/apache/spark/pull/20005 for analysis).
>>> To cut it short, the root cause is malformed package info from
>>> https://cran.r-project.org/src/contrib/PACKAGES
>>> from server side, and this had to be fixed by requesting it to CRAN
>>> sysaadmin's help.
>>>
>>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am
>>> pretty sure it's the same issue
>>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2
>>> times
>>> https://issues.apache.org/jira/browse/SPARK-22812
>>>
>>> This happened 5 times for roughly about 10 months, causing blocking
>>> almost all PRs in Apache Spark.
>>> Historically, it blocked whole PRs for few days once, and whole Spark
>>> community had to stop working.
>>>
>>> I assume this has been not a super big big issue so far for other
>>> projects or other people because apparently
>>> higher version of R has some logics to handle this malformed documents
>>> (at least I verified R 3.4.0 works fine).
>>>
>>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated
>>> from what I have seen before),
>>> which is unable to parse the malformed server's response.
>>>
>>> So, I want to talk about how we are going to handle this. Possible
>>> solutions are:
>>>
>>> 1. We should start a talk with CRAN sysadmin to permanently prevent this
>>> issue
>>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test
>>> low R versions)
>>> 3. ...
>>>
>>> If if we fine, I would like to suggest to forward this email to CRAN
>>> sysadmin to discuss further about this.
>>>
>>> Adding Liang-Chi Felix and Shivaram who I already talked about this few
>>> times before.
>>>
>>> Thanks all.
>>>
>>>
>>>
>>>





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] SparkR CRAN feasibility check server problem

Hyukjin Kwon
I made a PR to officially drop R prior to version 3.4 (https://github.com/apache/spark/pull/23012).
The tests will probably fail for now since it produces warnings for using R 3.1.x.

2018년 11월 11일 (일) 오전 3:00, Felix Cheung <[hidden email]>님이 작성:
It’s a great point about min R version. From what I see, mostly because of fixes and packages support, most users of R are fairly up to date? So perhaps 3.4 as min version is reasonable esp. for Spark 3.

Are we getting traction with CRAN sysadmin? It seems like this has been broken a few times.



From: Liang-Chi Hsieh <[hidden email]>
Sent: Saturday, November 10, 2018 2:32 AM
To: [hidden email]
Subject: Re: [discuss] SparkR CRAN feasibility check server problem
 

Yeah, thanks Hyukjin Kwon for bringing this up for discussion.

I don't know how higher versions of R are widely used across R community. If
R version 3.1.x was not very commonly used, I think we can discuss to
upgrade minimum R version in next Spark version.

If we ended up with not upgrading, we can discuss with CRAN sysadmin to fix
it by the service side automatically that prevents malformed R packages
info. So we don't need to fix it manually every time.



Hyukjin Kwon wrote
>> Can upgrading R able to fix the issue. Is this perhaps not necessarily
> malform but some new format for new versions perhaps?
> That's my guess. I am not totally sure about it tho.
>
>> Anyway we should consider upgrading R version if that fixes the problem.
> Yea, we should. If we should, it should be more them R 3.4. Maybe it's
> good
> time to start to talk about minimum R version. 3.1.x is too old. It's
> released 4.5 years ago.
> R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0,
> deprecating lower versions, bumping up R to 3.4 might be reasonable
> option.
>
> Adding Shane as well.
>
> If we ended up with not upgrading it, I will forward this email to CRAN
> sysadmin to discuss further anyway.
>
>
>
> 2018년 11월 2일 (금) 오후 12:51, Felix Cheung &lt;

> felixcheung@

> &gt;님이 작성:
>
>> Thanks for being this up and much appreciate with keeping on top of this
>> at all times.
>>
>> Can upgrading R able to fix the issue. Is this perhaps not necessarily
>> malform but some new format for new versions perhaps? Anyway we should
>> consider upgrading R version if that fixes the problem.
>>
>> As an option we could also disable the repo check in Jenkins but I can
>> see
>> that could also be problematic.
>>
>>
>> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon &lt;

> gurwls223@

> &gt; wrote:
>>
>>> Hi all,
>>>
>>> I want to raise the CRAN failure issue because it started to block Spark
>>> PRs time to time. Since the number
>>> of PRs grows hugely in Spark community, this is critical to not block
>>> other PRs.
>>>
>>> There has been a problem at CRAN (See
>>> https://github.com/apache/spark/pull/20005 for analysis).
>>> To cut it short, the root cause is malformed package info from
>>> https://cran.r-project.org/src/contrib/PACKAGES
>>> from server side, and this had to be fixed by requesting it to CRAN
>>> sysaadmin's help.
>>>
>>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am
>>> pretty sure it's the same issue
>>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2
>>> times
>>> https://issues.apache.org/jira/browse/SPARK-22812
>>>
>>> This happened 5 times for roughly about 10 months, causing blocking
>>> almost all PRs in Apache Spark.
>>> Historically, it blocked whole PRs for few days once, and whole Spark
>>> community had to stop working.
>>>
>>> I assume this has been not a super big big issue so far for other
>>> projects or other people because apparently
>>> higher version of R has some logics to handle this malformed documents
>>> (at least I verified R 3.4.0 works fine).
>>>
>>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated
>>> from what I have seen before),
>>> which is unable to parse the malformed server's response.
>>>
>>> So, I want to talk about how we are going to handle this. Possible
>>> solutions are:
>>>
>>> 1. We should start a talk with CRAN sysadmin to permanently prevent this
>>> issue
>>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test
>>> low R versions)
>>> 3. ...
>>>
>>> If if we fine, I would like to suggest to forward this email to CRAN
>>> sysadmin to discuss further about this.
>>>
>>> Adding Liang-Chi Felix and Shivaram who I already talked about this few
>>> times before.
>>>
>>> Thanks all.
>>>
>>>
>>>
>>>





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] SparkR CRAN feasibility check server problem

Hyukjin Kwon
Looks it's happening again. Liang-Chi, do you mind if I ask it again?

FYI, R 3.4 is officially deprecated as of https://github.com/apache/spark/pull/23012
We could upgrade R version to 3.4.x in Jenkins, which deals with the malformed(?) responses after 3.0 release.
Then, we could get rid of this problem..!

2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon <[hidden email]>님이 작성:
I made a PR to officially drop R prior to version 3.4 (https://github.com/apache/spark/pull/23012).
The tests will probably fail for now since it produces warnings for using R 3.1.x.

2018년 11월 11일 (일) 오전 3:00, Felix Cheung <[hidden email]>님이 작성:
It’s a great point about min R version. From what I see, mostly because of fixes and packages support, most users of R are fairly up to date? So perhaps 3.4 as min version is reasonable esp. for Spark 3.

Are we getting traction with CRAN sysadmin? It seems like this has been broken a few times.



From: Liang-Chi Hsieh <[hidden email]>
Sent: Saturday, November 10, 2018 2:32 AM
To: [hidden email]
Subject: Re: [discuss] SparkR CRAN feasibility check server problem
 

Yeah, thanks Hyukjin Kwon for bringing this up for discussion.

I don't know how higher versions of R are widely used across R community. If
R version 3.1.x was not very commonly used, I think we can discuss to
upgrade minimum R version in next Spark version.

If we ended up with not upgrading, we can discuss with CRAN sysadmin to fix
it by the service side automatically that prevents malformed R packages
info. So we don't need to fix it manually every time.



Hyukjin Kwon wrote
>> Can upgrading R able to fix the issue. Is this perhaps not necessarily
> malform but some new format for new versions perhaps?
> That's my guess. I am not totally sure about it tho.
>
>> Anyway we should consider upgrading R version if that fixes the problem.
> Yea, we should. If we should, it should be more them R 3.4. Maybe it's
> good
> time to start to talk about minimum R version. 3.1.x is too old. It's
> released 4.5 years ago.
> R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0,
> deprecating lower versions, bumping up R to 3.4 might be reasonable
> option.
>
> Adding Shane as well.
>
> If we ended up with not upgrading it, I will forward this email to CRAN
> sysadmin to discuss further anyway.
>
>
>
> 2018년 11월 2일 (금) 오후 12:51, Felix Cheung &lt;

> felixcheung@

> &gt;님이 작성:
>
>> Thanks for being this up and much appreciate with keeping on top of this
>> at all times.
>>
>> Can upgrading R able to fix the issue. Is this perhaps not necessarily
>> malform but some new format for new versions perhaps? Anyway we should
>> consider upgrading R version if that fixes the problem.
>>
>> As an option we could also disable the repo check in Jenkins but I can
>> see
>> that could also be problematic.
>>
>>
>> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon &lt;

> gurwls223@

> &gt; wrote:
>>
>>> Hi all,
>>>
>>> I want to raise the CRAN failure issue because it started to block Spark
>>> PRs time to time. Since the number
>>> of PRs grows hugely in Spark community, this is critical to not block
>>> other PRs.
>>>
>>> There has been a problem at CRAN (See
>>> https://github.com/apache/spark/pull/20005 for analysis).
>>> To cut it short, the root cause is malformed package info from
>>> https://cran.r-project.org/src/contrib/PACKAGES
>>> from server side, and this had to be fixed by requesting it to CRAN
>>> sysaadmin's help.
>>>
>>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am
>>> pretty sure it's the same issue
>>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2
>>> times
>>> https://issues.apache.org/jira/browse/SPARK-22812
>>>
>>> This happened 5 times for roughly about 10 months, causing blocking
>>> almost all PRs in Apache Spark.
>>> Historically, it blocked whole PRs for few days once, and whole Spark
>>> community had to stop working.
>>>
>>> I assume this has been not a super big big issue so far for other
>>> projects or other people because apparently
>>> higher version of R has some logics to handle this malformed documents
>>> (at least I verified R 3.4.0 works fine).
>>>
>>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated
>>> from what I have seen before),
>>> which is unable to parse the malformed server's response.
>>>
>>> So, I want to talk about how we are going to handle this. Possible
>>> solutions are:
>>>
>>> 1. We should start a talk with CRAN sysadmin to permanently prevent this
>>> issue
>>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test
>>> low R versions)
>>> 3. ...
>>>
>>> If if we fine, I would like to suggest to forward this email to CRAN
>>> sysadmin to discuss further about this.
>>>
>>> Adding Liang-Chi Felix and Shivaram who I already talked about this few
>>> times before.
>>>
>>> Thanks all.
>>>
>>>
>>>
>>>





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] SparkR CRAN feasibility check server problem

Liang-Chi Hsieh

Thanks for letting me know! I will look into it and ask CRAN admin for help.


Hyukjin Kwon wrote
> Looks it's happening again. Liang-Chi, do you mind if I ask it again?
>
> FYI, R 3.4 is officially deprecated as of
> https://github.com/apache/spark/pull/23012
> We could upgrade R version to 3.4.x in Jenkins, which deals with the
> malformed(?) responses after 3.0 release.
> Then, we could get rid of this problem..!
>
> 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon &lt;

> gurwls223@

> &gt;님이 작성:
>
>> I made a PR to officially drop R prior to version 3.4 (
>> https://github.com/apache/spark/pull/23012).
>> The tests will probably fail for now since it produces warnings for using
>> R 3.1.x.
>>
>> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung &lt;

> felixcheung_m@

> &gt;님이 작성:
>>
>>> It’s a great point about min R version. From what I see, mostly because
>>> of fixes and packages support, most users of R are fairly up to date? So
>>> perhaps 3.4 as min version is reasonable esp. for Spark 3.
>>>
>>> Are we getting traction with CRAN sysadmin? It seems like this has been
>>> broken a few times.
>>>
>>>
>>> ------------------------------
>>> *From:* Liang-Chi Hsieh &lt;

> viirya@

> &gt;
>>> *Sent:* Saturday, November 10, 2018 2:32 AM
>>> *To:*

> dev@.apache

>>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem
>>>
>>>
>>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
>>>
>>> I don't know how higher versions of R are widely used across R
>>> community.
>>> If
>>> R version 3.1.x was not very commonly used, I think we can discuss to
>>> upgrade minimum R version in next Spark version.
>>>
>>> If we ended up with not upgrading, we can discuss with CRAN sysadmin to
>>> fix
>>> it by the service side automatically that prevents malformed R packages
>>> info. So we don't need to fix it manually every time.
>>>
>>>
>>>
>>> Hyukjin Kwon wrote
>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>>> necessarily
>>> > malform but some new format for new versions perhaps?
>>> > That's my guess. I am not totally sure about it tho.
>>> >
>>> >> Anyway we should consider upgrading R version if that fixes the
>>> problem.
>>> > Yea, we should. If we should, it should be more them R 3.4. Maybe it's
>>> > good
>>> > time to start to talk about minimum R version. 3.1.x is too old. It's
>>> > released 4.5 years ago.
>>> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark
>>> 3.0,
>>> > deprecating lower versions, bumping up R to 3.4 might be reasonable
>>> > option.
>>> >
>>> > Adding Shane as well.
>>> >
>>> > If we ended up with not upgrading it, I will forward this email to
>>> CRAN
>>> > sysadmin to discuss further anyway.
>>> >
>>> >
>>> >
>>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung &lt;
>>>
>>> > felixcheung@
>>>
>>> > &gt;님이 작성:
>>> >
>>> >> Thanks for being this up and much appreciate with keeping on top of
>>> this
>>> >> at all times.
>>> >>
>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>>> necessarily
>>> >> malform but some new format for new versions perhaps? Anyway we
>>> should
>>> >> consider upgrading R version if that fixes the problem.
>>> >>
>>> >> As an option we could also disable the repo check in Jenkins but I
>>> can
>>> >> see
>>> >> that could also be problematic.
>>> >>
>>> >>
>>> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon &lt;
>>>
>>> > gurwls223@
>>>
>>> > &gt; wrote:
>>> >>
>>> >>> Hi all,
>>> >>>
>>> >>> I want to raise the CRAN failure issue because it started to block
>>> Spark
>>> >>> PRs time to time. Since the number
>>> >>> of PRs grows hugely in Spark community, this is critical to not
>>> block
>>> >>> other PRs.
>>> >>>
>>> >>> There has been a problem at CRAN (See
>>> >>> https://github.com/apache/spark/pull/20005 for analysis).
>>> >>> To cut it short, the root cause is malformed package info from
>>> >>> https://cran.r-project.org/src/contrib/PACKAGES
>>> >>> from server side, and this had to be fixed by requesting it to CRAN
>>> >>> sysaadmin's help.
>>> >>>
>>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I
>>> am
>>> >>> pretty sure it's the same issue
>>> >>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved
>>> 2
>>> >>> times
>>> >>> https://issues.apache.org/jira/browse/SPARK-22812
>>> >>>
>>> >>> This happened 5 times for roughly about 10 months, causing blocking
>>> >>> almost all PRs in Apache Spark.
>>> >>> Historically, it blocked whole PRs for few days once, and whole
>>> Spark
>>> >>> community had to stop working.
>>> >>>
>>> >>> I assume this has been not a super big big issue so far for other
>>> >>> projects or other people because apparently
>>> >>> higher version of R has some logics to handle this malformed
>>> documents
>>> >>> (at least I verified R 3.4.0 works fine).
>>> >>>
>>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not
>>> updated
>>> >>> from what I have seen before),
>>> >>> which is unable to parse the malformed server's response.
>>> >>>
>>> >>> So, I want to talk about how we are going to handle this. Possible
>>> >>> solutions are:
>>> >>>
>>> >>> 1. We should start a talk with CRAN sysadmin to permanently prevent
>>> this
>>> >>> issue
>>> >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to
>>> test
>>> >>> low R versions)
>>> >>> 3. ...
>>> >>>
>>> >>> If if we fine, I would like to suggest to forward this email to CRAN
>>> >>> sysadmin to discuss further about this.
>>> >>>
>>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about this
>>> few
>>> >>> times before.
>>> >>>
>>> >>> Thanks all.
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail:

> dev-unsubscribe@.apache

>>>
>>>





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] SparkR CRAN feasibility check server problem

Liang-Chi Hsieh

Sorry for late. There is a malformed record at CRAN package page again. I've
already asked CRAN admin for help. It should be fixed soon according to past
experience.

Related discussion will be in
https://issues.apache.org/jira/browse/SPARK-24152. I will post here if I get
reply from CRAN admin.

Thanks.


Liang-Chi Hsieh wrote

> Thanks for letting me know! I will look into it and ask CRAN admin for
> help.
>
>
> Hyukjin Kwon wrote
>> Looks it's happening again. Liang-Chi, do you mind if I ask it again?
>>
>> FYI, R 3.4 is officially deprecated as of
>> https://github.com/apache/spark/pull/23012
>> We could upgrade R version to 3.4.x in Jenkins, which deals with the
>> malformed(?) responses after 3.0 release.
>> Then, we could get rid of this problem..!
>>
>> 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon &lt;
>
>> gurwls223@
>
>> &gt;님이 작성:
>>
>>> I made a PR to officially drop R prior to version 3.4 (
>>> https://github.com/apache/spark/pull/23012).
>>> The tests will probably fail for now since it produces warnings for
>>> using
>>> R 3.1.x.
>>>
>>> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung &lt;
>
>> felixcheung_m@
>
>> &gt;님이 작성:
>>>
>>>> It’s a great point about min R version. From what I see, mostly because
>>>> of fixes and packages support, most users of R are fairly up to date?
>>>> So
>>>> perhaps 3.4 as min version is reasonable esp. for Spark 3.
>>>>
>>>> Are we getting traction with CRAN sysadmin? It seems like this has been
>>>> broken a few times.
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* Liang-Chi Hsieh &lt;
>
>> viirya@
>
>> &gt;
>>>> *Sent:* Saturday, November 10, 2018 2:32 AM
>>>> *To:*
>
>> dev@.apache
>
>>>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem
>>>>
>>>>
>>>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
>>>>
>>>> I don't know how higher versions of R are widely used across R
>>>> community.
>>>> If
>>>> R version 3.1.x was not very commonly used, I think we can discuss to
>>>> upgrade minimum R version in next Spark version.
>>>>
>>>> If we ended up with not upgrading, we can discuss with CRAN sysadmin to
>>>> fix
>>>> it by the service side automatically that prevents malformed R packages
>>>> info. So we don't need to fix it manually every time.
>>>>
>>>>
>>>>
>>>> Hyukjin Kwon wrote
>>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>>>> necessarily
>>>> > malform but some new format for new versions perhaps?
>>>> > That's my guess. I am not totally sure about it tho.
>>>> >
>>>> >> Anyway we should consider upgrading R version if that fixes the
>>>> problem.
>>>> > Yea, we should. If we should, it should be more them R 3.4. Maybe
>>>> it's
>>>> > good
>>>> > time to start to talk about minimum R version. 3.1.x is too old. It's
>>>> > released 4.5 years ago.
>>>> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark
>>>> 3.0,
>>>> > deprecating lower versions, bumping up R to 3.4 might be reasonable
>>>> > option.
>>>> >
>>>> > Adding Shane as well.
>>>> >
>>>> > If we ended up with not upgrading it, I will forward this email to
>>>> CRAN
>>>> > sysadmin to discuss further anyway.
>>>> >
>>>> >
>>>> >
>>>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung &lt;
>>>>
>>>> > felixcheung@
>>>>
>>>> > &gt;님이 작성:
>>>> >
>>>> >> Thanks for being this up and much appreciate with keeping on top of
>>>> this
>>>> >> at all times.
>>>> >>
>>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>>>> necessarily
>>>> >> malform but some new format for new versions perhaps? Anyway we
>>>> should
>>>> >> consider upgrading R version if that fixes the problem.
>>>> >>
>>>> >> As an option we could also disable the repo check in Jenkins but I
>>>> can
>>>> >> see
>>>> >> that could also be problematic.
>>>> >>
>>>> >>
>>>> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon &lt;
>>>>
>>>> > gurwls223@
>>>>
>>>> > &gt; wrote:
>>>> >>
>>>> >>> Hi all,
>>>> >>>
>>>> >>> I want to raise the CRAN failure issue because it started to block
>>>> Spark
>>>> >>> PRs time to time. Since the number
>>>> >>> of PRs grows hugely in Spark community, this is critical to not
>>>> block
>>>> >>> other PRs.
>>>> >>>
>>>> >>> There has been a problem at CRAN (See
>>>> >>> https://github.com/apache/spark/pull/20005 for analysis).
>>>> >>> To cut it short, the root cause is malformed package info from
>>>> >>> https://cran.r-project.org/src/contrib/PACKAGES
>>>> >>> from server side, and this had to be fixed by requesting it to CRAN
>>>> >>> sysaadmin's help.
>>>> >>>
>>>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I
>>>> am
>>>> >>> pretty sure it's the same issue
>>>> >>> https://issues.apache.org/jira/browse/SPARK-25923 <-
>>>> reopen/resolved
>>>> 2
>>>> >>> times
>>>> >>> https://issues.apache.org/jira/browse/SPARK-22812
>>>> >>>
>>>> >>> This happened 5 times for roughly about 10 months, causing blocking
>>>> >>> almost all PRs in Apache Spark.
>>>> >>> Historically, it blocked whole PRs for few days once, and whole
>>>> Spark
>>>> >>> community had to stop working.
>>>> >>>
>>>> >>> I assume this has been not a super big big issue so far for other
>>>> >>> projects or other people because apparently
>>>> >>> higher version of R has some logics to handle this malformed
>>>> documents
>>>> >>> (at least I verified R 3.4.0 works fine).
>>>> >>>
>>>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not
>>>> updated
>>>> >>> from what I have seen before),
>>>> >>> which is unable to parse the malformed server's response.
>>>> >>>
>>>> >>> So, I want to talk about how we are going to handle this. Possible
>>>> >>> solutions are:
>>>> >>>
>>>> >>> 1. We should start a talk with CRAN sysadmin to permanently prevent
>>>> this
>>>> >>> issue
>>>> >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to
>>>> test
>>>> >>> low R versions)
>>>> >>> 3. ...
>>>> >>>
>>>> >>> If if we fine, I would like to suggest to forward this email to
>>>> CRAN
>>>> >>> sysadmin to discuss further about this.
>>>> >>>
>>>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about this
>>>> few
>>>> >>> times before.
>>>> >>>
>>>> >>> Thanks all.
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail:
>
>> dev-unsubscribe@.apache
>
>>>>
>>>>
>
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

> dev-unsubscribe@.apache





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] SparkR CRAN feasibility check server problem

Hyukjin Kwon
Thanks, Liang-chi.

On Thu, 13 Dec 2018, 8:29 am Liang-Chi Hsieh <[hidden email] wrote:

Sorry for late. There is a malformed record at CRAN package page again. I've
already asked CRAN admin for help. It should be fixed soon according to past
experience.

Related discussion will be in
https://issues.apache.org/jira/browse/SPARK-24152. I will post here if I get
reply from CRAN admin.

Thanks.


Liang-Chi Hsieh wrote
> Thanks for letting me know! I will look into it and ask CRAN admin for
> help.
>
>
> Hyukjin Kwon wrote
>> Looks it's happening again. Liang-Chi, do you mind if I ask it again?
>>
>> FYI, R 3.4 is officially deprecated as of
>> https://github.com/apache/spark/pull/23012
>> We could upgrade R version to 3.4.x in Jenkins, which deals with the
>> malformed(?) responses after 3.0 release.
>> Then, we could get rid of this problem..!
>>
>> 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon &lt;
>
>> gurwls223@
>
>> &gt;님이 작성:
>>
>>> I made a PR to officially drop R prior to version 3.4 (
>>> https://github.com/apache/spark/pull/23012).
>>> The tests will probably fail for now since it produces warnings for
>>> using
>>> R 3.1.x.
>>>
>>> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung &lt;
>
>> felixcheung_m@
>
>> &gt;님이 작성:
>>>
>>>> It’s a great point about min R version. From what I see, mostly because
>>>> of fixes and packages support, most users of R are fairly up to date?
>>>> So
>>>> perhaps 3.4 as min version is reasonable esp. for Spark 3.
>>>>
>>>> Are we getting traction with CRAN sysadmin? It seems like this has been
>>>> broken a few times.
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* Liang-Chi Hsieh &lt;
>
>> viirya@
>
>> &gt;
>>>> *Sent:* Saturday, November 10, 2018 2:32 AM
>>>> *To:*
>
>> dev@.apache
>
>>>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem
>>>>
>>>>
>>>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
>>>>
>>>> I don't know how higher versions of R are widely used across R
>>>> community.
>>>> If
>>>> R version 3.1.x was not very commonly used, I think we can discuss to
>>>> upgrade minimum R version in next Spark version.
>>>>
>>>> If we ended up with not upgrading, we can discuss with CRAN sysadmin to
>>>> fix
>>>> it by the service side automatically that prevents malformed R packages
>>>> info. So we don't need to fix it manually every time.
>>>>
>>>>
>>>>
>>>> Hyukjin Kwon wrote
>>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>>>> necessarily
>>>> > malform but some new format for new versions perhaps?
>>>> > That's my guess. I am not totally sure about it tho.
>>>> >
>>>> >> Anyway we should consider upgrading R version if that fixes the
>>>> problem.
>>>> > Yea, we should. If we should, it should be more them R 3.4. Maybe
>>>> it's
>>>> > good
>>>> > time to start to talk about minimum R version. 3.1.x is too old. It's
>>>> > released 4.5 years ago.
>>>> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark
>>>> 3.0,
>>>> > deprecating lower versions, bumping up R to 3.4 might be reasonable
>>>> > option.
>>>> >
>>>> > Adding Shane as well.
>>>> >
>>>> > If we ended up with not upgrading it, I will forward this email to
>>>> CRAN
>>>> > sysadmin to discuss further anyway.
>>>> >
>>>> >
>>>> >
>>>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung &lt;
>>>>
>>>> > felixcheung@
>>>>
>>>> > &gt;님이 작성:
>>>> >
>>>> >> Thanks for being this up and much appreciate with keeping on top of
>>>> this
>>>> >> at all times.
>>>> >>
>>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>>>> necessarily
>>>> >> malform but some new format for new versions perhaps? Anyway we
>>>> should
>>>> >> consider upgrading R version if that fixes the problem.
>>>> >>
>>>> >> As an option we could also disable the repo check in Jenkins but I
>>>> can
>>>> >> see
>>>> >> that could also be problematic.
>>>> >>
>>>> >>
>>>> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon &lt;
>>>>
>>>> > gurwls223@
>>>>
>>>> > &gt; wrote:
>>>> >>
>>>> >>> Hi all,
>>>> >>>
>>>> >>> I want to raise the CRAN failure issue because it started to block
>>>> Spark
>>>> >>> PRs time to time. Since the number
>>>> >>> of PRs grows hugely in Spark community, this is critical to not
>>>> block
>>>> >>> other PRs.
>>>> >>>
>>>> >>> There has been a problem at CRAN (See
>>>> >>> https://github.com/apache/spark/pull/20005 for analysis).
>>>> >>> To cut it short, the root cause is malformed package info from
>>>> >>> https://cran.r-project.org/src/contrib/PACKAGES
>>>> >>> from server side, and this had to be fixed by requesting it to CRAN
>>>> >>> sysaadmin's help.
>>>> >>>
>>>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I
>>>> am
>>>> >>> pretty sure it's the same issue
>>>> >>> https://issues.apache.org/jira/browse/SPARK-25923 <-
>>>> reopen/resolved
>>>> 2
>>>> >>> times
>>>> >>> https://issues.apache.org/jira/browse/SPARK-22812
>>>> >>>
>>>> >>> This happened 5 times for roughly about 10 months, causing blocking
>>>> >>> almost all PRs in Apache Spark.
>>>> >>> Historically, it blocked whole PRs for few days once, and whole
>>>> Spark
>>>> >>> community had to stop working.
>>>> >>>
>>>> >>> I assume this has been not a super big big issue so far for other
>>>> >>> projects or other people because apparently
>>>> >>> higher version of R has some logics to handle this malformed
>>>> documents
>>>> >>> (at least I verified R 3.4.0 works fine).
>>>> >>>
>>>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not
>>>> updated
>>>> >>> from what I have seen before),
>>>> >>> which is unable to parse the malformed server's response.
>>>> >>>
>>>> >>> So, I want to talk about how we are going to handle this. Possible
>>>> >>> solutions are:
>>>> >>>
>>>> >>> 1. We should start a talk with CRAN sysadmin to permanently prevent
>>>> this
>>>> >>> issue
>>>> >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to
>>>> test
>>>> >>> low R versions)
>>>> >>> 3. ...
>>>> >>>
>>>> >>> If if we fine, I would like to suggest to forward this email to
>>>> CRAN
>>>> >>> sysadmin to discuss further about this.
>>>> >>>
>>>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about this
>>>> few
>>>> >>> times before.
>>>> >>>
>>>> >>> Thanks all.
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail:
>
>> dev-unsubscribe@.apache
>
>>>>
>>>>
>
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

> dev-unsubscribe@.apache





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] SparkR CRAN feasibility check server problem

Liang-Chi Hsieh

Just got reply from CRAN admin. It should be fixed now.


Hyukjin Kwon wrote
> Thanks, Liang-chi.
>
> On Thu, 13 Dec 2018, 8:29 am Liang-Chi Hsieh &lt;

> viirya@

>  wrote:
>
> &gt;
>> Sorry for late. There is a malformed record at CRAN package page again.
>> I've
>> already asked CRAN admin for help. It should be fixed soon according to
>> past
>> experience.
>>
>> Related discussion will be in
>> https://issues.apache.org/jira/browse/SPARK-24152. I will post here if I
>> get
>> reply from CRAN admin.
>>
>> Thanks.
>>
>>
>> Liang-Chi Hsieh wrote
>> > Thanks for letting me know! I will look into it and ask CRAN admin for
>> > help.
>> >
>> >
>> > Hyukjin Kwon wrote
>> >> Looks it's happening again. Liang-Chi, do you mind if I ask it again?
>> >>
>> >> FYI, R 3.4 is officially deprecated as of
>> >> https://github.com/apache/spark/pull/23012
>> >> We could upgrade R version to 3.4.x in Jenkins, which deals with the
>> >> malformed(?) responses after 3.0 release.
>> >> Then, we could get rid of this problem..!
>> >>
>> >> 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon &lt;
>> >
>> >> gurwls223@
>> >
>> >> &gt;님이 작성:
>> >>
>> >>> I made a PR to officially drop R prior to version 3.4 (
>> >>> https://github.com/apache/spark/pull/23012).
>> >>> The tests will probably fail for now since it produces warnings for
>> >>> using
>> >>> R 3.1.x.
>> >>>
>> >>> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung &lt;
>> >
>> >> felixcheung_m@
>> >
>> >> &gt;님이 작성:
>> >>>
>> >>>> It’s a great point about min R version. From what I see, mostly
>> because
>> >>>> of fixes and packages support, most users of R are fairly up to
>> date?
>> >>>> So
>> >>>> perhaps 3.4 as min version is reasonable esp. for Spark 3.
>> >>>>
>> >>>> Are we getting traction with CRAN sysadmin? It seems like this has
>> been
>> >>>> broken a few times.
>> >>>>
>> >>>>
>> >>>> ------------------------------
>> >>>> *From:* Liang-Chi Hsieh &lt;
>> >
>> >> viirya@
>> >
>> >> &gt;
>> >>>> *Sent:* Saturday, November 10, 2018 2:32 AM
>> >>>> *To:*
>> >
>> >> dev@.apache
>> >
>> >>>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server
>> problem
>> >>>>
>> >>>>
>> >>>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
>> >>>>
>> >>>> I don't know how higher versions of R are widely used across R
>> >>>> community.
>> >>>> If
>> >>>> R version 3.1.x was not very commonly used, I think we can discuss
>> to
>> >>>> upgrade minimum R version in next Spark version.
>> >>>>
>> >>>> If we ended up with not upgrading, we can discuss with CRAN sysadmin
>> to
>> >>>> fix
>> >>>> it by the service side automatically that prevents malformed R
>> packages
>> >>>> info. So we don't need to fix it manually every time.
>> >>>>
>> >>>>
>> >>>>
>> >>>> Hyukjin Kwon wrote
>> >>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>> >>>> necessarily
>> >>>> > malform but some new format for new versions perhaps?
>> >>>> > That's my guess. I am not totally sure about it tho.
>> >>>> >
>> >>>> >> Anyway we should consider upgrading R version if that fixes the
>> >>>> problem.
>> >>>> > Yea, we should. If we should, it should be more them R 3.4. Maybe
>> >>>> it's
>> >>>> > good
>> >>>> > time to start to talk about minimum R version. 3.1.x is too old.
>> It's
>> >>>> > released 4.5 years ago.
>> >>>> > R 3.4.0 is released 1.5 years ago. Considering the timing for
>> Spark
>> >>>> 3.0,
>> >>>> > deprecating lower versions, bumping up R to 3.4 might be
>> reasonable
>> >>>> > option.
>> >>>> >
>> >>>> > Adding Shane as well.
>> >>>> >
>> >>>> > If we ended up with not upgrading it, I will forward this email to
>> >>>> CRAN
>> >>>> > sysadmin to discuss further anyway.
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung &lt;
>> >>>>
>> >>>> > felixcheung@
>> >>>>
>> >>>> > &gt;님이 작성:
>> >>>> >
>> >>>> >> Thanks for being this up and much appreciate with keeping on top
>> of
>> >>>> this
>> >>>> >> at all times.
>> >>>> >>
>> >>>> >> Can upgrading R able to fix the issue. Is this perhaps not
>> >>>> necessarily
>> >>>> >> malform but some new format for new versions perhaps? Anyway we
>> >>>> should
>> >>>> >> consider upgrading R version if that fixes the problem.
>> >>>> >>
>> >>>> >> As an option we could also disable the repo check in Jenkins but
>> I
>> >>>> can
>> >>>> >> see
>> >>>> >> that could also be problematic.
>> >>>> >>
>> >>>> >>
>> >>>> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon &lt;
>> >>>>
>> >>>> > gurwls223@
>> >>>>
>> >>>> > &gt; wrote:
>> >>>> >>
>> >>>> >>> Hi all,
>> >>>> >>>
>> >>>> >>> I want to raise the CRAN failure issue because it started to
>> block
>> >>>> Spark
>> >>>> >>> PRs time to time. Since the number
>> >>>> >>> of PRs grows hugely in Spark community, this is critical to not
>> >>>> block
>> >>>> >>> other PRs.
>> >>>> >>>
>> >>>> >>> There has been a problem at CRAN (See
>> >>>> >>> https://github.com/apache/spark/pull/20005 for analysis).
>> >>>> >>> To cut it short, the root cause is malformed package info from
>> >>>> >>> https://cran.r-project.org/src/contrib/PACKAGES
>> >>>> >>> from server side, and this had to be fixed by requesting it to
>> CRAN
>> >>>> >>> sysaadmin's help.
>> >>>> >>>
>> >>>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open.
>> I
>> >>>> am
>> >>>> >>> pretty sure it's the same issue
>> >>>> >>> https://issues.apache.org/jira/browse/SPARK-25923 <-
>> >>>> reopen/resolved
>> >>>> 2
>> >>>> >>> times
>> >>>> >>> https://issues.apache.org/jira/browse/SPARK-22812
>> >>>> >>>
>> >>>> >>> This happened 5 times for roughly about 10 months, causing
>> blocking
>> >>>> >>> almost all PRs in Apache Spark.
>> >>>> >>> Historically, it blocked whole PRs for few days once, and whole
>> >>>> Spark
>> >>>> >>> community had to stop working.
>> >>>> >>>
>> >>>> >>> I assume this has been not a super big big issue so far for
>> other
>> >>>> >>> projects or other people because apparently
>> >>>> >>> higher version of R has some logics to handle this malformed
>> >>>> documents
>> >>>> >>> (at least I verified R 3.4.0 works fine).
>> >>>> >>>
>> >>>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not
>> >>>> updated
>> >>>> >>> from what I have seen before),
>> >>>> >>> which is unable to parse the malformed server's response.
>> >>>> >>>
>> >>>> >>> So, I want to talk about how we are going to handle this.
>> Possible
>> >>>> >>> solutions are:
>> >>>> >>>
>> >>>> >>> 1. We should start a talk with CRAN sysadmin to permanently
>> prevent
>> >>>> this
>> >>>> >>> issue
>> >>>> >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able
>> to
>> >>>> test
>> >>>> >>> low R versions)
>> >>>> >>> 3. ...
>> >>>> >>>
>> >>>> >>> If if we fine, I would like to suggest to forward this email to
>> >>>> CRAN
>> >>>> >>> sysadmin to discuss further about this.
>> >>>> >>>
>> >>>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about
>> this
>> >>>> few
>> >>>> >>> times before.
>> >>>> >>>
>> >>>> >>> Thanks all.
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Sent from:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/
>> >>>>
>> >>>>
>> ---------------------------------------------------------------------
>> >>>> To unsubscribe e-mail:
>> >
>> >> dev-unsubscribe@.apache
>> >
>> >>>>
>> >>>>
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe e-mail:
>>
>> > dev-unsubscribe@.apache
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail:

> dev-unsubscribe@.apache

>>
>>





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]