Quantcast

Filestream can not recognize the copied file

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Filestream can not recognize the copied file

萝卜丝炒饭
Hi all,
I tested filestream today, my code looks like:

val  fs = ssc.textFileStream(*)
erroelines = fs.filter( _.contains("erroe"))
erroelines.print
ssc.start()


when I edit a file and save it to the dir, it works well.
If i copy a file to the dir, it does work.

my issues are:
1, is it OK please? I means whether the code was designed to work as this.
2,which class  monitor the dir pls?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Filestream can not recognize the copied file

萝卜丝炒饭
hi all,

I checked the code just now.
Spark scan the dir and filter files by timestamp of file.
I copied files to the dir but not changed the timestamp.
I know the reason now.

thanks 

---Original---
From: "萝卜丝炒饭"<[hidden email]>
Date: 2017/2/23 17:07:33
To: "dev"<[hidden email]>;
Subject: Filestream can not recognize the copied file

Hi all,
I tested filestream today, my code looks like:

val  fs = ssc.textFileStream(*)
erroelines = fs.filter( _.contains("erroe"))
erroelines.print
ssc.start()


when I edit a file and save it to the dir, it works well.
If i copy a file to the dir, it does work.

my issues are:
1, is it OK please? I means whether the code was designed to work as this.
2,which class  monitor the dir pls?
Loading...