flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anandkumar Lakshmanan <an...@orzota.com>
Subject Re: how flume identifies a file transfer is complete or not
Date Sat, 26 Jul 2014 04:04:03 GMT
Hi Natty,

Thanks for the Reply.

So far I am verifying the transfer is complete or not by checking the 
file in the destination  or as you mentioned only.


On 07/25/2014 11:22 PM, Jonathan Natkins wrote:
> Hi Anand,
> What you're doing is a slightly odd way to use Flume. With the exec 
> source, Flume will execute that command, and consume the output as 
> events. Often the exec source is used to tail -F a file, which allows 
> you to pipe more data to the file and ingest additional events. By 
> using cat, Flume will cat the file, but then the source will become 
> useless, because the command will have finished, and there's no way 
> that I'm aware of to get an agent to start a new command. By using 
> tail -F, the command persists, and if you do `ps aux | grep flume`, 
> you would see a running tail -F command.
> As for figuring out when the transfer is complete, I don't think 
> there's a really good way other than checking the file itself, or 
> looking to see if the cat command is still running.
> Does that help?
> Thanks,
> Natty
> On Thu, Jul 24, 2014 at 2:00 AM, Anandkumar Lakshmanan 
> <anand@orzota.com <mailto:anand@orzota.com>> wrote:
>     Hi,
>     I am new to flume.
>     I am doing cat a file using exec source into hdfs.
>     While running it manually, I am able to see the file transferred
>     completely. But still flume in is running state.
>     How do I find when the complete transfer would be done.
>     Example:
>     My flume.conf
>     myAgent.sources.mySource.type = exec
>     myAgent.sources.mySource.command = cat /home/haas/file2.txt
>     And checking the transfer is complete or not, only by typing the
>     following command manually by comparing the file size.
>     hadoop fs -ls /user/flumedata/
>     Is there a way to know when the transfer is get completed?
>     Thanks.
>     Anand

View raw message