flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 周梦想 <abloz...@gmail.com>
Subject Re: how to know which line the agent have sent ?
Date Thu, 07 Feb 2013 03:24:09 GMT
So all users of flume don't care the agent break down  and miss send or
duplicate the content of logs? They have to write their own sources and
sinks?
They don't care the correct of logs? How they do if the flume agent exited?
I'm not yet  understand.

Andy

2013/2/7 周梦想 <ablozhou@gmail.com>

> I see,there is no easy way or configure way to know the detail of what has
> sent and what haven't.
> I have to write my own source or sink code to do this.
> Thank you,Alex and all friends.
>
> Andy
>
>
> 2013/2/6 Alexander Alten-Lorenz <wget.null@gmail.com>
>
>> You haven't a control in such situations, since tailDir uses tail and
>> holds the marker in memory.
>>
>> We had few days ago a thread about:
>>
>> http://search-hadoop.com/m/JV0lh2RDXLX/flume+tail+source+problem+and+performance&subj=flume+tail+source+problem+and+performance
>>
>> - Alex
>>
>> On Feb 6, 2013, at 3:45 AM, 周梦想 <ablozhou@gmail.com> wrote:
>>
>> > Hello,
>> >
>> > I'm using tailDirs('mydir') source of the agent to gather logs to hadoop
>> > hdfs. I notice some documents advise that if the agent collapsed,  I
>> have
>> > to remove files in 'mydir' and clear flume.agent.logdir. Thus I will
>> lose
>> > some data or have duplicate data. And I don't know which line the agent
>> > have sent to.
>> >
>> > I'm worrying about the agent failure and resend or miss-send the
>> content to
>> > collector. I want to know how to check which line of log file the agent
>> > have sent if the agent exit suddenly. The files in flute log dir, such
>> as
>> > sending,sent can't be read.
>> >
>> > Please give some advise to process such situation.
>> > Thanks.
>> >
>> > Andy Zhou
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>>
>

Mime
View raw message