flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sun, Lining" <Lining_...@intuit.com>
Subject RE: Flume Configuration & topology approach
Date Tue, 08 Apr 2014 17:49:27 GMT
We implemented memory mapping channel that has memory channel performance and also file channel
reliability. We plan to contribute it to open source in near future.


From: Jeff Lord [mailto:jlord@cloudera.com]
Sent: Monday, April 07, 2014 8:48 AM
To: user@flume.apache.org
Subject: Re: Flume Configuration & topology approach

No. If you need to guarantee delivery of events please use a file channel.

On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon <cshannon108@gmail.com<mailto:cshannon108@gmail.com>>

On Apr 7, 2014 9:35 AM, "Jeff Lord" <jlord@cloudera.com<mailto:jlord@cloudera.com>>
> On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal <durgapalmohit@gmail.com<mailto:durgapalmohit@gmail.com>>
>> Hi Jeff,
>> Yes, I am using the memory channel, and that's because I want it to be more reliable
and not miss any events/messages.
>> As I've read in flume documentation that the memory channel is fast but there could
be a chance of missing events if the in-memory buffer fills up.
> Memory channel is not reliable, meaning if the flume agent goes down or is restarted
while there are events in the channel than this data will be lost.
> For reliability please use the file channel.


I am using an upstream agent with a spooling directory source and a memory channel, and the
downstream agent uses a memory channel and an HDFS sink. If my downstream agent goes down
for any reason, are the entries lost in the downstream agent's memory channel still preserved
in the memory channel / file directory of the upstream agent?

All the best, Chris

View raw message