flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jl...@cloudera.com>
Subject Re: Flume Configuration & topology approach
Date Mon, 07 Apr 2014 15:48:11 GMT
No. If you need to guarantee delivery of events please use a file channel.
https://blogs.apache.org/flume/entry/apache_flume_filechannel


On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon
<cshannon108@gmail.com>wrote:

>
> On Apr 7, 2014 9:35 AM, "Jeff Lord" <jlord@cloudera.com> wrote:
> >
> >
> >
> >
> > On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal <durgapalmohit@gmail.com>
> wrote:
> >>
> >> Hi Jeff,
> >>
> >> Yes, I am using the memory channel, and that's because I want it to be
> more reliable and not miss any events/messages.
> >> As I've read in flume documentation that the memory channel is fast but
> there could be a chance of missing events if the in-memory buffer fills up.
> >
> >
> > Memory channel is not reliable, meaning if the flume agent goes down or
> is restarted while there are events in the channel than this data will be
> lost.
> > For reliability please use the file channel.
> >
>
> Jeff,
>
> I am using an upstream agent with a spooling directory source and a memory
> channel, and the downstream agent uses a memory channel and an HDFS sink.
> If my downstream agent goes down for any reason, are the entries lost in
> the downstream agent's memory channel still preserved in the memory channel
> / file directory of the upstream agent?
>
> All the best, Chris
>

Mime
View raw message