flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Shannon <cshannon...@gmail.com>
Subject Re: Flume Configuration & topology approach
Date Mon, 07 Apr 2014 17:48:48 GMT
Got it. Thanx.
On Apr 7, 2014 12:39 PM, "Jeff Lord" <jlord@cloudera.com> wrote:

> No not at all. Flume's transactional model guarantees delivery between
> hops.
> https://blogs.apache.org/flume/entry/flume_ng_architecture
>
>
> On Mon, Apr 7, 2014 at 10:16 AM, Christopher Shannon <
> cshannon108@gmail.com> wrote:
>
>> So, this basically means that Flume's transactional model is also
>> unreliable. That would have to mean that the downstream agent is sending an
>> ack to the upstream agent before it actually persists the event.
>>
>> On Apr 7, 2014 10:48 AM, "Jeff Lord" <jlord@cloudera.com> wrote:
>> >
>> > No. If you need to guarantee delivery of events please use a file
>> channel.
>> > https://blogs.apache.org/flume/entry/apache_flume_filechannel
>> >
>> >
>> > On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon <
>> cshannon108@gmail.com> wrote:
>> >>
>> >>
>> >> On Apr 7, 2014 9:35 AM, "Jeff Lord" <jlord@cloudera.com> wrote:
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal <
>> durgapalmohit@gmail.com> wrote:
>> >> >>
>> >> >> Hi Jeff,
>> >> >>
>> >> >> Yes, I am using the memory channel, and that's because I want it
to
>> be more reliable and not miss any events/messages.
>> >> >> As I've read in flume documentation that the memory channel is
fast
>> but there could be a chance of missing events if the in-memory buffer fills
>> up.
>> >> >
>> >> >
>> >> > Memory channel is not reliable, meaning if the flume agent goes down
>> or is restarted while there are events in the channel than this data will
>> be lost.
>> >> > For reliability please use the file channel.
>> >> >
>> >>
>> >> Jeff,
>> >>
>> >> I am using an upstream agent with a spooling directory source and a
>> memory channel, and the downstream agent uses a memory channel and an HDFS
>> sink. If my downstream agent goes down for any reason, are the entries lost
>> in the downstream agent's memory channel still preserved in the memory
>> channel / file directory of the upstream agent?
>> >>
>> >> All the best, Chris
>> >
>> >
>>  No. If you need to guarantee delivery of events please use a file
>> channel.
>> https://blogs.apache.org/flume/entry/apache_flume_filechannel
>>
>>
>> On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon <
>> cshannon108@gmail.com> wrote:
>>
>>>
>>> On Apr 7, 2014 9:35 AM, "Jeff Lord" <jlord@cloudera.com> wrote:
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal <
>>> durgapalmohit@gmail.com> wrote:
>>> >>
>>> >> Hi Jeff,
>>> >>
>>> >> Yes, I am using the memory channel, and that's because I want it to
>>> be more reliable and not miss any events/messages.
>>> >> As I've read in flume documentation that the memory channel is fast
>>> but there could be a chance of missing events if the in-memory buffer fills
>>> up.
>>> >
>>> >
>>> > Memory channel is not reliable, meaning if the flume agent goes down
>>> or is restarted while there are events in the channel than this data will
>>> be lost.
>>> > For reliability please use the file channel.
>>> >
>>>
>>> Jeff,
>>>
>>> I am using an upstream agent with a spooling directory source and a
>>> memory channel, and the downstream agent uses a memory channel and an HDFS
>>> sink. If my downstream agent goes down for any reason, are the entries lost
>>> in the downstream agent's memory channel still preserved in the memory
>>> channel / file directory of the upstream agent?
>>>
>>> All the best, Chris
>>>
>>
>>
>

Mime
View raw message