flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From terrey shih <terreys...@gmail.com>
Subject Re: flume avro event overflow ?
Date Mon, 20 Oct 2014 18:39:19 GMT

Yes, I have looked into that but that does not really work either because
the extra IO might slow down the whole agent which also has another channel
that has no problem sending/consuming the events.

It seems like a quite common issue.  What if the down stream agent is not
turned up physically (i.e. during maintenance period) ?  The upstream agent
would not know about it and would keeping sending the events until the
buffer goes over the limit.


On Mon, Oct 20, 2014 at 9:07 AM, Jeff Lord <jlord@cloudera.com> wrote:

> I know this is not exactly what you are asking for but have you had a look
> at the spillable memory channel.
> https://flume.apache.org/FlumeUserGuide.html#spillable-memory-channel
> On Sun, Oct 19, 2014 at 1:38 AM, terreyshih <terreyshih@gmail.com> wrote:
>> In other words, I would like to explicitly drop the events if the
>> channel/event queue if full.
>> thx
>> > On Oct 18, 2014, at 11:42 PM, terreyshih <terreyshih@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > If I have a upstream agent duplicating the event to two channels and
>> subsequently to two downstream sinks.  If one of the downstream sinks for
>> some reason can not process the events fast enough and the event overflows
>> back over the buffer limit of the upstream agent, would it cause any issues
>> for the other channel since the events are duplicated ?
>> >
>> > In general, what happens if the downstream agent is not able to handle
>> the speed of the upstream agent sending the events ?  I have seen
>> exceptions that says unable to send events or memory channel commit issues.
>> >
>> > However, the flume agent does not seem to handle that gracefully and
>> the whole agent just stops processing the events for all channels, even for
>> the channels that can keep up with events.
>> >
>> > What if I separate the two channels into two different agents, would
>> that solve the problem.
>> >
>> > Can upstream flume agent not just drop the events when one of the its
>> sinks or downstream agents cannot catch up  ?
>> >
>> > thx

View raw message