flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arvind Prabhakar <arv...@apache.org>
Subject Re: flume avro event overflow ?
Date Tue, 21 Oct 2014 14:16:11 GMT
Hi Terry,

In other words, I would like to explicitly drop the events if the
> channel/event queue if full.

You can do this by setting the second channel as an optional channel for
the multiplexing selector. If the optional channel is full and does not
accept events, they will be dropped by the agent without applying any back
pressure to the upstream flow.

Arvind Prabhakar

On Mon, Oct 20, 2014 at 11:39 AM, terrey shih <terreyshih@gmail.com> wrote:

> Hi
> Yes, I have looked into that but that does not really work either because
> the extra IO might slow down the whole agent which also has another channel
> that has no problem sending/consuming the events.
> It seems like a quite common issue.  What if the down stream agent is not
> turned up physically (i.e. during maintenance period) ?  The upstream agent
> would not know about it and would keeping sending the events until the
> buffer goes over the limit.
> thx
> On Mon, Oct 20, 2014 at 9:07 AM, Jeff Lord <jlord@cloudera.com> wrote:
>> I know this is not exactly what you are asking for but have you had a
>> look at the spillable memory channel.
>> https://flume.apache.org/FlumeUserGuide.html#spillable-memory-channel
>> On Sun, Oct 19, 2014 at 1:38 AM, terreyshih <terreyshih@gmail.com> wrote:
>>> In other words, I would like to explicitly drop the events if the
>>> channel/event queue if full.
>>> thx
>>> > On Oct 18, 2014, at 11:42 PM, terreyshih <terreyshih@gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > If I have a upstream agent duplicating the event to two channels and
>>> subsequently to two downstream sinks.  If one of the downstream sinks for
>>> some reason can not process the events fast enough and the event overflows
>>> back over the buffer limit of the upstream agent, would it cause any issues
>>> for the other channel since the events are duplicated ?
>>> >
>>> > In general, what happens if the downstream agent is not able to handle
>>> the speed of the upstream agent sending the events ?  I have seen
>>> exceptions that says unable to send events or memory channel commit issues.
>>> >
>>> > However, the flume agent does not seem to handle that gracefully and
>>> the whole agent just stops processing the events for all channels, even for
>>> the channels that can keep up with events.
>>> >
>>> > What if I separate the two channels into two different agents, would
>>> that solve the problem.
>>> >
>>> > Can upstream flume agent not just drop the events when one of the its
>>> sinks or downstream agents cannot catch up  ?
>>> >
>>> > thx

View raw message