flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From terreyshih <terreys...@gmail.com>
Subject Re: flume avro event overflow ?
Date Sun, 19 Oct 2014 08:38:29 GMT
In other words, I would like to explicitly drop the events if the channel/event queue if full.


> On Oct 18, 2014, at 11:42 PM, terreyshih <terreyshih@gmail.com> wrote:
> Hi,
> If I have a upstream agent duplicating the event to two channels and subsequently to
two downstream sinks.  If one of the downstream sinks for some reason can not process the
events fast enough and the event overflows back over the buffer limit of the upstream agent,
would it cause any issues for the other channel since the events are duplicated ?
> In general, what happens if the downstream agent is not able to handle the speed of the
upstream agent sending the events ?  I have seen exceptions that says unable to send events
or memory channel commit issues.  
> However, the flume agent does not seem to handle that gracefully and the whole agent
just stops processing the events for all channels, even for the channels that can keep up
with events.
> What if I separate the two channels into two different agents, would that solve the problem.
> Can upstream flume agent not just drop the events when one of the its sinks or downstream
agents cannot catch up  ?
> thx

View raw message