flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jl...@cloudera.com>
Subject Re: flume avro event overflow ?
Date Mon, 20 Oct 2014 16:07:27 GMT
I know this is not exactly what you are asking for but have you had a look
at the spillable memory channel.
https://flume.apache.org/FlumeUserGuide.html#spillable-memory-channel

On Sun, Oct 19, 2014 at 1:38 AM, terreyshih <terreyshih@gmail.com> wrote:

> In other words, I would like to explicitly drop the events if the
> channel/event queue if full.
>
> thx
>
> > On Oct 18, 2014, at 11:42 PM, terreyshih <terreyshih@gmail.com> wrote:
> >
> > Hi,
> >
> > If I have a upstream agent duplicating the event to two channels and
> subsequently to two downstream sinks.  If one of the downstream sinks for
> some reason can not process the events fast enough and the event overflows
> back over the buffer limit of the upstream agent, would it cause any issues
> for the other channel since the events are duplicated ?
> >
> > In general, what happens if the downstream agent is not able to handle
> the speed of the upstream agent sending the events ?  I have seen
> exceptions that says unable to send events or memory channel commit issues.
> >
> > However, the flume agent does not seem to handle that gracefully and the
> whole agent just stops processing the events for all channels, even for the
> channels that can keep up with events.
> >
> > What if I separate the two channels into two different agents, would
> that solve the problem.
> >
> > Can upstream flume agent not just drop the events when one of the its
> sinks or downstream agents cannot catch up  ?
> >
> > thx
>
>

Mime
View raw message