flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Malouf <malouf.g...@gmail.com>
Subject Re: FileChannel Replays consistently take a long time
Date Wed, 20 Aug 2014 04:12:04 GMT
Our capacity setting is:

agent-1.channels.trdbuy-bid-req-ch1.capacity = 100000000


Our current channel size can not be accessed because it still is in this
odd 'replay' mode.  There's not logs, but the cpu is cranking on the flume
node and the avro source ports have not yet opened.  The pattern we see is
that after anywhere from 15-30 minutes, the ports magically open and we can
continue.


This is because we are logging around 10k messages/second and did not want
to lose any data during brief interruptions.


On Wed, Aug 20, 2014 at 12:02 AM, Hari Shreedharan <
hshreedharan@cloudera.com> wrote:

> How large is your channel (and how long does it take to replay?)
>
> Gary Malouf wrote:
>
>
> For the record, we are using Flume 1.4.0 packaged with CDH5.0.2
>
>
> On Tue, Aug 19, 2014 at 11:55 PM, Gary Malouf <malouf.gary@gmail.com
> <mailto:malouf.gary@gmail.com>> wrote:
>
>     We are repeatedly running into cases where the replays of from a
>     file channel going to HDFS take an eternity.
>
>     I've read this thread
>     <
> http://mail-archives.apache.org/mod_mbox/flume-dev/201306.mbox/%3CCAHbPYVbmeD6pKZkDADmyaW_gC_p7cQDefPsycWKnkY72TFiAuQ@mail.gmail.com%3E
> >,
>
>     but I just am not convinced that our checkpoints are constantly
>     being corrupted.
>
>     We are seeing messages such as:
>
>     20 Aug 2014 03:52:26,849 INFO  [lifecycleSupervisor-1-2]
>     (org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>:57)
>
>       - Reading checkpoint metadata from
>     /opt/flume/brq/ch1/checkpoint/checkpoint.meta
>
>
>     How can it be that this takes so long?
>
>

Mime
View raw message