flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sharninder <sharnin...@gmail.com>
Subject Re: Flume NG and S3
Date Tue, 01 Jul 2014 04:26:42 GMT
No reason to not use flume except for the fact that S3, since its over the
wire, will be a lot slower than a local hdfs cluster in which case you need
a big enough channel to hold events not yet processed out of the sink. If
you have a fast enough pipe, you can very well use flume for this sort of

The reason the author might have moved to kafka, and I'm just speculating
here, is that kafka provides him better buffering support for exactly the
case I've written above.


On Mon, Jun 30, 2014 at 7:57 PM, Máté Gulyás <gulyasm@dmlab.hu> wrote:

> Hi!
> I would like to use flume to aggregate and send logs to an S3 bucket.
> I did some research, but the last article I found on the topic was
> more then a year old and the author abandoned Flume for Kafka. My
> other concern is that most of the articles were written for Flume OG,
> not NG.
> Is there any reason why I should not use flume to sink messages to S3?
> Thanks in advance.
> Mate Gulyas
> Lead Developer at Dmlab

View raw message