flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tao Li <litao.bupt...@gmail.com>
Subject Re: [KafkaChannel] About kafka producer type "sync" or "aync"
Date Mon, 13 Apr 2015 17:48:22 GMT
OK, I got it. Thanks a lot.

2015-04-14 1:26 GMT+08:00 Gwen Shapira <gshapira@cloudera.com>:

> We implemented the KafkaChannel with the "sync" producer, because we
> want to guarantee the transaction success and handle errors
> appropriately.
> We think it is worth the performance loss. Even with the sync
> producer, we found that the Kafka Channel can handle 20K events per
> second on my (pretty simple) setup. By tuning the batch size you can
> have some control over the latency and throughput.
>
> I also started testing the new Kafka producer (0.8.2.0 and up) - the
> way we use the producer in the channel (batches), there was almost no
> performance improvement.
>
> Gwen
>
> On Mon, Apr 13, 2015 at 9:41 AM, Tao Li <litao.buptsse@gmail.com> wrote:
> > Hi all:
> >
> > JIRA: https://issues.apache.org/jira/browse/FLUME-2500
> >
> > I have a question about this issue.
> >
> > Flume support transaction between source and channel. So for kafka
> channel:
> > 1. If we use "sync" kafka producer, it surely can guarantee the
> transaction,
> > but sync send has low performance.
> > 2. If we use "async" kafka producer, the performance will be better, but
> it
> > can't guarantee the transaction.
> >
> > How do you think on it?
>

Mime
View raw message