flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun MA <mj.saber1...@gmail.com>
Subject Re: Flafka: Kafka channel unable to deliver event
Date Mon, 06 Jul 2015 00:37:46 GMT
Hi Rufus,

Thank you so much for your help, I do bypass the issue. 

Another question I have is that can I have two Flafka agents consume from one topic (agents
run on same machine or different machine)? Will the two agents have exact same two copies
or each agent will consume one part of the topic? The purpose I want to run two agents on
same topic is that I want to have a high availability. What I want is that two agents will
consume different portion of the topic but if one agent down, the other one will consume everything
from the topic. 
From what I tested, I bring up two agents on same machine with different agent name (everything
else are the same), and only the first one can consume messages from topic. I’m wondering
if it is the right behavior. If so, is there anyway I can solve this single point of failure?

Bests,
Jun

> On Jul 4, 2015, at 10:22 AM, Johny Rufus <jrufus@cloudera.com> wrote:
> 
> Been looking into this for some time and found a couple of issues. Raised Jiras for both,
there are workarounds to get past these errors.
> 
> https://issues.apache.org/jira/browse/FLUME-2734 <https://issues.apache.org/jira/browse/FLUME-2734>
> https://issues.apache.org/jira/browse/FLUME-2735 <https://issues.apache.org/jira/browse/FLUME-2735>
> 
> For timeout to take effect, you need to specify as - agent1.channels.channel1.timeout
= 1000000 (This is a temporary work around, kafka.consumer.timeout.ms <http://kafka.consumer.timeout.ms/>
should work as per the guide when FLUME-2734 is resolved)
> 
> To get past the IllegalStateException, I had to download the zookeeper jar from http://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper/3.3.6
<http://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper/3.3.6>
> and put it in the lib directory. This is also a workaround to get past the issue, until
we figure out the root cause.
> 
> 
> Thanks,
> Rufus
> 
> On Fri, Jul 3, 2015 at 10:09 PM, Jun Ma <mj.saber1990@gmail.com <mailto:mj.saber1990@gmail.com>>
wrote:
> Oops, thanks for figuring that out.
> Any idea why unable to deliver event?
> 
> On Fri, Jul 3, 2015 at 8:00 PM, Johny Rufus <jrufus@cloudera.com <mailto:jrufus@cloudera.com>>
wrote:
> There is a typo in your property "cnannel1", hence the property is not set 
> 
> a1.channels.cnannel1.kafka.consumer.timeout.ms <http://a1.channels.cnannel1.kafka.consumer.timeout.ms/>
= 1000000
> 
> Thanks,
> Rufus
> 
> On Fri, Jul 3, 2015 at 4:49 PM, Jun Ma <mj.saber1990@gmail.com <mailto:mj.saber1990@gmail.com>>
wrote:
> Thanks for your reply. But from what I read, the magic things Flafka does is that you
don't need to have a source, you can directly move things from channel to sink.
> http://blog.cloudera.com/blog/2014/11/flafka-apache-flume-meets-apache-kafka-for-event-processing/
<http://blog.cloudera.com/blog/2014/11/flafka-apache-flume-meets-apache-kafka-for-event-processing/>
(see Flume's kafka channel)
> 
> 
> On Fri, Jul 3, 2015 at 1:26 PM, Foo Lim <foo.lim@vungle.com <mailto:foo.lim@vungle.com>>
wrote:
> There's an error that you didn't specify a source:
> 
> Agent configuration for 'a1' has no sources.
> 
> 
> 
> 


Mime
View raw message