flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Horrocks <ch...@hor.rocks>
Subject Re: how to make KafkaSource consume the existing messages
Date Thu, 13 Oct 2016 10:56:17 GMT

Which version of Kafka are you using?

Off the top of my head it should be: tier2.sources.source1.kafka.auto.offset.reset = earliest

Of course changing the group ID or if it's an older version of Kafka removing the corresponding
offset znode from zookeeper ought to do the trick

-- Chris Horrocks

On Thu, Oct 13, 2016 at 8:55 am, Ping PW Wang <'wpwang@cn.ibm.com'> wrote:

I used KafkaSource to consume the messages from Kafka. I found only new messages were received
while the old existing message not. I tried to use a new consumer group and update the parameter
"auto.offset.reset = latest" to "earliest", but this does not work.

tier2.sources.source1.kafka.consumer.group.id = test-consumer-group-new
tier2.sources.source1.kafka.consumer.auto.offset.reset = earliest

Anyone knows how to make KafkaSource consume the existing messages?
Thanks a lot for any advice!
View raw message