flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hemanth Abbina <Heman...@eiqnetworks.com>
Subject Re: Possibility of persisting the connection
Date Tue, 17 Nov 2015 17:58:54 GMT
Hi Gonzalo,

Thanks for your response.

No, the Kafka sink connection is not the same all times.I have observed the connections closing
and reconnecting.

Sent from my HTC

----- Reply message -----
From: "Gonzalo Herreros" <gherreros@gmail.com>
To: "user" <user@flume.apache.org>
Subject: Possibility of persisting the connection
Date: Tue, Nov 17, 2015 11:08 PM

For the sink, I would be surprised if the connection to kafka is not the same all the time.
For the http source you could create a custom source where you keep a long lived http connection
and have some way of detecting where a batch of events is sent (e.g. a new line character).


On 17 November 2015 at 17:16, Hemanth Abbina <HemanthA@eiqnetworks.com<mailto:HemanthA@eiqnetworks.com>>

Though it's against the basic design principle of Flume, I have one question.

Is this possible to persist the connection between source & sink and re-use ?

We are using HTTP source, File channel & Kafka sink and with that configuration, not getting
the expected throughput because of the reconnections of the source & sink for every event.

So, would it be possible to re-use the same HTTP and Kafka connections for multiple transactions
? (even with a custom source & sink)


View raw message