flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: AvroSink and LoadBalancingRpcClient
Date Thu, 10 Jan 2013 07:30:21 GMT
The LoadBalancing capability similar to the LoadBalancingRpcClient can be configured for multiple
Avro Sinks using a LoadBalancingSinkProcessor, if you are looking for that functionality.



Hari 

-- 
Hari Shreedharan


On Wednesday, January 9, 2013 at 11:05 PM, Connor Woodson wrote:

> Short answer: there is no way in the current AvroSink to configure the RpcClient, limiting
you to just a single host connection (I'm not sure how well it recovers if that host goes
down).
> 
> The AvroSink is incredibly simplified from what the RPCClient can do and exposes none
of the background functionality. Right now, the only way around that is to create a custom
sink based off of the AvroSink source code and instead of setting the RPCClient up the way
it currently is, you pass into the RPCClient.getInstance() a set of user supplied properties.
To implement this in an unsafe way (not checking any of the user's values) would only take
a couple lines of code I believe. It is a work around, but it will enable all of the various
RPCClient capabilities such as failover or loadbalancing mode and allow it to connect to multiple
hosts.
> 
> This is something that (I think) there is a JIRA filed for; but if not, it would be very
helpful for this to be implemented into the actual AvroSink (and something that should be
linked to that is RPCClient.getInstance accepting a Context object, simply for ease of use).

> 
> - Connor
> 
> 
> On Wed, Jan 9, 2013 at 10:55 PM, Denny Ye <dennyy99@gmail.com (mailto:dennyy99@gmail.com)>
wrote:
> > hi all, 
> >     I didn't find the relationship between AvroSink and other types of RpcClient,
including LoadBalancingRpcClient. In my opinion, user can set the specified RpcClient type
from AvroSink with several strategies and host selectors. Also, I cannot get information from
source code and user guide. Did I miss something about this? 
> >      Wish someone can support, thanks!
> > 
> > -Regards
> > Denny Ye
> > 
> > 
> 
> 
> 


Mime
View raw message