flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: Flume hops behind HAProxy
Date Thu, 15 Nov 2012 17:35:00 GMT
That will probably work. From my earlier mail:

"I assume it would be connection based round robin?  Might work just
fine, but probably best to the use built-in support."

On Thu, Nov 15, 2012 at 11:27 AM, Rahul Ravindran <rahulrv@yahoo.com> wrote:
> HAProxy has a TCP mode where it round robins TCP connections. Does it need
> to understand something specific about the wire protocol used by Flume?
>
> ________________________________
> From: Brock Noland <brock@cloudera.com>
> To: user@flume.apache.org; Rahul Ravindran <rahulrv@yahoo.com>
> Sent: Wednesday, November 14, 2012 6:20 PM
>
> Subject: Re: Flume hops behind HAProxy
>
>> It would be round robin but not sticky sessions( so each request could
>> Goto any random flume hop)
>
> Does HAProxy understand the protocol?  To round robin requests like
> this it needs to understand the communication protocol, which I
> suppose would work if you were using the HTTPSource.
>
> On Wed, Nov 14, 2012 at 4:46 PM, Rahul Ravindran <rahulrv@yahoo.com> wrote:
>> HAProxy is widely deployed already in our deployment and Ops is familiar
>> with dealing with it for hosts which go down etc.
>>
>> ________________________________
>> From: "Camp, Roy" <rcamp@ebay.com>
>> To: "user@flume.apache.org" <user@flume.apache.org>
>> Sent: Wednesday, November 14, 2012 2:15 PM
>> Subject: RE: Flume hops behind HAProxy
>>
>> Out of curiosity, what is the use case vs using the built in load
>> balancing?
>>
>>
>>
>> -----Original Message-----
>> From: Rahul Ravindran [mailto:rahulrv@yahoo.com]
>> Sent: Wednesday, November 14, 2012 1:49 PM
>> To: user@flume.apache.org
>> Cc: user@flume.apache.org
>> Subject: Re: Flume hops behind HAProxy
>>
>> It would be round robin but not sticky sessions( so each request could
>> Goto
>> any random flume hop)
>>
>> Sent from my phone.Excuse the terseness.
>>
>> On Nov 14, 2012, at 1:33 PM, Brock Noland <brock@cloudera.com> wrote:
>>
>>> I assume it would be connection based round robin?  Might work just
>>> fine, but probably best to the use built-in support.
>>>
>>> On Wed, Nov 14, 2012 at 2:46 PM, Rahul Ravindran <rahulrv@yahoo.com>
>>> wrote:
>>>> Resending given I sent it during off-hours.
>>>>
>>>> ________________________________
>>>> From: Rahul Ravindran <rahulrv@yahoo.com>
>>>> To: "user@flume.apache.org" <user@flume.apache.org>
>>>> Sent: Tuesday, November 13, 2012 5:52 PM
>>>> Subject: Flume hops behind HAProxy
>>>>
>>>> Hi,
>>>>  Before I try it, I wanted to check if there were any known issues
>>>> with this. We will have multiple flume agents sending an Avro stream
>>>> each to a smaller set of intermediate flume hops. Are there any
>>>> issues/concerns around having the flume agents send their streams to
>>>> an HAProxy which will round robin between the different flume hops.
>>>> Any issue around the transaction mechanism with this setup?
>>>>
>>>> I know that there is a selector mechanism in Flume to do this, but
>>>> our operations extensively use HAProxy, and are most familiar with it.
>>>>
>>>> Thanks,
>>>> ~Rahul.
>>>
>>>
>>>
>>> --
>>> Apache MRUnit - Unit testing MapReduce -
>>> http://incubator.apache.org/mrunit/
>>
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
>
>



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Mime
View raw message