flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juhani Connolly <juhani_conno...@cyberagent.co.jp>
Subject Re: Source and Sink on different machine
Date Fri, 28 Dec 2012 08:09:49 GMT
You connect up the various machines using avro sink/source.

You set up 2 & 3 with avro source, a channel and hdfs sink.

Then you set up machine one with whatever source, channel and two avro 
sinks, and a sink 
processor(http://flume.apache.org/FlumeUserGuide.html#flume-sink-processors) 
of your personal preference. If you use failover when the main sink 
fails, events will go through the backup instead.

On 12/28/2012 04:53 PM, Abhijeet Pathak wrote:
>
> I want to setup a failover configuration.
>
> On Machine1 I've a source.
>
> On Machine2 & Machine3, I have (hbase) sinks.
>
> I want to configure them in failover mode. That is, by default events 
> should go to sink on Machine2's sink. But if Machine2 fails then they 
> should automatically go to Machine3's sink.
>
> Is that possible in Flume-NG? How?
>
> Thanks,
>
> Abhijeet
>
> *From:*Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* Friday, December 28, 2012 1:18 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Source and Sink on different machine
>
> if I understood it correctly, you want to put your log data into hbase 
> which is hosted remotely?
>
> yes thats easily doable.
>
> Flume-NG does everything alone instead of older master-slave 
> architecture. It has currently source channel and sink. you can 
> collect logs on a machine via an agent and using same agent you can 
> sink them into hbase  on a remote host
>
> On Fri, Dec 28, 2012 at 12:06 PM, Abhijeet Pathak 
> <Abhijeet.Pathak@kpitcummins.com 
> <mailto:Abhijeet.Pathak@kpitcummins.com>> wrote:
>
> Hi,
>
> It was possible to have source and sinks on different machines using 
> Flume 0.9x.
>
> Is it possible to have source and sinks on different machines in 
> Flume-NG (1.3.0)?
>
> I want to use tail source on one machine and hbase sink on another 
> machine.
>
> How should I go about it?
>
> Thanks,
>
> Abhijeet
>
>
> This message contains information that may be privileged or 
> confidential and is the property of the KPIT Cummins Infosystems Ltd. 
> It is intended only for the person to whom it is addressed. If you are 
> not the intended recipient, you are not authorized to read, print, 
> retain copy, disseminate, distribute, or use this message or any part 
> thereof. If you receive this message in error, please notify the 
> sender immediately and delete all copies of this message. KPIT Cummins 
> Infosystems Ltd. does not accept any liability for virus infected mails.
>
>
>
> -- 
> Nitin Pawar
>
>
> This message contains information that may be privileged or 
> confidential and is the property of the KPIT Cummins Infosystems Ltd. 
> It is intended only for the person to whom it is addressed. If you are 
> not the intended recipient, you are not authorized to read, print, 
> retain copy, disseminate, distribute, or use this message or any part 
> thereof. If you receive this message in error, please notify the 
> sender immediately and delete all copies of this message. KPIT Cummins 
> Infosystems Ltd. does not accept any liability for virus infected mails. 


Mime
View raw message