flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: Flume-ng - Distributed
Date Wed, 10 Oct 2012 18:30:15 GMT
Most commonly this is done using something like puppet or chef. Like Roy said, you can use
the same config file, but different agent names so you an deploy the same file, yet have different
configurations for the agents on different machines.


Hari 

-- 
Hari Shreedharan


On Wednesday, October 10, 2012 at 11:19 AM, Camp, Roy wrote:

> You have to manually start each node with its specific configuration.  However, you can
put the configuration for all your different setups into one config file, but you will still
need to place a copy of it on every machine.  Simply define which agent config to use with
the --name param when starting.  
>  
> Thanks,
>  
> Roy
>  
>  
> From: Juan Gentile [mailto:juan.gentile@globant.com] 
> Sent: Wednesday, October 10, 2012 9:54 AM
> To: user@flume.apache.org (mailto:user@flume.apache.org)
> Subject: Re: Flume-ng - Distributed 
>  
> Thank you both very much, I've been reading the documentation you sent me and this brings
another question, is there a way to submit my flume configuration to a cluster or I have to
manually start up each node with the its specific configuration?
>  
> 
> Thank you!
> On Wed, Oct 10, 2012 at 1:51 AM, Mike Percy <mpercy@apache.org (mailto:mpercy@apache.org)>
wrote:
> +1 on what Roy said, with a minor terminology quibble: in Flume NG the Avro collector
component is called the Avro Source.
>  
> 
> Also, here are links to the docs with working image links and table of contents:
> 
>  
> 
> http://flume.apache.org/FlumeUserGuide.html
> 
> http://flume.apache.org/FlumeDeveloperGuide.html
>  
> 
> Regards,
> 
> Mike
> 
>  
> On Tue, Oct 9, 2012 at 5:52 PM, Camp, Roy <rcamp@ebay.com (mailto:rcamp@ebay.com)>
wrote:
> You would run a flume-ng instance on each node with an avro-sink.  Then on your collector
machine you will run another flume-ng instance with an avro-collector.
>  
> If you run more than one collector you can setup sink groups and define that it does
failover or load balancing.
>  
> The concept of a flume master from flume 0.9.x does not exist on flume-ng.  I personally
use the node and collector configs in the same config file under a different agent name, and
then keep them synced on all machines.  
>  
> These two docs are pretty helpful:
> https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst
> https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeDeveloperGuide.rst

>  
> Thanks,
>  
> Roy
>  
>  
>  
>  
> From: Juan Gentile [mailto:juan.gentile@globant.com] 
> Sent: Tuesday, October 09, 2012 11:04 AM
> To: user@flume.apache.org (mailto:user@flume.apache.org)
> Subject: Flume-ng - Distributed 
>  
> Hi,
>  
> 
> I'm new to Flume-ng, I'd like to ask you if you can tell me how I can accomplish to have
an agent distributed in a cluster. I've have developed my own source and sink version that
reads from a queue and the sink stores the messages read to hdfs. If I want to have this running
in multiple instances, do I have to submit it on each node?
> 
>  
> 
> This is my conf file:
> 
> agent1.channels.channel1.type = memory
> 
> agent1.channels.channel1.capacity = 1000
> 
> agent1.channels.channel1.transactionCapacity = 1000
> 
>  
> 
> agent1.sources.source1.channels = channel1
> 
> agent1.sources.source1.type = MySource
> 
>  
> 
> agent1.sinks.sink1.channel = channel1
> 
> agent1.sinks.sink1.type = MySink
> 
>  
> 
> agent1.channels = channel1
> 
> agent1.sources = source1
> 
> agent1.sinks = sink1
> 
> 
>  
> 
>  
> 
> I see that there is the concept of 'master' a 'node' in the previous version of flume,
do I have something similar here?
> 
>  
> 
> Thanks,
> 
> Juan
> 
> 
> 
> 
> 
> 
>  
> 
> 
> 
> 
> 
>  
> 
> 
> 
> 



Mime
View raw message