From user-return-2470-apmail-flume-user-archive=flume.apache.org@flume.apache.org Wed Oct 10 19:56:31 2012 Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D78309F1E for ; Wed, 10 Oct 2012 19:56:31 +0000 (UTC) Received: (qmail 76726 invoked by uid 500); 10 Oct 2012 19:56:31 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 76685 invoked by uid 500); 10 Oct 2012 19:56:31 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 76632 invoked by uid 99); 10 Oct 2012 19:56:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Oct 2012 19:56:31 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of iainwrig@gmail.com designates 209.85.219.51 as permitted sender) Received: from [209.85.219.51] (HELO mail-oa0-f51.google.com) (209.85.219.51) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Oct 2012 19:56:25 +0000 Received: by mail-oa0-f51.google.com with SMTP id n12so920429oag.38 for ; Wed, 10 Oct 2012 12:56:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=aY9gKjaTLSKCKYAeU30VK3/svbfytCTd7I+fKB6XG+U=; b=JT4hJgS9SU9EvTj5sEPNkMJvh6SVClOOB0PJyo706LZBwaYAyUXpwRx5/hPat9Q4bP oWh0y9+1IDjf+W2mcok2GMJcz7YzmKS9I8zSQwRWNNqRhh/rhXmH9MyUxLS2BBwAW/Vk WeEnkaKUqskQSc6++W7/+ut0em0mPMXAW3GRgzRjVGrBTM+0pyMEidSql8ImOFgAltuC YqimZBwalGZRkf/FobKTMnA/tZLxYl2zO5s/JUApp/efjtyBK66c2Ghst4wpy51k+MiE UnM82pqdLA++Zby8U5xsBbpe5HjtPXbte+IINmakMkqYoW1v4O+gs3ArGlSJ2tMsOz6a avGw== Received: by 10.60.13.2 with SMTP id d2mr20420201oec.110.1349898963706; Wed, 10 Oct 2012 12:56:03 -0700 (PDT) MIME-Version: 1.0 Received: by 10.76.153.40 with HTTP; Wed, 10 Oct 2012 12:55:43 -0700 (PDT) In-Reply-To: References: From: iain wright Date: Wed, 10 Oct 2012 12:55:43 -0700 Message-ID: Subject: Re: Flume-ng - Distributed To: user@flume.apache.org Content-Type: multipart/alternative; boundary=e89a8ff2567236e1d404cbb9d803 --e89a8ff2567236e1d404cbb9d803 Content-Type: text/plain; charset=ISO-8859-1 I don't mean to hijack the thread, but is this tiered approach recommended over reading from a local queue and having 10 or so nodes write directly to hbase when using the async hbase sink? -- Iain Wright This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message. On Tue, Oct 9, 2012 at 5:52 PM, Camp, Roy wrote: > You would run a flume-ng instance on each node with an avro-sink. Then > on your collector machine you will run another flume-ng instance with an > avro-collector.**** > > ** ** > > If you run more than one collector you can setup sink groups and define > that it does failover or load balancing.**** > > ** ** > > The concept of a flume master from flume 0.9.x does not exist on > flume-ng. I personally use the node and collector configs in the same > config file under a different agent name, and then keep them synced on all > machines. **** > > ** ** > > These two docs are pretty helpful:**** > > > https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst > > https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeDeveloperGuide.rst > **** > > ** ** > > Thanks,**** > > ** ** > > Roy**** > > ** ** > > ** ** > > ** ** > > ** ** > > *From:* Juan Gentile [mailto:juan.gentile@globant.com] > *Sent:* Tuesday, October 09, 2012 11:04 AM > *To:* user@flume.apache.org > *Subject:* Flume-ng - Distributed**** > > ** ** > > Hi,**** > > ** ** > > I'm new to Flume-ng, I'd like to ask you if you can tell me how I can > accomplish to have an agent distributed in a cluster. I've have developed > my own source and sink version that reads from a queue and the sink stores > the messages read to hdfs. If I want to have this running in multiple > instances, do I have to submit it on each node?**** > > ** ** > > This is my conf file:**** > > agent1.channels.channel1.type = memory**** > > agent1.channels.channel1.capacity = 1000**** > > agent1.channels.channel1.transactionCapacity = 1000**** > > ** ** > > agent1.sources.source1.channels = channel1**** > > agent1.sources.source1.type = MySource**** > > ** ** > > agent1.sinks.sink1.channel = channel1**** > > agent1.sinks.sink1.type = MySink**** > > ** ** > > agent1.channels = channel1**** > > agent1.sources = source1**** > > agent1.sinks = sink1**** > > ** ** > > ** ** > > I see that there is the concept of 'master' a 'node' in the previous > version of flume, do I have something similar here?**** > > ** ** > > Thanks,**** > > Juan**** > --e89a8ff2567236e1d404cbb9d803 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I don't mean to hijack the thread, but is this tiered approach recommen= ded over reading from a local queue and having 10 or so nodes write directl= y to hbase when using the async hbase sink?

--
Iain Wright

This email message is confident= ial, intended only for the recipient(s) named above and may contain informa= tion that is privileged, exempt from disclosure under applicable law. If yo= u are not the intended recipient, do not disclose or disseminate the messag= e to anyone except the intended recipient. If you have received this messag= e in error, or are not the named recipient(s), please immediately notify th= e sender by return email, and delete all copies of this message.


On Tue, Oct 9, 2012 at 5:52 PM, Camp, Ro= y <rcamp@ebay.com> wrote:

You would run a flume-ng = instance on each node with an avro-sink.=A0 Then on your collector machine = you will run another flume-ng instance with an avro-collector.

=A0<= /p>

If you run more than one = collector you can setup sink groups and define that it does failover or loa= d balancing.

=A0<= /p>

The concept of a flume ma= ster from flume 0.9.x does not exist on flume-ng.=A0 I personally use the n= ode and collector configs in the same config file under a different agent name, and then keep them synced on all machines.=A0

=A0<= /p>

These two docs are pretty= helpful:

https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/= FlumeUserGuide.rst
https://github.com/apache/flume/b= lob/trunk/flume-ng-doc/sphinx/FlumeDeveloperGuide.rst

=A0<= /p>

Thanks,

=A0<= /p>

Roy<= /p>

=A0<= /p>

=A0<= /p>

=A0<= /p>

=A0<= /p>

From: Juan Gen= tile [mailto:= juan.gentile@globant.com]
Sent: Tuesday, October 09, 2012 11:04 AM
To: user@= flume.apache.org
Subject: Flume-ng - Distributed

=A0

Hi,

=A0

I'm new to Flume-ng, I'd like to ask you if = you can tell me how I can accomplish to have an agent distributed in a clus= ter. I've have developed my own source and sink version that reads from= a queue and the sink stores the messages read to hdfs. If I want to have this running in multiple instances, do I have t= o submit it on each node?

=A0

This is my conf file:

agent1.channels.channel1.type =3D memory

agent1.channels.channel1.capacity =3D 1000=

agent1.channels.channel1.transactionCapacity =3D 100= 0

=A0

agent1.sources.source1.channels =3D channel1<= u>

agent1.sources.source1.type =3D MySource

=A0

agent1.sinks.sink1.channel =3D channel1

agent1.sinks.sink1.type =3D MySink

=A0

agent1.channels =3D channel1

agent1.sources =3D source1

agent1.sinks =3D sink1

=A0

=A0

I see that there is the concept of 'master' = a 'node' in the previous version of flume, do I have something simi= lar here?

=A0

Thanks,

Juan


--e89a8ff2567236e1d404cbb9d803--