From user-return-3671-apmail-flume-user-archive=flume.apache.org@flume.apache.org Thu Feb 7 18:07:34 2013 Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8F8C5E004 for ; Thu, 7 Feb 2013 18:07:34 +0000 (UTC) Received: (qmail 78629 invoked by uid 500); 7 Feb 2013 18:07:34 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 78502 invoked by uid 500); 7 Feb 2013 18:07:34 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 78494 invoked by uid 99); 7 Feb 2013 18:07:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Feb 2013 18:07:33 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of sesh12@gmail.com designates 209.85.215.43 as permitted sender) Received: from [209.85.215.43] (HELO mail-la0-f43.google.com) (209.85.215.43) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Feb 2013 18:07:27 +0000 Received: by mail-la0-f43.google.com with SMTP id ek20so2936131lab.16 for ; Thu, 07 Feb 2013 10:07:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=pw28KUsOlXo4DKEG3ZMhQJV8BdQqeMeMmR3JVA9ED50=; b=XtJdaWUUFQzPyXVLYZ/5DAcd5MidZV2JAtt4h59Pa+dVFXNbtJrRpBz8kDVgPWam0G hXJeIAuObVLI/bdajTrb9LVzcuidv7KI46X8lRg9Z/O8y6SFN8l+dXG8yjFuX6SObiv4 K5OGcZUCBCG50EQ3yZ9KRv04FTZuQT+B2mf1ORQWCQTJJbJ7LYfMdAg6yySikr/cCLKA AwI5jRNNkdWv1qqY0LC9t/Htr2dZ5qjYfa32BCYD2Kr+Jx9FhtqrfoNfJVbPvUA5xRaU 4NrKGShwS0s21nRt6ufGR95fTRNKNVkCrKXEih4k/dhJjZUTopw3e6nngfCyGsXfduxU y5jg== MIME-Version: 1.0 X-Received: by 10.152.148.4 with SMTP id to4mr2118555lab.39.1360260426520; Thu, 07 Feb 2013 10:07:06 -0800 (PST) Received: by 10.112.80.166 with HTTP; Thu, 7 Feb 2013 10:07:06 -0800 (PST) In-Reply-To: References: Date: Thu, 7 Feb 2013 12:07:06 -0600 Message-ID: Subject: Re: Does Flume NG requires to be installed on all the sources? From: Seshu V To: user@flume.apache.org Content-Type: multipart/alternative; boundary=e89a8f2345b386566304d5264f6d X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f2345b386566304d5264f6d Content-Type: text/plain; charset=ISO-8859-1 Hello Jeff, Thanks for the reply. My use case is not really special. We have multiple products and each product emits traditional log messages in different servers. I would like to stream those into HDFS. The logs are generally in apache or log4j format. So, I have many sources from where I want to stream the logs into HDFS. I can have a channel/collector machine where I install flume. I guess, my question is, do I need to install flume on the servers where the log messages lie and do I need to install flume in HDFS namenode too? Thanks, - Seshu On Wed, Feb 6, 2013 at 7:47 PM, Jeff Lord wrote: > Seshu, > > It really is going to depend on your use case. > Though it sounds that you may need to run an agent on each of the source > machines. > Which source do you plan to use? It may also be the case that you can use > the flume rpc client to write data directly from your application to the > flume collector machine. > > http://flume.apache.org/FlumeDeveloperGuide.html#rpc-client-interface > > -Jeff > > > On Wed, Feb 6, 2013 at 4:49 PM, Seshu V wrote: > >> Hi All, >> >> I have used Flume 0.9.3 a while back, it worked fine at that time. >> Now, I am looking to use 'Flume NG', started reading documentation today. >> In Flume 0.9.3, I installed flume agents on the servers wherever I had the >> data source. And, I had a collector machine separately. My sink was >> HDFS. I see that Flume NG is using Channel. >> My question is that I have multiple source servers and my sink is >> HDFS. I also have another machine for Channel (collector in old days). >> Do I need to install flume NG in all the source machines and Channel >> machine? Or can I install flume NG only on the Channel server and >> (somehow) specify in the configuration to pull data from source machines >> and specify the sink as HDFS? >> Thanks in advance for your replies.. >> >> Thanks, >> - Seshu >> >> > > --e89a8f2345b386566304d5264f6d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello Jeff,=A0

=A0 =A0Thanks for the reply. =A0My use ca= se is not really special. =A0We have multiple products and each product emi= ts traditional log messages in different servers. =A0I would like to stream= those into HDFS. =A0The logs are generally in apache or log4j format. =A0<= /div>
=A0 =A0So, I have many sources from where I want to stream the logs in= to HDFS. =A0 I can have a channel/collector machine where I install flume. = =A0 I guess, my question is, do I need to install flume on the servers wher= e the log messages lie and do I need to install flume in HDFS namenode too?=

Thanks,
- Seshu =A0

On Wed, Feb 6, 2013 at 7:47 PM, Jeff Lord <jlord@cloudera.com> wrote:
Seshu,

I= t really is going to depend on your use case.
Though it sounds th= at you may need to run an agent on each of the source machines.
Which source do you plan to use? It may also be the case that you can = use the flume rpc client to write data directly from your application to th= e flume collector machine.

-Jeff


On Wed, Feb 6, 20= 13 at 4:49 PM, Seshu V <sesh12@gmail.com> wrote:
Hi All,

=A0 =A0 I have us= ed Flume 0.9.3 a while back, it worked fine at that time. =A0Now, I am look= ing to use 'Flume NG', started reading documentation today. =A0In F= lume 0.9.3, I installed flume agents on the servers wherever I had the data= source. =A0 And, I had a collector machine separately. =A0My sink was HDFS= . =A0 I see that Flume NG is using Channel. =A0=A0=A0
=A0 =A0 My question is that I have multiple source servers and my sink= is HDFS. =A0I also have another machine for Channel (collector in old days= ). =A0 Do I need to install flume NG =A0in all the source machines and Chan= nel machine? =A0Or can I install flume NG only on the Channel server and (s= omehow) specify in the configuration to pull data from source machines and = specify the sink as HDFS?
=A0 =A0 =A0Thanks in advance for your replies..

Thanks,
- Seshu
=A0


--e89a8f2345b386566304d5264f6d--