flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will McQueen <w...@cloudera.com>
Subject Re: Flume ng not running
Date Sat, 31 Mar 2012 07:58:53 GMT
Hi Mohit,

The prefix in each line of the config file says "foo", and represents the
name of the agent. So you would need to specify "--name foo" on the command
line so that flume knows which agent to use (because a single config file
can actually specify configs for multiple agents). Also, please see the
help by running "./bin/flume-ng help". I've posted the output here for your

Usage: ./bin/flume-ng <command> [options]...

  help                  display this help text
  node                  run a Flume node
  avro-client           run an avro Flume client

global options:
  --conf,-c <conf>      use configs in <conf> directory
  --classpath,-C <cp>   override the classpath for flume jars

node options:
  --conf-file,-f        specify a config file (required)
  --name,-n             the name of this node (required)
  --help,-h             display help text

avro-client options:
  --host,-H <host>      hostname to which events will be sent (required)
  --port,-p <port>      port of the avro source (required)
  --filename,-F <file>  text file to stream to avro source [default: std
  --help,-h             display help text

Note that if <conf> directory is specified, then it is always included first
in the classpath.

So look at "commands". You need to specify exactly one of those... you're
specifying "node", as expected.
Next, look at "global options"... where global means that these options are
relevant whether you choose the "node" command or the "avro-client"
command. You specified the --conf option, but the value you specified is
not the conf dir. You specified the conf file (which needs to be the value
of the --conf-file option). Also, no need to specify --classpath.
Next, the "node options" section shows 3 options for when you specify the
"node" command. I see that you're missing both "--conf-file" and "--name",
both of which are listed as required.
No need to look at "avro-client" options for your particular case.

So something like this should work:
./bin/flume-ng node --conf ../conf --conf-file
../conf/flume-conf.properties --name foo


On Fri, Mar 30, 2012 at 4:57 PM, Mohit Anchlia <mohitanchlia@gmail.com>wrote:

> I am trying to test how flume ng works so I create a conf and tried to run:
>  foo.sources = avroSrc
> foo.channels = memoryChannel
> foo.sinks = hdfsSink
> # For each one of the sources, the type is defined
> foo.sources.avroSrc.type = avro
> # The channel can be defined as follows.
> foo.sources.avroSrc.channels = memoryChannel
> # Each sink's type must be defined
> foo.sinks.hdfsSink.type = hdfs
> foo.sinks.hdfsSink.path = hdfs://dsdb1:9000/flume
> #Specify the channel the sink should use
> foo.sinks.hdfsSink.channel = memoryChannel
> # Each channel's type is defined.
> foo.channels.memoryChannel.type = memory
> # Other config values specific to each type of channel(sink or source)
> # can be defined as well
> # In this case, it specifies the capacity of the memory channel
> foo.channels.memoryChannel.capacity = 100
> But java process is not running:
> [root@dslg1 flume-1.2.0-incubating-SNAPSHOT]# ./bin/flume-ng  node --conf
> ../conf/flume-conf.properties
> + exec /usr/java/jdk1.6.0_24/bin/java -Xmx20m -cp
> '../conf/flume-conf.properties:/root/.mohit/flume-1.2.0-incubating-SNAPSHOT/lib/*'
> -Djava.library.path= org.apache.flume.node.Application
> log4j:WARN No appenders could be found for logger
> (org.apache.flume.node.Application).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>  --
> Can you please suggest what I might be doing wrong?

View raw message