flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kartik Vashishta <kartik.u...@gmail.com>
Subject Flume/hadoop question
Date Sun, 20 Mar 2016 15:12:53 GMT
Team,

I have been following this web page:
http://cuddletech.com/?p=795

I have been using the most recent version of the software.

I have been able to install teh agent and the collector, I cannot however
get the logs to be written to the HDFS path.

flume was instaleed on the webserver and on the hadoop slave, however,
while the two collectors don't show any obvious errors, and logs are
written in /var/log/flume, they are not present on the HDFS path.

flume.conf on the collector (running on hadoop slave):
collector.sources = AvroIn
collector.sources.AvroIn.type = avro
collector.sources.AvroIn.bind = 0.0.0.0
collector.sources.AvroIn.port = 4545
collector.sources.AvroIn.channels = mc1 mc2

## Channels ########################################################
## Source writes to 2 channels, one for each sink (Fan Out)
collector.channels = mc1 mc2

# http://flume.apache.org/FlumeUserGuide.html#memory-channel
collector.channels.mc1.type = memory
collector.channels.mc1.capacity = 100

collector.channels.mc2.type = memory
collector.channels.mc2.capacity = 100

## Sinks ###########################################################
collector.sinks = LocalOut HadoopOut

## Write copy to Local Filesystem (Debugging)
# http://flume.apache.org/FlumeUserGuide.html#file-roll-sink
collector.sinks.LocalOut.type = file_roll
collector.sinks.LocalOut.sink.directory = /var/log/flume
collector.sinks.LocalOut.sink.rollInterval = 0
collector.sinks.LocalOut.channel = mc1

## Write to HDFS
# http://flume.apache.org/FlumeUserGuide.html#hdfs-sink
collector.sinks.HadoopOut.type = hdfs
collector.sinks.HadoopOut.channel = mc2
collector.sinks.HadoopOut.hdfs.path =
/flume1/events/%{log_type}/%{host}/%y-%m-%d
#collector.sinks.HadoopOut.hdfs.path =
/opt/hadoop/hadoop/dfs/name/data/%{log_type}/%{host}/%y-%m-%d
collector.sinks.HadoopOut.hdfs.fileType = DataStream
collector.sinks.HadoopOut.hdfs.writeFormat = Text
collector.sinks.HadoopOut.hdfs.rollSize = 0
collector.sinks.HadoopOut.hdfs.rollCount = 10000
collector.sinks.HadoopOut.hdfs.rollInterval = 600

Command to start flume on collector 9hadoop slave)
 bin/flume-ng agent -c conf -f conf/flume.conf
 -Dflume.root.logger=INFO,console -n collector


NO ERRORS reported, only information messages

I did try this on the Hadoop master:
[hadoop@test49 hadoop]$ /opt/hadoop/hadoop/bin/hadoop fs -mkdir /flume1
/opt/hadoop/hadoop/bin/hadoop fs -mkdir /flume1/events

BUT
/opt/hadoop/hadoop/bin/hadoop fs -ls /flume1/events
shows nothing

On the hadoop slave:
[hadoop@test51 conf]$ /opt/hadoop/hadoop/bin/hadoop fs -ls /flume1/events
Exception in thread "main" java.lang.RuntimeException: core-site.xml not
found


Not sure if this is expected behaviour.

Please advise. Thanks in advance.

Mime
View raw message