flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From prabhu k <prabhu.fl...@gmail.com>
Subject tail source exec unable to HDFS sink.
Date Mon, 17 Sep 2012 10:17:18 GMT
Hi Users,

I have followed the below link for sample text file to HDFS sink using tail
source.

http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html#more

I have executed flume-ng like as below command. it seems got stuck. and
attached flume.conf file.

#bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf


flume.conf
==========
agent1.sources = tail
agent1.channels = MemoryChannel-2
agent1.sinks = HDFS

agent1.sources.tail.type = exec
agent1.sources.tail.command = tail -F
/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
agent1.sources.tail.channels = MemoryChannel-2

agent1.sources.tail.interceptors = hostint
agent1.sources.tail.interceptors.hostint.type =
org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.tail.interceptors.hostint.preserveExisting = true
agent1.sources.tail.interceptors.hostint.useIP = false

agent1.sinks.HDFS.hdfs.channel = MemoryChannel-2
agent1.sinks.HDFS.hdfs.type = hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://<hostname>:54310/user

agent1.sinks.HDFS.hdfs.fileType = dataStream
agent1.sinks.HDFS.hdfs.writeFormat = text
agent1.channels.MemoryChannel-2.type = memory



flume.log
==========
12/09/17 15:40:05 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 1
12/09/17 15:40:05 INFO node.FlumeNode: Flume node starting - agent1
12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Node manager
starting
12/09/17 15:40:05 INFO properties.PropertiesFileConfigurationProvider:
Configuration provider starting
12/09/17 15:40:05 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 10
12/09/17 15:40:05 INFO properties.PropertiesFileConfigurationProvider:
Reloading configuration file:conf/flume.conf
12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
12/09/17 15:40:05 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
agent1
12/09/17 15:40:05 WARN conf.FlumeConfiguration: Configuration empty for:
HDFS.Removed.
12/09/17 15:40:05 INFO conf.FlumeConfiguration: Post-validation flume
configuration contains configuration  for agents: [agent1]
12/09/17 15:40:05 INFO properties.PropertiesFileConfigurationProvider:
Creating channels
12/09/17 15:40:05 INFO properties.PropertiesFileConfigurationProvider:
created channel MemoryChannel-2
12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Starting new
configuration:{ sourceRunners:{tail=EventDrivenSourceRunner: {
source:org.apache.flume.source.ExecSource@c24c0 }} sinkRunners:{}
channels:{MemoryChannel-2=org.apache.flume.channel.MemoryChannel@140c281} }
12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Starting
Channel MemoryChannel-2
12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Starting
Source tail
12/09/17 15:40:05 INFO source.ExecSource: Exec source starting with
command:tail -F
/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt

Please suggest and help me on this issue.

Mime
View raw message