flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: tail source exec unable to HDFS sink.
Date Thu, 20 Sep 2012 14:48:56 GMT
It actually looks like it's working. Are you sure no data is showing
up in hdfs and that new data is being appended to the file you are
tailing?

On Thu, Sep 20, 2012 at 3:48 AM, prabhu k <prabhu.flume@gmail.com> wrote:
> Hi Brock,
>
> I have changed, as per your suggestion,but still the issue same, script
> seems stuck, pasted flume.log file.
>
> flume.log
> ==========
> 12/09/20 14:07:40 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
> supervisor 1
> 12/09/20 14:07:40 INFO node.FlumeNode: Flume node starting - agent1
> 12/09/20 14:07:40 INFO nodemanager.DefaultLogicalNodeManager: Node manager
> starting
> 12/09/20 14:07:40 INFO properties.PropertiesFileConfigurationProvider:
> Configuration provider starting
> 12/09/20 14:07:40 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
> supervisor 10
> 12/09/20 14:07:40 INFO properties.PropertiesFileConfigurationProvider:
> Reloading configuration file:conf/flume.conf
> 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Processing:HDFS
> 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
> agent1
> 12/09/20 14:07:40 INFO conf.FlumeConfiguration: Post-validation flume
> configuration contains configuration  for agents: [agent1]
> 12/09/20 14:07:40 INFO properties.PropertiesFileConfigurationProvider:
> Creating channels
> 12/09/20 14:07:40 INFO properties.PropertiesFileConfigurationProvider:
> created channel MemoryChannel-2
> 12/09/20 14:07:40 INFO sink.DefaultSinkFactory: Creating instance of sink
> HDFS typehdfs
>
>
> On Mon, Sep 17, 2012 at 8:01 PM, Brock Noland <brock@cloudera.com> wrote:
>>
>> Hi,
>>
>> I believe, this line:
>> agent1.sinks.HDFS.hdfs.type = hdfs
>>
>> should be:
>> agent1.sinks.HDFS.type = hdfs
>>
>> Brock
>>
>> On Mon, Sep 17, 2012 at 5:17 AM, prabhu k <prabhu.flume@gmail.com> wrote:
>> > Hi Users,
>> >
>> > I have followed the below link for sample text file to HDFS sink using
>> > tail
>> > source.
>> >
>> >
>> > http://cloudfront.blogspot.in/2012/06/how-to-use-host-escape-sequence-in.html#more
>> >
>> > I have executed flume-ng like as below command. it seems got stuck. and
>> > attached flume.conf file.
>> >
>> > #bin/flume-ng agent -n agent1 -c /conf -f conf/flume.conf
>> >
>> >
>> > flume.conf
>> > ==========
>> > agent1.sources = tail
>> > agent1.channels = MemoryChannel-2
>> > agent1.sinks = HDFS
>> >
>> > agent1.sources.tail.type = exec
>> > agent1.sources.tail.command = tail -F
>> >
>> > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
>> > agent1.sources.tail.channels = MemoryChannel-2
>> >
>> > agent1.sources.tail.interceptors = hostint
>> > agent1.sources.tail.interceptors.hostint.type =
>> > org.apache.flume.interceptor.HostInterceptor$Builder
>> > agent1.sources.tail.interceptors.hostint.preserveExisting = true
>> > agent1.sources.tail.interceptors.hostint.useIP = false
>> >
>> > agent1.sinks.HDFS.hdfs.channel = MemoryChannel-2
>> > agent1.sinks.HDFS.hdfs.type = hdfs
>> > agent1.sinks.HDFS.hdfs.path = hdfs://<hostname>:54310/user
>> >
>> > agent1.sinks.HDFS.hdfs.fileType = dataStream
>> > agent1.sinks.HDFS.hdfs.writeFormat = text
>> > agent1.channels.MemoryChannel-2.type = memory
>> >
>> >
>> >
>> > flume.log
>> > ==========
>> > 12/09/17 15:40:05 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
>> > supervisor 1
>> > 12/09/17 15:40:05 INFO node.FlumeNode: Flume node starting - agent1
>> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Node
>> > manager
>> > starting
>> > 12/09/17 15:40:05 INFO properties.PropertiesFileConfigurationProvider:
>> > Configuration provider starting
>> > 12/09/17 15:40:05 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
>> > supervisor 10
>> > 12/09/17 15:40:05 INFO properties.PropertiesFileConfigurationProvider:
>> > Reloading configuration file:conf/flume.conf
>> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
>> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
>> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
>> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
>> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Processing:HDFS
>> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
>> > agent1
>> > 12/09/17 15:40:05 WARN conf.FlumeConfiguration: Configuration empty for:
>> > HDFS.Removed.
>> > 12/09/17 15:40:05 INFO conf.FlumeConfiguration: Post-validation flume
>> > configuration contains configuration  for agents: [agent1]
>> > 12/09/17 15:40:05 INFO properties.PropertiesFileConfigurationProvider:
>> > Creating channels
>> > 12/09/17 15:40:05 INFO properties.PropertiesFileConfigurationProvider:
>> > created channel MemoryChannel-2
>> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Starting
>> > new
>> > configuration:{ sourceRunners:{tail=EventDrivenSourceRunner: {
>> > source:org.apache.flume.source.ExecSource@c24c0 }} sinkRunners:{}
>> >
>> > channels:{MemoryChannel-2=org.apache.flume.channel.MemoryChannel@140c281} }
>> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Starting
>> > Channel MemoryChannel-2
>> > 12/09/17 15:40:05 INFO nodemanager.DefaultLogicalNodeManager: Starting
>> > Source tail
>> > 12/09/17 15:40:05 INFO source.ExecSource: Exec source starting with
>> > command:tail -F
>> >
>> > /usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/flume_test.txt
>> >
>> > Please suggest and help me on this issue.
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>
>



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Mime
View raw message