flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@gmail.com>
Subject Re: flume ng error while going for hdfs sink
Date Thu, 05 Jul 2012 07:52:16 GMT
Hi,

looks like the sink can't connect to hdfs, try agent1.sinks.HDFS.hdfs.path = hdfs://localhost:54310/users/flume
or similar.

- Alex 


On Jul 5, 2012, at 9:28 AM, Amit Handa wrote:

> HI All,
> 
> While trying to run Flume ng using HDFS SInk, and using avro Client.. i am
> getting IOException. Kindly help in resolving this issue
> 
> Exception log is as follows:
> 2012-07-05 12:01:32,789 (conf-file-poller-0) [INFO -
> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
> Creating instance of sink HDFS typehdfs
> 2012-07-05 12:01:32,816 (conf-file-poller-0) [DEBUG -
> org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)]
> java.io.IOException: config()
>    at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
>    at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
>    at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
>    at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
>    at
> org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:516)
>    at
> org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:239)
>    at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
>    at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:373)
>    at
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
>    at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
>    at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
>    at
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
>    at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>    at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>    at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>    at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>    at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>    at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>    at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>    at java.lang.Thread.run(Thread.java:662)
> 
> 
> Flume.conf file, contents are :
> # Define a memory channel called ch1 on agent1
> agent1.channels.ch1.type = memory
> 
> # Define an Avro source called avro-source1 on agent1 and tell it
> # to bind to 0.0.0.0:41414. Connect it to channel ch1.
> agent1.sources.avro-source1.channels = ch1
> agent1.sources.avro-source1.type = avro
> agent1.sources.avro-source1.bind = 0.0.0.0
> agent1.sources.avro-source1.port = 41414
> 
> # Define a hdfs sink that simply logs all events it receives
> # and connect it to the other end of the same channel.
> agent1.sinks.HDFS.channel = ch1
> agent1.sinks.HDFS.type = hdfs
> agent1.sinks.HDFS.hdfs.path = hdfs://localhost:54310/ (in my setup hdfs is
> using port 54310
> agent1.sinks.HDFS.hdfs.file.Type = DataStream
> 
> 
> # Finally, now that we've defined all of our components, tell
> # agent1 which ones we want to activate.
> agent1.channels = ch1
> agent1.sources = avro-source1
> #agent1.sinks = fileroll_sink1
> agent1.sinks = HDFS
> 
> Flume NG Command
> bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1


--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF


Mime
View raw message