flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ed Judge <ejud...@gmail.com>
Subject Re: HDFS sink to a remote HDFS node
Date Tue, 30 Sep 2014 19:12:58 GMT
I added commons-configuration and there is now another missing dependency.  What do you mean
by “all of Hadoop’s dependencies”?


On Sep 30, 2014, at 2:51 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:

> You actually need to add of all Hadoop’s dependencies to Flume classpath. Looks like
Apache Commons Configuration is missing in classpath.
> 
> Thanks,
> Hari
> 
> 
> On Tue, Sep 30, 2014 at 11:48 AM, Ed Judge <ejudgie@gmail.com> wrote:
> 
> Thank you.  I am using hadoop 2.5 which I think uses protobuf-java-2.5.0.jar.
> 
> I am getting the following error even after adding those 2 jar files to my flume-ng classpath:
> 
> 30 Sep 2014 18:27:03,269 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61)
 - Configuration provider starting
> 30 Sep 2014 18:27:03,278 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)
 - Reloading configuration file:./src.conf
> 30 Sep 2014 18:27:03,288 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)
 - Processing:k1
> 30 Sep 2014 18:27:03,289 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930)
 - Added sinks: k1 Agent: a1
> 30 Sep 2014 18:27:03,289 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)
 - Processing:k1
> 30 Sep 2014 18:27:03,292 WARN  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.<init>:101)
 - Configuration property ignored: i# = Describe the sink
> 30 Sep 2014 18:27:03,292 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)
 - Processing:k1
> 30 Sep 2014 18:27:03,292 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)
 - Processing:k1
> 30 Sep 2014 18:27:03,293 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)
 - Processing:k1
> 30 Sep 2014 18:27:03,293 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)
 - Processing:k1
> 30 Sep 2014 18:27:03,293 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)
 - Processing:k1
> 30 Sep 2014 18:27:03,293 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)
 - Processing:k1
> 30 Sep 2014 18:27:03,293 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)
 - Processing:k1
> 30 Sep 2014 18:27:03,312 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)
 - Post-validation flume configuration contains configuration for agents: [a1]
> 30 Sep 2014 18:27:03,312 INFO  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:150)
 - Creating channels
> 30 Sep 2014 18:27:03,329 INFO  [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:40)
 - Creating instance of channel c1 type memory
> 30 Sep 2014 18:27:03,351 INFO  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:205)
 - Created channel c1
> 30 Sep 2014 18:27:03,352 INFO  [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:39)
 - Creating instance of source r1, type org.apache.flume.source.twitter.TwitterSource
> 30 Sep 2014 18:27:03,363 INFO  [conf-file-poller-0] (org.apache.flume.source.twitter.TwitterSource.configure:110)
 - Consumer Key:        'tobhMtidckJoe1tByXDmI4pW3'
> 30 Sep 2014 18:27:03,363 INFO  [conf-file-poller-0] (org.apache.flume.source.twitter.TwitterSource.configure:111)
 - Consumer Secret:     '6eZKRpd6JvGT3Dg9jtd9fG9UMEhBzGxoLhLUGP1dqzkKznrXuQ'
> 30 Sep 2014 18:27:03,363 INFO  [conf-file-poller-0] (org.apache.flume.source.twitter.TwitterSource.configure:112)
 - Access Token:        '1588514408-o36mOSbXYCVacQ3p6Knsf6Kho17iCwNYLZyA9V5'
> 30 Sep 2014 18:27:03,364 INFO  [conf-file-poller-0] (org.apache.flume.source.twitter.TwitterSource.configure:113)
 - Access Token Secret: 'vBtp7wKsi2BOQqZSBpSBQSgZcc93oHea38T9OdckDCLKn'
> 30 Sep 2014 18:27:03,825 INFO  [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:40)
 - Creating instance of sink: k1, type: hdfs
> 30 Sep 2014 18:27:03,874 ERROR [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:145)
 - Failed to start agent because dependencies were not found in classpath. Error follows.
> java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:38)
> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:36)
> 	at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:106)
> 	at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:208)
> 	at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:553)
> 	at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:272)
> 	at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
> 	at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
> 	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
> 	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> 	... 17 more
> 30 Sep 2014 18:27:33,491 INFO  [agent-shutdown-hook] (org.apache.flume.lifecycle.LifecycleSupervisor.stop:79)
 - Stopping lifecycle supervisor 10
> 30 Sep 2014 18:27:33,493 INFO  [agent-shutdown-hook] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.stop:83)
 - Configuration provider stopping
> [vagrant@localhost 6]$ 
> 
> Is there another jar file I need?
> 
> Thanks.
> 
> On Sep 29, 2014, at 9:04 PM, shengyi.pan <shengyi.pan@gmail.com> wrote:
> 
>> you need hadoop-common-x.x.x.jar and hadoop-hdfs-x.x.x.jar under your flume-ng classpath,
and the dependent hadoop jar version must match your hadoop system.
>>  
>> if sink to hadoop-2.0.0,  you should use "protobuf-java-2.4.1.jar" (defaultly, flume-1.5.0
uses "protobuf-java-2.5.0.jar", the jar file is under flume lib directory ), because the pb
interface of hdfs-2.0 is compiled wtih protobuf-2.4, while using protobuf-2.5 the flume-ng
will fail to start....
>>  
>>  
>>  
>>  
>> 2014-09-30
>> shengyi.pan
>> 发件人:Ed Judge <ejudgie@gmail.com>
>> 发送时间:2014-09-29 22:38
>> 主题:HDFS sink to a remote HDFS node
>> 收件人:"user@flume.apache.org"<user@flume.apache.org>
>> 抄送:
>>  
>> I am trying to run the flume-ng agent on one node with an HDFS sink pointing to an
HDFS filesystem on another node.
>> Is this possible?  What packages/jar files are needed on the flume agent node for
this to work?  Secondary goal is to install only what is needed on the flume-ng node.
>> 
>> # Describe the sink
>> a1.sinks.k1.type = hdfs
>> a1.sinks.k1.hdfs.path = hdfs://<remote IP address>/tmp/
>> 
>> 
>> Thanks,
>> Ed
> 
> 


Mime
View raw message