flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Himanshu Patidar <himanshu.pati...@hotmail.com>
Subject RE: HDFS Sink Error
Date Tue, 21 Jan 2014 09:29:13 GMT
I am getting the same error even after building flume with guava-11.0.2
Thanks,Himanshu

Date: Tue, 21 Jan 2014 17:18:06 +0800
Subject: Re: HDFS Sink Error
From: chenshangan521@gmail.com
To: user@flume.apache.org

you can try this patch https://issues.apache.org/jira/browse/FLUME-2172and build flume with
guava-11.0.2 the same version as hadoop2.0.x used, currently flume use guava-10.0.1, so just
change the version 

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>10.0.1</version>
</dependency>


On Tue, Jan 21, 2014 at 3:35 PM, Himanshu Patidar <himanshu.patidar@hotmail.com> wrote:




Hi,
I am trying to feed a single node HDFS cluster.
But I getting this error :
2014-01-21 08:29:44,426 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.HDFSDataStream.configure(HDFSDataStream.java:56)]
Serializer = TEXT, UseRawLocalFileSystem = false
2014-01-21 08:29:44,474 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:219)]
Creating hdfs://xyz.16.137.81:54545/flume/FlumeData.1390289384427.tmp
2014-01-21 08:29:44,878 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:422)]
process failedjava.lang.UnsupportedOperationException: This is supposed to be overridden by
subclasses.
        at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
       at org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$FsPermissionProto.getSerializedSize(HdfsProtos.java:5407)
        at com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:749)
       at com.google.protobuf.CodedOutputStream.computeMessageSize(CodedOutputStream.java:530)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$CreateRequestProto.getSerializedSize(ClientNamenodeProtocolProtos.java:2371)
       at com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:149)
       at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:193)
        at $Proxy11.create(Unknown Source)        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:601)        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
       at $Proxy11.create(Unknown Source)        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:192)
        at org.apache.hadoop.hdfs.DFSOutputStream.<init>(DFSOutputStream.java:1298)
       at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1317)
       at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1242)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1199)        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:273)
       at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:262)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:79)
       at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:851)        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:832)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:731)        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:720)
       at org.apache.flume.sink.hdfs.HDFSDataStream.open(HDFSDataStream.java:80)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:227)        at
org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:220)        at org.apache.flume.sink.hdfs.BucketWriter$8$1.run(BucketWriter.java:557)
        at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:160)  
     at org.apache.flume.sink.hdfs.BucketWriter.access$1000(BucketWriter.java:56)        at
org.apache.flume.sink.hdfs.BucketWriter$8.call(BucketWriter.java:554)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
       at java.lang.Thread.run(Thread.java:722)

My conf file :-

Naming the components in this Agent###############################httpagent.sources = http-sourcehttpagent.sinks
= local-file-sinkhttpagent.channels = ch3

# Define / Configure Source###############################httpagent.sources.http-source.type
= org.apache.flume.source.http.HTTPSource
httpagent.sources.http-source.channels = ch3
httpagent.sources.http-source.port = 44444

# Local File Sink###############################httpagent.sinks.local-file-sink.type = hdfshttpagent.sinks.local-file-sink.channel
= ch3
httpagent.sinks.local-file-sink.hdfs.path = hdfs://xyz.16.137.81:54545/flumehttpagent.sinks.local-file-sink.hdfs.fileType
= DataStream#httpagent.sinks.local-file-sink.hdfs.filePrefix = events-
httpagent.sinks.local-file-sink.hdfs.round = truehttpagent.sinks.local-file-sink.hdfs.roundValue
= 1httpagent.sinks.local-file-sink.hdfs.roundUnit = minute
# Channels
###############################httpagent.channels.ch3.type = memoryhttpagent.channels.ch3.capacity
= 1000httpagent.channels.ch3.transactionCapacity = 100


Apart from that I have these jars for hadoop/hdfs in the /lib folder :-
commons-codec-1.4.jar
commons-configuration-1.6.jar
commons-httpclient-3.1.jar
hadoop-annotations-2.0.0-cdh4.2.0.jar
hadoop-auth-2.0.0-cdh4.2.0.jar
hadoop-client-2.0.0-mr1-cdh4.2.0.jar
hadoop-common-2.0.0-cdh4.2.0.jar
hadoop-core-2.0.0-mr1-cdh4.2.0.jarhadoop-hdfs-2.0.0-cdh4.2.0.jar
protobuf-java-2.5.0.jar

I believe this error is coming from protobuf-java-2.5.0.jar file.

Any suggestions will be of great help!!!



Thanks,Himanshu 		 	   		  


-- 
have a good day! chenshang'an

 		 	   		  
Mime
View raw message