flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: flume-ng Unable to put batch on required channel error
Date Fri, 01 Mar 2013 05:24:35 GMT
Try increasing the callTimeout for HDFS Sink - that is in milliseconds. Also, you should try
using a batch size/txnEvent max of at least 100 (or more). Using more HDFS sinks will also
improve performance by increased parallel writes  since each sink is single threaded and writes
to only one file at a time. (note that each HDFS Sink should write to a different directory).
  

--  
Hari Shreedharan


On Thursday, February 28, 2013 at 7:10 PM, 周梦想 wrote:

> Hello,
>  
> I wrote a test java file to put local files to hdfs, it put 1.1GB files to hdfs spent
about 35 seconds. while using flume, 100MB files it should spend about 3-5 minutes.  
> why flume is so slow and always report error? how to enlarge the flume hdfs sink throughput?
>  
> Andy
>  
> 2013/3/1 周梦想 <ablozhou@gmail.com (mailto:ablozhou@gmail.com)>
> > hello,
> > I saw lot's of flume-ng 1.3.1 agent Exception said  Unable to put batch on required
channel,and then disconnected.
> > what is the effect? lost data or duplicate data?
> > What caused this error?
> > How to avoid this error?
> >  
> > some of my conf:
> > agent46.channels.memch1.type = memory
> > agent46.channels.memch1.capacity = 100000
> > agent46.channels.memch1.transactionCapactiy = 1000
> >  
> >  
> > agent46.sources.userlogsrc.type = avro
> >  
> > agent46.sinks.myhdfssink.type = hdfs
> > agent46.sinks.myhdfssink.hdfs.rollInterval = 620
> > agent46.sinks.myhdfssink.hdfs.appendTimeout = 1000
> > agent46.sinks.myhdfssink.hdfs.rollSize = 209715200
> > agent46.sinks.myhdfssink.hdfs.rollCount = 600000
> > agent46.sinks.myhdfssink.hdfs.batchSize = 50
> > agent46.sinks.myhdfssink.hdfs.txnEventMax = 100000
> > agent46.sinks.myhdfssink.hdfs.threadsPoolSize= 100
> >  
> >  
> > ...
> > 01 Mar 2013 03:00:16,368 ERROR [pool-7-thread-3] (org.apache.flume.source.AvroSource.appendBatch:261)
 - Avro source gamelogsrc: Unable to process event batch. Exception follows.
> > org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name:
memch1}
> >         at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> >         at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)
> >         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> >         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:601)
> >         at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
> >         at org.apache.avro.ipc.Responder.respond(Responder.java:149)
> >         at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
> >         at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> >         at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
> >         at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> >         at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
> >         at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> >         at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
> >         at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
> >         at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
> >         at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
> >         at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
> >         at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
> >         at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
> >         at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> >         at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
> >         at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> >         at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
> >         at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> >         at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
> >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >         at java.lang.Thread.run(Thread.java:722)
> > Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't
be acquired Sinks are likely not keeping up with sources, or the buffer size is too tight
> >         at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:126)
> >         at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> >         at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
> >         ... 28 more
> > 01 Mar 2013 03:00:20,503 DEBUG [conf-file-poller-0] (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:188)
 - Checking file:conf/hdfs.conf for changes
> > 01 Mar 2013 03:00:21,193 INFO  [pool-9-thread-1] (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
 - [id: 0x40044674, /192.168.133.44:2404 (http://192.168.133.44:2404) :> /192.168.133.47:5140
(http://192.168.133.47:5140)] DISCONNECTED
> >  
> > dleUpstream:171)  - [id: 0x40044674, /192.168.133.44:2404 (http://192.168.133.44:2404)
:> /192.168.133.47:5140 (http://192.168.133.47:5140)] UNBOUND
> > 01 Mar 2013 03:00:21,193 INFO  [pool-9-thread-1] (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream:171)
 - [id: 0x40044674, /192.168.133.44:2404 (http://192.168.133.44:2404) :> /192.168.133.47:5140
(http://192.168.133.47:5140)] CLOSED
> > 01 Mar 2013 03:00:21,194 INFO  [pool-9-thread-1] (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed:209)
 - Connection to /192.168.133.44:2404 (http://192.168.133.44:2404) disconnected.
> >  
> >  
> > Thanks,
> > Andy
> >  
>  
>  
>  


Mime
View raw message