flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: Shutdowning HDFS server leads to flume agent shutdown
Date Fri, 07 Nov 2014 10:25:08 GMT
You have a memory channel with capacity of 10M events. Your VM might
be going OOM if there is events are buffered and not enough heap is
allocated.


On Fri, Nov 7, 2014 at 3:03 PM, mahendran m <mahendranec@hotmail.com> wrote:
> Hi All,
>
> I am new to Apache flume . I have configured using thrift source to send log
> to HDFS my Config as below
>
> # list sources, sinks and channels in the agent
> a1.sources = r1
> a1.sinks = k1
> a1.channels = c1
>
> # avro sink properties
> a1.sources.r1.type = thrift
> a1.sources.r1.bind = localhost
> a1.sources.r1.port = 44444
> a1.sources.r1.interceptors = i1
> a1.sources.r1.interceptors.i1.type =
> DecoderInterceptor.CustomInterceptor$Builder
> a1.sources.r1.interceptors.i1.referalUrl=referalUrl
> a1.sources.r1.interceptors.i1.referalHost=referalHost
>
> #HDFS sink
> a1.sinks.k1.type = hdfs
> a1.sinks.k1.hdfs.fileType = DataStream
> a1.sinks.k1.hdfs.fileSuffix= .txt
> a1.sinks.k1.hdfs.rollSize = 1048576
> a1.sinks.k1.hdfs.rollCount = 0
> a1.sinks.k1.hdfs.rollInterval = 0
> a1.sinks.k1.hdfs.batchSize = 1000
> a1.sinks.k1.hdfs.minBlockReplicas = 1
> a1.sinks.k1.hdfs.callTimeout = 60000
> a1.sinks.k1.hdfs.path = hdfs://localhost:9000/flumeChannel100/Thrift
>
> # Use a channel which buffers events in memory
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000000
> a1.channels.c1.transactionCapacity = 1000
> a1.channels.c1.byteCapacityBufferPercentage = 10
> a1.channels.c1.byteCapacity = 5368709120
>
> # define the flow
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
>
> when i stated HDFS, flume service and generated the logs from c# application
> . my logs are moved to HDFS everything OK still now. but when stopped HDFS
> service . flume agent itself get stopped. Is this default behavior ? . or
> any wen wrong .
>
> Regards,
> Mahendran
>
>



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Mime
View raw message