flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmed Vila <av...@devlogic.eu>
Subject Re: Shutdowning HDFS server leads to flume agent shutdown
Date Fri, 07 Nov 2014 09:47:44 GMT
Hi Mahendran,

Your HDFS Sink should stop, but not the thrift source.
That's one of the basic ideas behind Flume - buffer.

So, once your HDFS is stopped, Sink will stop draining channel and channel
should buffer messages from thrift source until the channel itself gets
filled up.
In that process, unless channel is filled up, your thrift source will not
know anything about downstream failure.


On Fri, Nov 7, 2014 at 10:36 AM, Needham, Guy <Guy.Needham@virginmedia.co.uk
> wrote:

>  Hi Mahendran,
>
> yes that is expected behaviour - I suspect that if you look in the logs
> for this agent, it will have thrown an exception when you shut down the
> HDFS, as it is depending on a compatible HDFS being available.
>
>
> Regards,
> Guy Needham | Data Discovery
> Virgin Media | Enterprise Data, Design & Management
> Bartley Wood Business Park, Hook, Hampshire RG27 9UP
> D 01256 75 3362
> I welcome VSRE emails. Learn more at http://vsre.info/
>
>
>  ------------------------------
> *From:* mahendran m [mailto:mahendranec@hotmail.com]
> *Sent:* 07 November 2014 09:33
> *To:* flume-user@incubator.apache.org
> *Subject:* Shutdowning HDFS server leads to flume agent shutdown
>
>      Hi All,
>
> I am new to Apache flume . I have configured using thrift source to send
> log to HDFS my Config as below
>
>  # list sources, sinks and channels in the agent
> a1.sources = r1
> a1.sinks = k1
> a1.channels = c1
>
>  # avro sink properties
> a1.sources.r1.type = thrift
> a1.sources.r1.bind = localhost
> a1.sources.r1.port = 44444
> a1.sources.r1.interceptors = i1
> a1.sources.r1.interceptors.i1.type =
> DecoderInterceptor.CustomInterceptor$Builder
> a1.sources.r1.interceptors.i1.referalUrl=referalUrl
> a1.sources.r1.interceptors.i1.referalHost=referalHost
>
>  #HDFS sink
> a1.sinks.k1.type = hdfs
> a1.sinks.k1.hdfs.fileType = DataStream
> a1.sinks.k1.hdfs.fileSuffix= .txt
> a1.sinks.k1.hdfs.rollSize = 1048576
> a1.sinks.k1.hdfs.rollCount = 0
> a1.sinks.k1.hdfs.rollInterval = 0
> a1.sinks.k1.hdfs.batchSize = 1000
> a1.sinks.k1.hdfs.minBlockReplicas = 1
> a1.sinks.k1.hdfs.callTimeout = 60000
> a1.sinks.k1.hdfs.path = hdfs://localhost:9000/flumeChannel100/Thrift
>
>  # Use a channel which buffers events in memory
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 10000000
> a1.channels.c1.transactionCapacity = 1000
> a1.channels.c1.byteCapacityBufferPercentage = 10
> a1.channels.c1.byteCapacity = 5368709120
>
>  # define the flow
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
>
>  when i stated HDFS, flume service and generated the logs from c#
> application . my logs are moved to HDFS everything OK still now. but when
> stopped HDFS service . flume agent itself get stopped. Is this default
> behavior ? . or any wen wrong .
>
>  Regards,
> Mahendran
>
>
>
> --------------------------------------------------------------------
> Save Paper - Do you really need to print this e-mail?
>
> Visit www.virginmedia.com for more information, and more fun.
>
> This email and any attachments are or may be confidential and legally
> privileged
> and are sent solely for the attention of the addressee(s). If you have
> received this
> email in error, please delete it from your system: its use, disclosure or
> copying is
> unauthorised. Statements and opinions expressed in this email may not
> represent
> those of Virgin Media. Any representations or commitments in this email are
> subject to contract.
>
> Registered office: Media House, Bartley Wood Business Park, Hook,
> Hampshire, RG27 9UP
> Registered in England and Wales with number 2591237
>



-- 

Best regards,
Ahmed Vila | Senior software developer
DevLogic | Sarajevo | Bosnia and Herzegovina

Office : +387 33 942 123
Mobile: +387 62 139 348

Website: www.devlogic.eu
E-mail   : avila@devlogic.eu
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended
recipient(s) only. This email contains confidential information. It should
not be copied, disclosed to, retained or used by, any party other than the
intended recipient. Any unauthorised distribution, dissemination or copying
of this E-mail or its attachments, and/or any use of any information
contained in them, is strictly prohibited and may be illegal. If you are
not an intended recipient then please promptly delete this e-mail and any
attachment and all copies and inform the sender directly via email. Any
emails that you send to us may be monitored by systems or persons other
than the named communicant for the purposes of ascertaining whether the
communication complies with the law and company policies.

-- 
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended 
recipient(s) only. This email contains confidential information. It should 
not be copied, disclosed to, retained or used by, any party other than the 
intended recipient. Any unauthorised distribution, dissemination or copying 
of this E-mail or its attachments, and/or any use of any information 
contained in them, is strictly prohibited and may be illegal. If you are 
not an intended recipient then please promptly delete this e-mail and any 
attachment and all copies and inform the sender directly via email. Any 
emails that you send to us may be monitored by systems or persons other 
than the named communicant for the purposes of ascertaining whether the 
communication complies with the law and company policies.

Mime
View raw message