flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Re: Flume restart
Date Fri, 05 Dec 2014 16:15:52 GMT
I have to join this thread for .... obvious reasons ;)

Bojan - you'll get better help if you share your configs.  If you are
monitoring Flume with something, sharing various metrics/charts will help,
too.  You could also run Flume under a Java profiler and see what's eating
your heap.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Thu, Dec 4, 2014 at 4:40 PM, Ahmed Vila <avila@devlogic.eu> wrote:

> Zdravo Bojane :)
>
> Flume is well designed and it shouldn't eat up the memory. On the other
> hand, miss-configuration can effectively bring server to a crawl and
> eventually produce events loss.
> Pasting your configuration in here, along with basic hardware info behind
> it and a size of your single event in bytes would be helpful.
>
> The most common things to blame is using inappropriately large memory
> channel size for a given amount of memory, transaction size, HDFS sink
> batch size etc. because all of them are stored in memory.
>
> Anyway, you can achieve graceful restart by changing flume's configuration
> file modification time - basically just touching it.
> It will sense a change and as a result close sources, sinks and channels,
> and start them again without overhead of booting up JVM.
> That should trigger java garbage collector to clean up resources
> associated to those closed instances of sources, sinks and channels.
> As a result, you might loose some events if you're using memory channel
> since I think it doesn't have a shutdown procedure. Sink should flush batch
> to the HDFS, but that also should be tested.
>
> Also, tweaking Java GC could be of a help, but I never had a need to do so
> with Flume.
>
>
> On Thu, Dec 4, 2014 at 9:18 PM, Bojan Kostić <blood9raven@gmail.com>
> wrote:
>
>> Hi,
>>
>> I have a problem with my flume setup. Overtime they just take too much
>> memory. And i need them to restart every now and then, I searched web and i
>> did not found any clue how to fix this. Some people blame HDFS...
>> For now i just kill process with TERM signal and then wait for couple of
>> minutes to shutdown. Now i wish to do this automatically every day. But i
>> don't want to lose logs. Is there a way to do this? I checked flume-ng
>> script and there is only start. I could write my own sh script which will
>> send TERM signal and then check for flume process and if there is none
>> start again. But first i want to check is there some smarter way to do this.
>>
>> Best
>> Bojan
>>
>
>
>
> --
>
> Best regards,
> Ahmed Vila | Senior software developer
> DevLogic | Sarajevo | Bosnia and Herzegovina
>
> Office : +387 33 942 123
> Mobile: +387 62 139 348
>
> Website: www.devlogic.eu
> E-mail   : avila@devlogic.eu
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.
>
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.

Mime
View raw message