flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Marcus <jmar...@meetup.com>
Subject Re: issues with configuration updates and clean shutdowns
Date Wed, 15 Jan 2014 15:23:22 GMT
Brock & Jeff:

Thanks so much for your questions.  I've created a gist of an agent with a
simple config that demonstrates the issue and accompanying log files.

https://gist.github.com/joshmarcus/8438107

I'm using flume 1.4.0.

The logs describe a scenario where flume is started, I send log messages,
and then I touch the config (no modifications except for modification
time).  If I edit the config, the situation is the same, but the touch is
to try and minimize this test scenario.  The HDFS file is renamed from
.tmp, but future messages are not logged.

I experimented with sending TERM (for shutdown) and HUP (for potential
reload) but neither work, but I haven't included logs for those situations
(as that's not what I'm actually trying to do here).

--j


On Wed, Jan 15, 2014 at 9:39 AM, Jeff Lord <jlord@cloudera.com> wrote:

> Josh,
>
> If you modify your config than the flume agent will see the config has
> changed and reload any components that have been modified. Are you able to
> provide the logs from the flume agent which occur following a modification
> of the config? What command are you using to signal for a shutdown?
>
> -Jeff
>
>
> On Wed, Jan 15, 2014 at 6:24 AM, Josh Marcus <jmarcus@meetup.com> wrote:
>
>> Hey folks,
>>
>> I didn't get any response at all to this, so let me ask more generally:
>> does automatic reload upon config change generally work?  Does
>> sending a signal for shutdown or reload generally work?  Is this just a
>> symptom of our configuration?
>>
>> --j
>>
>>
>>
>> On Thu, Jan 9, 2014 at 6:50 PM, Josh Marcus <jmarcus@meetup.com> wrote:
>>
>>> Hey folks,
>>>
>>> I'm setting up new flume agents, and I'm noticing issues with both live
>>> updates to the configuration and cleanly shutting down an agent.   I was
>>> wondering if anyone should shed light on whether what I'm seeing is based
>>> on known issues or if there are any likely culprits.
>>>
>>> First of all, if I update the configuration, the agent notices the
>>> updated configuation, shuts down (successfully) but doesn't restart with
>>> the updated configuration.
>>>
>>> Secondly, sending a signal (I've tried a variety) to a running flume
>>> agent doesn't cleanly shutdown the agent -- for example, the .tmp output
>>> file doesn't get moved to its permanent location.
>>>
>>> So, if I update the configuration (or just touch it), I get a clean
>>> shutdown -- but it won't restart.  But I don't currently have alternate way
>>> that will reliably shut it down.
>>>
>>> Any advice?
>>>
>>> --j
>>>
>>>
>>
>

Mime
View raw message