flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Marcus <jmar...@meetup.com>
Subject Re: issues with configuration updates and clean shutdowns
Date Tue, 21 Jan 2014 15:41:36 GMT
Hey folks,

Any thoughts now that I've posted a config and logs?

--j


On Wed, Jan 15, 2014 at 10:23 AM, Josh Marcus <jmarcus@meetup.com> wrote:

> Brock & Jeff:
>
> Thanks so much for your questions.  I've created a gist of an agent with a
> simple config that demonstrates the issue and accompanying log files.
>
> https://gist.github.com/joshmarcus/8438107
>
> I'm using flume 1.4.0.
>
> The logs describe a scenario where flume is started, I send log messages,
> and then I touch the config (no modifications except for modification
> time).  If I edit the config, the situation is the same, but the touch is
> to try and minimize this test scenario.  The HDFS file is renamed from
> .tmp, but future messages are not logged.
>
> I experimented with sending TERM (for shutdown) and HUP (for potential
> reload) but neither work, but I haven't included logs for those situations
> (as that's not what I'm actually trying to do here).
>
> --j
>
>
> On Wed, Jan 15, 2014 at 9:39 AM, Jeff Lord <jlord@cloudera.com> wrote:
>
>> Josh,
>>
>> If you modify your config than the flume agent will see the config has
>> changed and reload any components that have been modified. Are you able to
>> provide the logs from the flume agent which occur following a modification
>> of the config? What command are you using to signal for a shutdown?
>>
>> -Jeff
>>
>>
>> On Wed, Jan 15, 2014 at 6:24 AM, Josh Marcus <jmarcus@meetup.com> wrote:
>>
>>> Hey folks,
>>>
>>> I didn't get any response at all to this, so let me ask more generally:
>>> does automatic reload upon config change generally work?  Does
>>> sending a signal for shutdown or reload generally work?  Is this just a
>>> symptom of our configuration?
>>>
>>> --j
>>>
>>>
>>>
>>> On Thu, Jan 9, 2014 at 6:50 PM, Josh Marcus <jmarcus@meetup.com> wrote:
>>>
>>>> Hey folks,
>>>>
>>>> I'm setting up new flume agents, and I'm noticing issues with both live
>>>> updates to the configuration and cleanly shutting down an agent.   I was
>>>> wondering if anyone should shed light on whether what I'm seeing is based
>>>> on known issues or if there are any likely culprits.
>>>>
>>>> First of all, if I update the configuration, the agent notices the
>>>> updated configuation, shuts down (successfully) but doesn't restart with
>>>> the updated configuration.
>>>>
>>>> Secondly, sending a signal (I've tried a variety) to a running flume
>>>> agent doesn't cleanly shutdown the agent -- for example, the .tmp output
>>>> file doesn't get moved to its permanent location.
>>>>
>>>> So, if I update the configuration (or just touch it), I get a clean
>>>> shutdown -- but it won't restart.  But I don't currently have alternate way
>>>> that will reliably shut it down.
>>>>
>>>> Any advice?
>>>>
>>>> --j
>>>>
>>>>
>>>
>>
>

Mime
View raw message