flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From iain wright <iainw...@gmail.com>
Subject Re: Dynamic sink configuration
Date Wed, 29 Oct 2014 01:09:15 GMT
not sure if its the same for embedded agent but our standalone flume
daemons watch their config for changes and do a graceful reload when
changes are detected (without dropping events in the channel)

we use ansible to push changes to our sink config on upstream tiers when we
add nodes to our downstream flume aggregate tiers

should be easy to test by touching your flume config and seeing if its
re-initializes and spits out info to log with startup info

Iain Wright

This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.

On Tue, Oct 28, 2014 at 4:54 PM, Pavel Zalunin <wr4bbit@gmail.com> wrote:

> Hi,
> We have embedded agent, zookeeper is huge overhead.
> Pavel.
> 2014-10-28 20:11 GMT+03:00 Paweł Róg <prog88@gmail.com>:
>> Hi,
>> And as far as I know flume can be configured by zookeeper :D
>> https://issues.apache.org/jira/browse/FLUME-1491
>> --
>> Pawel Rog
>> 2014-10-28 17:02 GMT+01:00 Otis Gospodnetic <otis.gospodnetic@gmail.com>:
>>> As far as I know, Flume is configured only through the config and I
>>> *think* Flume automatically reloads the config when it detects the config
>>> changed - at least that is what Ashish wrote here:
>>> http://search-hadoop.com/m/DEeB4w0kHg1
>>> Otis
>>> --
>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>> Solr & Elasticsearch Support * http://sematext.com/
>>> On Mon, Oct 27, 2014 at 7:42 PM, Pavel Zalunin <wr4bbit@gmail.com>
>>> wrote:
>>>> Hi,
>>>> We have embedded flume agent and we need to update some parameters
>>>> dynamically after sink/source runners are started.
>>>> For example, having sink which sends events to some url. At some point
>>>> we need to change this url (add request parameter for example).
>>>> Straightforward way is to add method which will accept changes and
>>>> update internal state, but it looks like a 'hack', because all flume sinks,
>>>> that I've seen configured just once.
>>>> Maybe there are another, 'flume-way' to do such things?
>>>> Is restarting sink (I think restart other related things also needed)
>>>> only one option?
>>>> Thanks,
>>>> Pavel.

View raw message