flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject Re: sync issue between collector and agent in distribute mode
Date Mon, 01 Aug 2011 17:45:18 GMT
[NOTE, moved to flume-user@incubator.apache.org]


I think it is a better practice currently to set all the configurations at
the master with hardcoded ips and then to bring up the nodes.

The reconfiguration portion in the nodes have been having problems. The
usability of the logical source/sink depends on reconfiguration being
reliable and this is shortcoming in the current implementation.  We know
about this issue and are working of fixing this.

Here's a link to the status of the features currently:


On Mon, Jul 25, 2011 at 8:09 PM, Junxian Yan <junxian.yan@gmail.com> wrote:

> Hi Guys
> I have a complex flume environmnet.
> I took one machine as master and created all logical collectors on this
> machine. And created one logical agent on each other machines(about ten
> machines).
> And the flume configuration is dynamic. When each agent machine launched,
> there was a bunch of scripts will connect to master and write configuration
> by some of local environment variable, the config script look like below, I
> remove some production information from configure file
>        connect MASTER:35873
>         exec map MASTER ip-10-243-118-208_collector
>         exec map MASTER ip-10-243-118-208_error_collector
>         exec map ip-10-243-118-208 ip-10-243-118-208_agent
>         exec map ip-10-243-118-208 ip-10-243-118-208_error_agent
>          exec config ip-10-243-118-208_collector logicalSource
> 'collectorSink("s3n://log/%Y-%m-%d/","log-%Y%m%d-ip-10-243-118-208.")'
>          exec config ip-10-243-118-208_error_collector logicalSource
> 'collectorSink("s3n://error/%Y-%m-%d/","error-%Y%m%d-ip-10-243-118-208.")'
>         waitForNodesActive 30000 ip-10-243-118-208_collector
> ip-10-243-118-208_error_collector
>         exec config ip-10-243-118-208_agent
> 'tailDir("PATH",startFromEnd=true)'
> 'logicalSink("ip-10-243-118-208_collector")'
>         exec config ip-10-243-118-208_error_agent
> 'tailDir("PATH",startFromEnd=true)'
> 'logicalSink("ip-10-243-118-208_error_collector")'
> Now my problem is: the agent can not be configured everytime, and if I
> reboot this machine, the agent will be configured correctly. I went through
> the command executing history, found the last two lines is not executed. I
> think the waitFor function should not block the following steps, is that
> true?
> And when I remove the waitFor steps. there will be some agent configuration
> shown as ERROR state. But in the command executing history, all commands are
> executed successfully
> R

// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com

View raw message