flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander C.H. Lorenz" <wget.n...@googlemail.com>
Subject Re: Configuring many agents makes heartbeats down
Date Thu, 17 Nov 2011 08:39:59 GMT
are all agents on one server? Could be hit the fs.file-max in sysctl.
Please check:

1. In /etc/security/limits.conf
* soft nofile 1024
* hard nofile 65535
2. Increase ulimit by "ulimit -n 65535"
3. echo 65535 > /proc/sys/fs/file-max
4. In /etc/sysctl.conf
fs.file-max=65535


best,
 Alex

2011/11/17 Mingjie Lai <mjlai09@gmail.com>

> Hi.
>
> > (1) How many agents are assured ?
>
> Tell the truth, I'm not sure how many people ever tried so many logical
> nodes for one physical node. :(
>
> > 2011-11-16 11:38:00.687 WARN com.cloudera.flume.agent.LivenessManager:
> > Heartbeats are backing up, currently behind by 189 heartbeats
>
> Did you see it with one logical node? It could be caused by other issue,
> such as https://issues.apache.org/jira/browse/Flume-808
>
>
> On 11/16/2011 09:45 AM, Y. Sakamoto wrote:
> > Hi,
> > I tried to configure 53 agents (logical agents, on 1 physical server) by
> > "flume shell" all at one.
> > After this, heartbeats failed and following message was written in the
> > collector log :
> >
> > 2011-11-16 11:38:00.687 WARN com.cloudera.flume.agent.LivenessManager:
> > Heartbeats are backing up, currently behind by 189 heartbeats
> >
> >
> > I wonder that the interval of heartbeats is too short to process many
> > agents.
> > So I have questions,
> >
> > (1) How many agents are assured ?
> >
> > (2) If my assumption is correct, which approach is better ?
> >       a. Wait until one agent configuration is finished,
> >           then next configuration continue.
> >       b. Change the interval of heartbeats longer (about 1 minutes) .
> >
> >
> > Thanks,
> >    Y. Sakamoto.
> >
> >
>



-- 
Alexander Lorenz
http://mapredit.blogspot.com

*P **Think of the environment: please don't print this email unless you
really need to.*

Mime
View raw message