flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan Naik <ros...@hortonworks.com>
Subject Re: Flume source channel sink tunning
Date Thu, 08 Oct 2015 16:50:05 GMT
1 - Bump up the -Xmx in flume-env.sh as the default is quite small
2 - increase the capacity on the channel. Looks like your source is running much faster than
the sink can keep up. You can try adding more sinks to improve drain rate.

From: IT CTO <goi.cto@gmail.com<mailto:goi.cto@gmail.com>>
Reply-To: "user@flume.apache.org<mailto:user@flume.apache.org>" <user@flume.apache.org<mailto:user@flume.apache.org>>
Date: Thursday, October 8, 2015 9:46 AM
To: "user@flume.apache.org<mailto:user@flume.apache.org>" <user@flume.apache.org<mailto:user@flume.apache.org>>
Subject: Flume source channel sink tunning


I am using SpoolDir with memory channel to write to hdfs sink.
When I use a single spoolDir I get single threaded performance so based on some mails I read
I splinted the source to 5 spoolDir all writing to the same memory channel which writes to
the hdfs.

Now I am getting different errors
1) GC error => not engough memory for the channel => increase Xmx for the agent
2) Channel is full => the sink is not kipping up with the channel

so I find myself playing with the different parameters.
any best practice here or path to follow to get it tuned?
I feel that even if it works it will easily break given other events

Eran | "You don't need eyes to see, you need vision" (Faithless)

View raw message