flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From IT CTO <goi....@gmail.com>
Subject Re: Flume source channel sink tunning
Date Thu, 08 Oct 2015 17:45:21 GMT
Each if the sinks will run in a different proccess?
Any way I can share the same sink configuration for both sinks?
Eran

בתאריך יום ה׳, 8 באוק׳ 2015, 19:50 מאת Roshan Naik <roshan@hortonworks.com>:

> 1 – Bump up the –Xmx in flume-env.sh as the default is quite small
> 2 – increase the capacity on the channel. Looks like your source is
> running much faster than the sink can keep up. You can try adding more
> sinks to improve drain rate.
>
> From: IT CTO <goi.cto@gmail.com>
> Reply-To: "user@flume.apache.org" <user@flume.apache.org>
> Date: Thursday, October 8, 2015 9:46 AM
> To: "user@flume.apache.org" <user@flume.apache.org>
> Subject: Flume source channel sink tunning
>
> Hi,
>
> I am using SpoolDir with memory channel to write to hdfs sink.
> When I use a single spoolDir I get single threaded performance so based on
> some mails I read I splinted the source to 5 spoolDir all writing to the
> same memory channel which writes to the hdfs.
>
> Now I am getting different errors
> 1) GC error => not engough memory for the channel => increase Xmx for the
> agent
> 2) Channel is full => the sink is not kipping up with the channel
>
> so I find myself playing with the different parameters.
> *any best practice here or path to follow to get it tuned?*
> I feel that even if it works it will easily break given other events
>
> Eran
> --
> Eran | "You don't need eyes to see, you need vision" (Faithless)
>
-- 
Eran | "You don't need eyes to see, you need vision" (Faithless)

Mime
View raw message