flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan Naik <ros...@hortonworks.com>
Subject Re: Flume source channel sink tunning
Date Thu, 08 Oct 2015 19:12:20 GMT
You can connect multiple sinks to same memory channel. Each sink (and source) will get a thread
within the same flume agent process.
Each sink should differ in the destination file it writes to.
Configure each sink separately in the config file and set its  ...channel =
Property to the same channel.

-roshan


From: IT CTO <goi.cto@gmail.com<mailto:goi.cto@gmail.com>>
Reply-To: "user@flume.apache.org<mailto:user@flume.apache.org>" <user@flume.apache.org<mailto:user@flume.apache.org>>
Date: Thursday, October 8, 2015 10:45 AM
To: "user@flume.apache.org<mailto:user@flume.apache.org>" <user@flume.apache.org<mailto:user@flume.apache.org>>
Subject: Re: Flume source channel sink tunning


Each if the sinks will run in a different proccess?
Any way I can share the same sink configuration for both sinks?
Eran

בתאריך יום ה׳, 8 באוק׳ 2015, 19:50 מאת Roshan Naik <roshan@hortonworks.com<mailto:roshan@hortonworks.com>>:
1 – Bump up the –Xmx in flume-env.sh as the default is quite small
2 – increase the capacity on the channel. Looks like your source is running much faster
than the sink can keep up. You can try adding more sinks to improve drain rate.

From: IT CTO <goi.cto@gmail.com<mailto:goi.cto@gmail.com>>
Reply-To: "user@flume.apache.org<mailto:user@flume.apache.org>" <user@flume.apache.org<mailto:user@flume.apache.org>>
Date: Thursday, October 8, 2015 9:46 AM
To: "user@flume.apache.org<mailto:user@flume.apache.org>" <user@flume.apache.org<mailto:user@flume.apache.org>>
Subject: Flume source channel sink tunning

Hi,

I am using SpoolDir with memory channel to write to hdfs sink.
When I use a single spoolDir I get single threaded performance so based on some mails I read
I splinted the source to 5 spoolDir all writing to the same memory channel which writes to
the hdfs.

Now I am getting different errors
1) GC error => not engough memory for the channel => increase Xmx for the agent
2) Channel is full => the sink is not kipping up with the channel

so I find myself playing with the different parameters.
any best practice here or path to follow to get it tuned?
I feel that even if it works it will easily break given other events

Eran
--
Eran | "You don't need eyes to see, you need vision" (Faithless)
--
Eran | "You don't need eyes to see, you need vision" (Faithless)
Mime
View raw message