flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From IT CTO <goi....@gmail.com>
Subject Flume source channel sink tunning
Date Thu, 08 Oct 2015 16:46:36 GMT

I am using SpoolDir with memory channel to write to hdfs sink.
When I use a single spoolDir I get single threaded performance so based on
some mails I read I splinted the source to 5 spoolDir all writing to the
same memory channel which writes to the hdfs.

Now I am getting different errors
1) GC error => not engough memory for the channel => increase Xmx for the
2) Channel is full => the sink is not kipping up with the channel

so I find myself playing with the different parameters.
*any best practice here or path to follow to get it tuned?*
I feel that even if it works it will easily break given other events

Eran | "You don't need eyes to see, you need vision" (Faithless)

View raw message