flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cameron Gandevia <cgande...@gmail.com>
Subject File channel configuration
Date Wed, 31 Oct 2012 03:08:45 GMT

I'm trying to figure out the best way to configure the file channel for
maximum throughput and have a couple of questions.

1. What is the best hard disk layout? An ssd for the checkpoint directory
and a separate disk for each file channel on the agent?

2. Can multiple discs be utilized for a single channel? I could only seem
to configure a single data directory.

3. There is a comment in the documentation that mentions adding more sinks
to drain the channel faster. If my final agent sink was hdfs does that mean
configuring two hdfs sinks using a sink group to drain a single channel on
an agent? I noticed you can configure thread pools on the hdfs sink but
haven't looked into it.

4. Does it make sense to have my agent run two channels both with sinks
writing to a single hdfs cluster each configured with q separate data disk
and have the previous agent round robin deliver to it?

Thanks for any input anyone has

View raw message