I am using a File Channel connected to
an AMQP Source and an HDFS Sink. Events are coming in at a rate around
1500 msg/sec, and the AMQP source is batching them 1000 a shot. The
writing to the file channel seems to be keeping up well with this rate.
However, when the HDFS Sink, also batch size 1000, is trying to read out
of the channel it cannot even come close to keeping up that rate. I
haven't set up the data directory and checkpoint directory to write to
different disks yet, but I was hoping I was doing something obviously
wrong that would account for this behaviour. I have also played with
the batch size of the HDFS sink and that doesn't seem to make much of a
difference. I also realize that I can add additional sinks, but was more curious if people of experienced the same behavior I am seeing.