flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chandrashekhar Shaw <sha...@yahoo-inc.com>
Subject Issues with flume while trying to transfer big size files [~1GB per hour]
Date Tue, 24 Jul 2012 07:07:10 GMT

I am using flume for transferring logs from 20 agent boxes to 1 collector box. The file size
which I am trying to transfer is of about 1 GB per hour. So every hour my agents run to transfer
the log file generated during the last hour.
I am using DFO sink for file transfer.

But then I am facing following problems:

1) Some of the agents are not able to transfer the whole file even in one hour and the processes
run continuously for hours.
2) Some of the agents stop within 5-6 minutes without sending the file completely.

I want to know 

1) if somebody else has faced the same issues?
2) What could be the problem?

Chandrashekhar Shaw
View raw message