flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mahendran m <mahendra...@hotmail.com>
Subject RE: Perfromacnce issue in file channel
Date Tue, 11 Nov 2014 04:54:40 GMT
Hi Hari ,
Thanks , yes i am sending the event one by one only as it is my requirement but memory channel
can easily process 1000 per second but file channel doesn't is this case how can i improve
the performance of file channel .
Regards,Mahendran

Date: Mon, 10 Nov 2014 11:34:57 -0800
From: hshreedharan@cloudera.com
To: user@flume.apache.org
CC: user@flume.apache.org
Subject: Re: Perfromacnce issue in file channel


Sending one event by one event is likely to cause performance issues. In addition, I have
no idea of the performance on Windows. Btw, 100-200 request per second is really orders of
magnitude lower than what File Channel can handle.

Thanks,
Hari


On Sun, Nov 9, 2014 at 7:49 AM, mahendran m <mahendranec@hotmail.com> wrote:




Hi All ,
I have tested the file channel and file channel performance. below the code snippet to generate
1000 event per sec 




while (watch.Elapsed.TotalSeconds < sec)
{

	if (counter <= 1000)

	{

		uuid = uuid + 1;



		var text = HttpUtility.UrlEncode("HI , How are you", Encoding.UTF8);



		var flumeEvent = new ThriftFlumeEvent();

		flumeEvent.Headers = new Dictionary<string, string>() { { "Host", "http://www.bing.com/"
} };

		flumeEvent.Body = Encoding.ASCII.GetBytes(text);

		Client.append(flumeEvent);

		counter += 1;

	}
}



i succeed in sending the 1000 event per sec when using memory channel , but in case if i used
file channel  i can  only reach upto 40 to 50 events per second


config for memory channel

# Use memory channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 1000
a1.channels.c1.byteCapacityBufferPercentage = 10
a1.channels.c1.byteCapacity = 5368709120





config for file channel



#using the file channel
a1.channels.c1.type = file
a1.channels.c1.capacity = 500000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.useDualCheckpoints = truea1.channels.c1.backupCheckpointDir = F:/.checkPoint


Can anyone explain why file channel is this much channel ? . if i used file channel in server
for directly logging each request is it better option as my server receive 100 to 200 request
per second


Regards,
Mahendran




 		 	   		  

 		 	   		  
Mime
View raw message