flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juhani Connolly <juhani_conno...@cyberagent.co.jp>
Subject Re: File Channel performance and fsync
Date Tue, 23 Oct 2012 07:08:23 GMT
Without the fsync guarrantees are weakened a lot more than the fsync 
lying case.

Also, you didn't mention the batch size on your avro sink that is 
sending data to the avro-source. This is a major factor on your 
throughput because each batch causes one sync. If you have big batches, 
you'll have few fsyncs and significantly better performance.

I am weirded out by the fact that Danny is getting improved performance 
by running multiple parallel file sinks... Are they each on separate 
disks or something? I can't imagine what could cause a performance gain 
if they were all on the same disk. Would likely expect more write head 
skipping around and degradation even...

On 10/23/2012 03:31 PM, Jagadish Bihani wrote:
> Hi Denny
>
> Thanks for the inputs.
> Btw when you say you tested another case without 'fsync'; I think
> you changed the file channel code to comment out 'flush' part of it.
> And if we rely on OS flushing then still it can be reasonably reliable.
> Is that right?
>
> Regards,
> Jagadish
>
> On 10/22/2012 07:08 PM, Denny Ye wrote:
>> hi Jagadish,
>>    I have tested performance of FileChannel recently. Here I can 
>> support the test report to you for your thinking and questions at 
>> this thread.
>>     Talking about the comparison between FileChannel and File Sink. 
>> FileChannel supports both sequential writer and random reader, there 
>> have so many times shift of magnetic head, it's slow than the 
>> sequential writing much more.
>>     'fsync' command has consuming much time than writing, almost 
>> 100times/sec, same as number mentioned from Brock. Also, I didn't 
>> know why there have such difference between your two servers. I think 
>> it might be related with OS version (usage between fsync and 
>> fdatasync instruction) or disk driver (RAID, caching strategy, and so 
>> on).
>>     Throughput of single FileChannel is almost 3-5MB/sec in my 
>> environment. Thus I used 5 channels with 18MB/sec. It's hard to 
>> believe the linear increasing with more channels. Meanwhile, it look 
>> like the limit of throughput with 'fsync' operation. I tested another 
>> case without 'fsync' operation after each batch, almost 
>> 35-40MB/sec(Also, I removed the pre-allocation at disk writing in 
>> this case).
>>     Hope useful for you.
>>
>>    PS : I heard that OS has demon thread to flush page cache to 
>> disk asynchronously with second latency, does it's effective for 
>> amount of data with tolerant loss?
>>
>> -Regards
>> Denny Ye
>>
>> 2012/10/22 Jagadish Bihani <jagadish.bihani@pubmatic.com 
>> <mailto:jagadish.bihani@pubmatic.com>>
>>
>>     Hi
>>
>>     I am writing this on top of another thread where there was
>>     discussion on "fsync lies" and
>>     only file channel used fsync and not file sink. :
>>
>>     -- I tested the fsync performance on 2 machines  (On 1 machine I
>>     was getting very good throughput
>>     using file channel and on another almost 100 times slower with
>>     almost same hardware configuration.)
>>     using following code
>>
>>
>>     #define PAGESIZE 4096
>>
>>     int main(int argc, char *argv[])
>>     {
>>
>>             char my_write_str[PAGESIZE];
>>             char my_read_str[PAGESIZE];
>>             char *read_filename= argv[1];
>>             int readfd,writefd;
>>
>>             readfd = open(read_filename,O_RDONLY);
>>             writefd = open("written_file",O_WRONLY|O_CREAT,777);
>>             int len=lseek(readfd,0,2);
>>             lseek(readfd,0,0);
>>             int iterations = len/PAGESIZE;
>>             int i;
>>             struct timeval t0,t1;
>>
>>            for(i=0;i<iterations;i++)
>>             {
>>
>>                     read(readfd,my_read_str,PAGESIZE);
>>                     write(writefd,my_read_str,PAGESIZE);
>>     *gettimeofday(&t0,0);**
>>     **                fsync(writefd);**
>>     **              gettimeofday(&t1,0);*
>>                     long elapsed = (t1.tv_sec-t0.tv_sec)*1000000 +
>>     t1.tv_usec-t0.tv_usec;
>>                     printf("Elapsed time is= %ld \n",elapsed);
>>              }
>>             close(readfd);
>>             close(writefd);
>>     }
>>
>>
>>     -- As expected it requires typically 50000 microseconds for fsync
>>     to complete on one machine and 200 microseconds
>>     on another machine it took 290 microseconds to complete on an
>>     average. So is machine with higher
>>     performance is doing a 'fsync lie'?
>>     i
>>     -- If I have understood it clearly; "fsync lie" means the data is
>>     not actually written to disk and it is in
>>     some disk/controller buffer.  I) Now if disk loses power due to
>>     some shutdown or any other disaster, data will
>>     be lost. II) Can data be lost even without it ? (e.g. if it is
>>     keeping data in some disk buffer and if fsync is being
>>     invoked continuously then will that data can also  be lost? If
>>     only part -I is true; then it can be acceptable
>>     because probability of shutdown is usually less in production
>>     environment. But if even II is true then there is a
>>     problem.
>>
>>     -- But on the machine where disk doesn't lie performance of flume
>>     using File channel is very low (I have seen it
>>     maximum 100 KB/sec even with sufficient  DirectMemory
>>     allocation.) Does anybody have stats about throughput
>>     of file channel ? Is anybody getting better performance with file
>>     channel (without fsync lies). What is the recommended
>>     usage of it for an average scenario ? (Transferring files of few
>>     MBs to HDFS sink continuously on typical hardware
>>     (16 core processors, 16 GB RAM etc.)
>>
>>
>>     Regards,
>>     Jagadish
>>
>>     On 10/10/2012 11:30 PM, Brock Noland wrote:
>>>     Hi,
>>>
>>>     On Wed, Oct 10, 2012 at 11:22 AM, Jagadish Bihani
>>>     <jagadish.bihani@pubmatic.com>  <mailto:jagadish.bihani@pubmatic.com>
 wrote:
>>>>     Hi Brock
>>>>
>>>>     I will surely look into 'fsync lies'.
>>>>
>>>>     But as per my experiments I think "file channel" is causing the issue.
>>>>     Because on those 2 machines (one with higher throughput and other with
>>>>     lower)
>>>>     I did following experiment:
>>>>
>>>>     cat Source -memory channel - file sink.
>>>>
>>>>     Now with this setup I got same throughput on both the machines. (around
3
>>>>     MB/sec)
>>>>     Now as I have used "File sink" it should also do "fsync" at some point
of
>>>>     time.
>>>>     'File Sink' and 'File Channel' both do disk writes.
>>>>     So if there is differences in disk behaviour then even in the 'File Sink'
it
>>>>     should be visible.
>>>>
>>>>     Am I missing something here?
>>>     File sink does not call fsync.
>>>
>>>>     Regards,
>>>>     Jagadish
>>>>
>>>>
>>>>
>>>>     On 10/10/2012 09:35 PM, Brock Noland wrote:
>>>>>     OK your disk that is giving you 40KB/second is telling you the truth
>>>>>     and the faster disk is lying to you. Look up "fsync lies" to see
what
>>>>>     I am referring to.
>>>>>
>>>>>     A spinning disk can do 100 fsync operations per second (this is done
>>>>>     at the end of every batch). That is how I estimated your event size,
>>>>>     40KB/second is doing 40KB / 100 =  409 bytes.
>>>>>
>>>>>     Once again, if you want increased performance, you should increase
the
>>>>>     batch size.
>>>>>
>>>>>     Brock
>>>>>
>>>>>     On Wed, Oct 10, 2012 at 11:00 AM, Jagadish Bihani
>>>>>     <jagadish.bihani@pubmatic.com>  <mailto:jagadish.bihani@pubmatic.com>
 wrote:
>>>>>>     Hi
>>>>>>
>>>>>>     Yes. It is around 480 - 500 bytes.
>>>>>>
>>>>>>
>>>>>>     On 10/10/2012 09:24 PM, Brock Noland wrote:
>>>>>>>     How big are your events? Average about 400 bytes?
>>>>>>>
>>>>>>>     Brock
>>>>>>>
>>>>>>>     On Wed, Oct 10, 2012 at 5:11 AM, Jagadish Bihani
>>>>>>>     <jagadish.bihani@pubmatic.com>  <mailto:jagadish.bihani@pubmatic.com>
 wrote:
>>>>>>>>     Hi
>>>>>>>>
>>>>>>>>     Thanks for the inputs Brock. After doing several experiments
>>>>>>>>     eventually problem boiled down to disks.
>>>>>>>>
>>>>>>>>         -- But I had used the same configuration (so all
software components
>>>>>>>>     are
>>>>>>>>     same in all 3 machines)
>>>>>>>>     on all 3 machines.
>>>>>>>>     -- In User guide it is written that if multiple file
channel instances
>>>>>>>>     are
>>>>>>>>     active on the same agent then
>>>>>>>>     different disks are preferable. But in my case only one
file channel is
>>>>>>>>     active per agent.
>>>>>>>>     -- Only one pattern I observed that on the machines where
I got better
>>>>>>>>     performance have multiple disks.
>>>>>>>>     But I don't understand how that will help if I have only
1 active file
>>>>>>>>     channel.
>>>>>>>>     -- What is the impact of the type of disk/disk device
driver on
>>>>>>>>     performance?
>>>>>>>>     I mean I don't understand
>>>>>>>>     with 1 disk I am getting 40 KB/sec and with other 2 MB/sec.
>>>>>>>>
>>>>>>>>     Could you please elaborate on File channel and disks
correlation.
>>>>>>>>
>>>>>>>>     Regards,
>>>>>>>>     Jagadish
>>>>>>>>
>>>>>>>>
>>>>>>>>     On 10/09/2012 08:01 PM, Brock Noland wrote:
>>>>>>>>
>>>>>>>>     Hi,
>>>>>>>>
>>>>>>>>     Using file channel, in terms of performance, the number
and type of
>>>>>>>>     disks is going to be much more predictive of performance
than CPU or
>>>>>>>>     RAM. Note that consumer level drives/controllers will
give you much
>>>>>>>>     "better" performance because they lie to you about when
your data is
>>>>>>>>     actually written to the drive. If you search for "fsync
lies" you'll
>>>>>>>>     find more information on this.
>>>>>>>>
>>>>>>>>     You probably want to increase the batch size to get better
performance.
>>>>>>>>
>>>>>>>>     Brock
>>>>>>>>
>>>>>>>>     On Tue, Oct 9, 2012 at 2:46 AM, Jagadish Bihani
>>>>>>>>     <jagadish.bihani@pubmatic.com>  <mailto:jagadish.bihani@pubmatic.com>
 wrote:
>>>>>>>>
>>>>>>>>     Hi
>>>>>>>>
>>>>>>>>     My flume setup is:
>>>>>>>>
>>>>>>>>     Source Agent : cat source - File Channel - Avro Sink
>>>>>>>>     Dest Agent :     avro source - File Channel - HDFS Sink.
>>>>>>>>
>>>>>>>>     There is only 1 source agent and 1 destination agent.
>>>>>>>>
>>>>>>>>     I measure throughput as amount of data written to HDFS
per second.
>>>>>>>>     ( I have rolling interval 30 sec; so If 60 MB file is
generated in 30
>>>>>>>>     sec
>>>>>>>>     the
>>>>>>>>     throughput is : -- 2 MB/sec ).
>>>>>>>>
>>>>>>>>     I have run source agent on various machines with different
hardware
>>>>>>>>     configurations :
>>>>>>>>     (In all cases I run flume agent with JAVA OPTIONS as
>>>>>>>>     "-DJAVA_OPTS="-Xms500m -Xmx1g -Dcom.sun.management.jmxremote
>>>>>>>>     -XX:MaxDirectMemorySize=2g")
>>>>>>>>
>>>>>>>>     JDK is 32 bit.
>>>>>>>>
>>>>>>>>     Experiment 1:
>>>>>>>>     =====
>>>>>>>>     RAM : 16 GB
>>>>>>>>     Processor: Intel Xeon E5620 @ 2.40 GHz (16 cores).
>>>>>>>>     64 bit Processor with 64 bit Kernel.
>>>>>>>>     Throughput: 2 MB/sec
>>>>>>>>
>>>>>>>>     Experiment 2:
>>>>>>>>     ======
>>>>>>>>     RAM : 4 GB
>>>>>>>>     Processor: Intel Xeon E5504  @ 2.00GHz (4 cores). 32
bit Processor
>>>>>>>>     64 bit Processor with 32 bit Kernel.
>>>>>>>>     Throughput : 30 KB/sec
>>>>>>>>
>>>>>>>>     Experiment 3:
>>>>>>>>     ======
>>>>>>>>     RAM : 8 GB
>>>>>>>>     Processor:Intel Xeon E5520 @ 2.27 GHz (16 cores).32 bit
Processor
>>>>>>>>     64 bit Processor with 32 bit Kernel.
>>>>>>>>     Throughput : 80 KB/sec
>>>>>>>>
>>>>>>>>         -- So as can be seen there is huge difference in
the throughput with
>>>>>>>>     same
>>>>>>>>     configuration but
>>>>>>>>     different hardware.
>>>>>>>>     -- In the first case where throughput is more RES is
around 160 MB in
>>>>>>>>     other
>>>>>>>>     cases it is in
>>>>>>>>     the range of 40 MB - 50 MB.
>>>>>>>>
>>>>>>>>     Can anybody please give insights that why there is this
huge difference
>>>>>>>>     in
>>>>>>>>     the throughput?
>>>>>>>>     What is the correlation between RAM and filechannel/HDFS
sink
>>>>>>>>     performance
>>>>>>>>     and also
>>>>>>>>     with 32-bit/64 bit kernel?
>>>>>>>>
>>>>>>>>     Regards,
>>>>>>>>     Jagadish
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>
>>
>


Mime
View raw message