lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [lucenenet] NightOwl888 commented on pull request #310: WIP Benchmarks between nuget versions
Date Mon, 20 Jul 2020 12:35:02 GMT

NightOwl888 commented on pull request #310:
URL: https://github.com/apache/lucenenet/pull/310#issuecomment-661002167


   Thanks for putting this together.
   
   I suspect the reason you are not seeing an improvement between versions is because you
are using a `RAMDirectory`, which bypasses the codecs and some other components that we have
improved.
   
   I added benchmarks using the `IndexFiles` and `SearchFiles` from `Lucene.Net.Demo` as templates.
These are set up to use a fixed seed to generate the same data over and over again so we don't
have to add several megabytes of test files to the repo just to do a benchmark.
   
   Index speed doesn't seem to have improved, but search speed has doubled since 4.8.0-beta00005
(the one with the most downloads on NuGet.org). Here are the results for .NET Core 3.1 on
on Windows.
   
   ## Indexing
   
   ``` ini
   
   BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.900 (1909/November2018Update/19H2)
   Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
   .NET Core SDK=3.1.301
     [Host]          : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT  [AttachedDebugger]
     4.8.0-beta00005 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT
     4.8.0-beta00006 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT
     4.8.0-beta00007 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT
     4.8.0-beta00008 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT
     4.8.0-beta00009 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT
     4.8.0-beta00010 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT
   
   InvocationCount=1  IterationCount=15  LaunchCount=2  
   UnrollFactor=1  WarmupCount=10  
   
   ```
   |     Method |             Job |                            NuGetReferences |     Mean
|    Error |   StdDev |      Gen 0 |     Gen 1 |     Gen 2 | Allocated |
   |----------- |---------------- |------------------------------------------- |---------:|---------:|---------:|-----------:|----------:|----------:|----------:|
   | IndexFiles | 4.8.0-beta00005 | Lucene.Net.Analysis.Common 4.8.0-beta00005 | 734.9 ms
| 37.14 ms | 55.58 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.09 MB |
   | IndexFiles | 4.8.0-beta00006 | Lucene.Net.Analysis.Common 4.8.0-beta00006 | 703.7 ms
| 13.92 ms | 20.83 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.07 MB |
   | IndexFiles | 4.8.0-beta00007 | Lucene.Net.Analysis.Common 4.8.0-beta00007 | 705.3 ms
| 15.95 ms | 23.38 ms | 43000.0000 | 8000.0000 | 7000.0000 |    221 MB |
   | IndexFiles | 4.8.0-beta00008 | Lucene.Net.Analysis.Common 4.8.0-beta00008 | 747.0 ms
| 42.43 ms | 59.47 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.39 MB |
   | IndexFiles | 4.8.0-beta00009 | Lucene.Net.Analysis.Common 4.8.0-beta00009 | 725.3 ms
| 25.59 ms | 36.70 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.31 MB |
   | IndexFiles | 4.8.0-beta00010 | Lucene.Net.Analysis.Common 4.8.0-beta00010 | 724.1 ms
| 15.41 ms | 22.10 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.23 MB |
   
   ## Searching
   
   ``` ini
   
   BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.900 (1909/November2018Update/19H2)
   Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
   .NET Core SDK=3.1.301
     [Host]          : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT  [AttachedDebugger]
     4.8.0-beta00005 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT
     4.8.0-beta00006 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT
     4.8.0-beta00007 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT
     4.8.0-beta00008 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT
     4.8.0-beta00009 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT
     4.8.0-beta00010 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.27001), X64
RyuJIT
   
   IterationCount=100  LaunchCount=3  WarmupCount=15  
   
   ```
   |      Method |             Job |                                                     
             NuGetReferences |      Mean |    Error |    StdDev |    Median |     Gen 0 |
    Gen 1 | Gen 2 | Allocated |
   |------------ |---------------- |----------------------------------------------------------------------------------
|----------:|---------:|----------:|----------:|----------:|----------:|------:|----------:|
   | SearchFiles | 4.8.0-beta00005 | Lucene.Net.Analysis.Common 4.8.0-beta00005,Lucene.Net.QueryParser
4.8.0-beta00005 | 171.18 ms | 2.004 ms | 10.283 ms | 169.51 ms | 9000.0000 | 1000.0000 | 
   - |  41.35 MB |
   | SearchFiles | 4.8.0-beta00006 | Lucene.Net.Analysis.Common 4.8.0-beta00006,Lucene.Net.QueryParser
4.8.0-beta00006 | 169.86 ms | 2.666 ms | 13.754 ms | 168.14 ms | 9000.0000 | 1000.0000 | 
   - |  41.35 MB |
   | SearchFiles | 4.8.0-beta00007 | Lucene.Net.Analysis.Common 4.8.0-beta00007,Lucene.Net.QueryParser
4.8.0-beta00007 | 173.67 ms | 3.417 ms | 17.595 ms | 169.20 ms | 9000.0000 | 1000.0000 | 
   - |  41.26 MB |
   | SearchFiles | 4.8.0-beta00008 | Lucene.Net.Analysis.Common 4.8.0-beta00008,Lucene.Net.QueryParser
4.8.0-beta00008 |  95.97 ms | 1.178 ms |  6.068 ms |  95.84 ms | 8666.6667 |  500.0000 | 
   - |  40.33 MB |
   | SearchFiles | 4.8.0-beta00009 | Lucene.Net.Analysis.Common 4.8.0-beta00009,Lucene.Net.QueryParser
4.8.0-beta00009 |  93.31 ms | 0.776 ms |  3.890 ms |  93.60 ms | 8666.6667 |  500.0000 | 
   - |  40.33 MB |
   | SearchFiles | 4.8.0-beta00010 | Lucene.Net.Analysis.Common 4.8.0-beta00010,Lucene.Net.QueryParser
4.8.0-beta00010 |  85.79 ms | 0.307 ms |  1.552 ms |  85.70 ms | 8666.6667 |  500.0000 | 
   - |  40.19 MB |
   
   
   It would be good to set up a stage on Azure DevOps to run these (off by default) so we
can see the results on different operating systems and target frameworks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message