lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Itamar Syn-Hershko <ita...@code972.com>
Subject Re: Bug in Lucene static initialization with multiple threads.
Date Tue, 07 Feb 2017 17:25:13 GMT
Hi Vincent, what version of Lucene.NET are you using?

--

Itamar Syn-Hershko
Freelance Developer & Consultant
Elasticsearch Consulting Partner
Microsoft MVP | Lucene.NET PMC
http://code972.com | @synhershko <https://twitter.com/synhershko>
http://BigDataBoutique.co.il/

On Tue, Feb 7, 2017 at 1:00 PM, Van Den Berghe, Vincent <
Vincent.VanDenBerghe@bvdinfo.com> wrote:

> Hello,
>
> Every once in a while, I get an error when using Lucene in a multithreaded
> scenario (meaning: using a single IndexWriter in multiple threads, or using
> a distinct IndexWriter in each thread: it doesn't matter).
> The exception chain thrown is:
>
> Unhandled Exception: System.ArgumentException: Could not instantiate
> implementing class for Lucene.Net.Analysis.Tokenattributes.
> ICharTermAttribute
> ---> System.ArgumentException: Could not find implementing class for
> ICharTermAttribute
> --->System.InvalidOperationException: Collection was modified;
> enumeration operation  may not execute.
>
> I could not understand what was going on, especially because it only
> occurred "sometimes". It took me a while to figure out, but I think it's a
> bug.
>
> Here's the stack trace of the exception when it occurs:
>
>                 [External Code]
> >             Lucene.Net.dll!Lucene.Net.Support.HashMap<Lucene.Net.
> Support.WeakDictionary<System.Type, System.WeakReference>.WeakKey<System.Type>,
> System.WeakReference>.GetEnumerator() Line 229           C#
>                [External Code]
>                Lucene.Net.dll!Lucene.Net.Support.WeakDictionary<System.Type,
> System.WeakReference>.Clean() Line 59           C#
>                Lucene.Net.dll!Lucene.Net.Support.WeakDictionary<System.Type,
> System.WeakReference>.CleanIfNeeded() Line 71         C#
>                Lucene.Net.dll!Lucene.Net.Support.WeakDictionary<System.Type,
> System.WeakReference>.Add(System.Type key, System.WeakReference value)
> Line 134           C#
>                 Lucene.Net.dll!Lucene.Net.Util.AttributeSource.
> AttributeFactory.DefaultAttributeFactory.GetClassForInterface<Lucene.
> Net.Analysis.Tokenattributes.ICharTermAttribute>() Line 90  C#
>                 Lucene.Net.dll!Lucene.Net.Util.AttributeSource.
> AttributeFactory.DefaultAttributeFactory.CreateAttributeInstance<
> Lucene.Net.Analysis.Tokenattributes.ICharTermAttribute>() Line 70  C#
>                 Lucene.Net.dll!Lucene.Net.Util.AttributeSource.
> AddAttribute<Lucene.Net.Analysis.Tokenattributes.ICharTermAttribute>()
> Line 350                C#
>                Lucene.Net.dll!Lucene.Net.Documents.Field.
> StringTokenStream.InitializeInstanceFields() Line 658         C#
>                Lucene.Net.dll!Lucene.Net.Documents.Field.
> StringTokenStream.StringTokenStream() Line 676                C#
>                Lucene.Net.dll!Lucene.Net.Documents.Field.
> GetTokenStream(Lucene.Net.Analysis.Analyzer analyzer) Line 629         C#
>                Lucene.Net.dll!Lucene.Net.Index.DocInverterPerField.
> ProcessFields(Lucene.Net.Index.IndexableField[] fields, int count) Line
> 105              C#
>                 Lucene.Net.dll!Lucene.Net.Index.DocFieldProcessor.
> ProcessDocument(Lucene.Net.Index.FieldInfos.Builder fieldInfos) Line 279
>         C#
>                 Lucene.Net.dll!Lucene.Net.Index.DocumentsWriterPerThread.
> UpdateDocument(System.Collections.Generic.IEnumerable<Lucene.Net.Index.IndexableField>
> doc, Lucene.Net.Analysis.Analyzer analyzer, Lucene.Net.Index.Term delTerm)
> Line 287                C#
>                 Lucene.Net.dll!Lucene.Net.Index.DocumentsWriter.
> UpdateDocument(System.Collections.Generic.IEnumerable<Lucene.Net.Index.IndexableField>
> doc, Lucene.Net.Analysis.Analyzer analyzer, Lucene.Net.Index.Term delTerm)
> Line 574                C#
>                Lucene.Net.dll!Lucene.Net.Index.IndexWriter.
> UpdateDocument(Lucene.Net.Index.Term term, System.Collections.Generic.
> IEnumerable<Lucene.Net.Index.IndexableField> doc,
> Lucene.Net.Analysis.Analyzer analyzer) Line 1830          C#
>                 Lucene.Net.dll!Lucene.Net.Index.IndexWriter.AddDocument(
> System.Collections.Generic.IEnumerable<Lucene.Net.Index.IndexableField>
> doc, Lucene.Net.Analysis.Analyzer analyzer) Line 1455   C#
>                 Lucene.Net.dll!Lucene.Net.Index.IndexWriter.AddDocument(
> System.Collections.Generic.IEnumerable<Lucene.Net.Index.IndexableField>
> doc) Line 1436   C#
>
> ... and to wit, here are the threads just rushing in to do the same:
>
> Not Flagged                        35428    17           Worker Thread <No
> Name>                Lucene.Net.dll!Lucene.Net.
> Support.WeakDictionary<System.Type, System.WeakReference>.Clean
>       Normal
> Not Flagged                        35444    11           Worker Thread <No
> Name>                Lucene.Net.dll!Lucene.Net.
> Support.WeakDictionary<System.Type, System.WeakReference>.Clean
>       Normal
> Not Flagged                        44124    12           Worker Thread <No
> Name>                Lucene.Net.dll!Lucene.Net.
> Support.WeakDictionary<System.Type, System.WeakReference>.Clean
>       Normal
> Not Flagged        >             44140    13           Worker Thread <No
> Name>                Lucene.Net.dll!Lucene.Net.
> Support.WeakDictionary<System.Type, System.WeakReference>.Clean
>       Normal
> Not Flagged                        47700    14           Worker Thread <No
> Name>                Lucene.Net.dll!Lucene.Net.
> Support.WeakDictionary<System.Type, System.WeakReference>.Clean
>       Normal
> Not Flagged                        28168    15           Worker Thread <No
> Name>                Lucene.Net.dll!Lucene.Net.
> Support.WeakDictionary<System.Type, System.WeakReference>.Clean
>       Normal
> Not Flagged                        30988    16           Worker Thread <No
> Name>                Lucene.Net.dll!Lucene.Net.
> Support.WeakDictionary<System.Type, System.WeakReference>.Clean
>       Normal
> Not Flagged                        21828    6              Worker Thread
> <No Name>                Lucene.Net.dll!Lucene.Net.
> Support.WeakDictionary<System.Type, System.WeakReference>.Clean
>       Normal
>
> The reason why it only reproduces "sometimes" is because of this little
> nugget of code:
>
>         private void CleanIfNeeded()
>         {
>             int currentColCount = GC.CollectionCount(0);
>             if (currentColCount > _gcCollections)
>             {
>                 Clean();
>                 _gcCollections = currentColCount;
>             }
>         }
>
> If one thread does a Clean() operation in the middle of another Clean()
> operation on the same collection that replaces the object being enumerated
> on, you get the exception. Always.
> To avoid the intermittence, create a bunch of threads like this and
> eliminate the test "if (currentColCount > _gcCollections)" so that the
> Clean() code is always executed. You'll get the exception every time.
>
> I will not post the correction, but there's a simple workaround: just make
> sure the static initializers are performed in a single thread.
> I.e. before creating your threads, do something like this:
>
> new global::Lucene.Net.Documents.TextField("dummy", "dummyvalue", global::
> Lucene.Net.Documents.Field.Store.NO).GetTokenStream(new (some Analyzer
> object));
>
> Replace "some Analyzer object" with an instance of an Analyzer object, it
> doesn't matter which one. It's meaningless, but it has the side effect of
> initializing the static fields without problems.
>
>
> Vincent
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message