lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shad Storhaug (Jira)" <j...@apache.org>
Subject [jira] [Commented] (LUCENENET-640) Sequential IndexWriter performance in concurrent environments.
Date Fri, 07 Feb 2020 20:40:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENENET-640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17032641#comment-17032641
] 

Shad Storhaug commented on LUCENENET-640:
-----------------------------------------

Elad,

Well, this particular issue has been addressed by swapping ConditionalWeakTable for WeakIdentityMap.
However, we are still using WeakDictionary in several places, most notably in the FieldCacheImpl
(see [LUCENENET-610|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-610]).
This is due to the lack of enumerator support before .NET Standard 2.1 primarily.

That said, the article that describes the problem specifically mentions it is [a problem with
Lucene.NET|https://ayende.com/blog/189761-A/production-postmortem-the-slow-slowdown-of-large-systems].
Thanks for the link. Hopefully, he is planning to submit the fix back to us after he thoroughly
tests it.


> Sequential IndexWriter performance in concurrent environments.
> --------------------------------------------------------------
>
>                 Key: LUCENENET-640
>                 URL: https://issues.apache.org/jira/browse/LUCENENET-640
>             Project: Lucene.Net
>          Issue Type: Bug
>          Components: Lucene.Net Core
>    Affects Versions: Lucene.Net 4.8.0
>            Reporter: Mathias Henriksen
>            Priority: Major
>              Labels: performance
>             Fix For: Lucene.Net 4.8.0
>
>         Attachments: AssertFinalBug.jpg, IdentityWeakReferenceBug.jpg, Program.cs, overviewBug.jpg
>
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> When creating Lucene.Net indices in parallel, sequential-like performance is experienced.
Profiling 8 concurrent IndexWriter instances writing in parallel shows that WeakIdentityMap::IdentityWeakReference::Equals
spends most time garbage collecting (94.91%) and TokenStream::AssertFinal (87.09% garbage
collecting) in my preliminary tests (see screenshots).
> The [WeakIdentityMap|https://github.com/apache/lucenenet/blob/master/src/Lucene.Net/Util/WeakIdentityMap.cs]
implementation uses an IdentityWeakReference as key, which is implemented as a class. By inspection
of this class, it is merely a System.Runtime.InteropServices.GCHandle wrapper as can be seen
in the mono project, manually wrapping of this struct in a struct rather than a class - will
eliminate some of the immense amounts of garbage collection.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message