lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Digy (JIRA)" <>
Subject [jira] Commented: (LUCENENET-358) CloseableThreadLocal memory leak in LocalDataStoreSlot (with workaround)
Date Tue, 27 Apr 2010 18:00:34 GMT


Digy commented on LUCENENET-358:

    It seems that your are making concurrent updates/searches on your index.
If so, then just use a single instance of IndexWriter throughout your application (without
closing it) and use IndexWriter.GetReader method whenever an IndexReader is required(ie. to
create an IndexSearcher and don't forget to close it after your search is finished). 

    With this approach, I am pretty sure that your mem-leakage will drop/disappear, but -of
course- this doesn't mean that we don't have to fix the bug, if exists.

PS: IndexModifier is deprecated and has a terrible performance.


> CloseableThreadLocal memory leak  in LocalDataStoreSlot (with workaround)
> -------------------------------------------------------------------------
>                 Key: LUCENENET-358
>                 URL:
>             Project: Lucene.Net
>          Issue Type: Bug
>         Environment: Microsoft WIndows Server 2008 Enterprise x64. SP2.
> .NET Framework 4.0
>            Reporter: Rezgar Cadro
>            Priority: Critical
>         Attachments: CloseableThreadLocal MemoryLeak.patch
> Recently we have been suffering from a severe memory leak when executing intense open/close
operations on IndexSearcher and IndexModifier. 
> Memory profiling showed that memory is being held by LocalDataStore[] objects.
> After some digging, the root of the problem has been found in CloseableThreadLocal class:
> private System.LocalDataStoreSlot t = System.Threading.Thread.AllocateDataSlot();
> What we see is that every instantiated object of CloseableThreadLocal causes new data
slot allocation performed for every thread. 
> Thread.AllocateDataSlot() does not simply allocate a new slot, replacing an old one,
but enlarging an existing buffer in-thread, appending data to the end of internal LocalDataStore[]
collection, which  causes a severe memory leak .
> As long as "t" variable is instantiated on every object creation, and (in current class
implementation) every object is used by a single thread, replacing "private System.LocalDataStoreSlot
t = System.Threading.Thread.AllocateDataSlot();" with simple "private object dataSlot;" and
removing "hardRefs" Dictionary solves the problem and prevents memory leak. 
> We have tried to implement the expected behavior by using [ThreadStatic] attribute instead
of LocalDataStoreSlot, but the attempt failed because of unexpected exceptions being thrown.
> Patch can be found at Lucene.Net repository under 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message