lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Garski <mgar...@myspace.com>
Subject RE: Search with Sort
Date Thu, 16 Oct 2008 00:26:31 GMT
Digy's patch will address the memory leakage in the field cache, but even though the garbage
collector will now free the memory associated with the sort values, you may still end up with
memory issues due to how the .NET runtime manages memory.

For example, if you have an index with 250,000 items in it and perform queries on it using
a sort based integer value on one of the fields.  The field cache will create an array of
250,000 integers to provide for fast access to the sort values.  As the array is a large object
(> 85,000 bytes) it will be allocated on the large object heap (LOH).  While during a garbage
collection memory is freed from the LOH, the LOH is not compacted like generations 0, 1, &
2.  If your application swaps over indexes without restarting, such as in a Windows Service,
the LOH could become fragmented over time.  This fragmentation can lead to an OutOfMemoryException
as the runtime is not able to allocate a contiguous chunk of memory for the array.

There is a good MSDN article on the LOH and how to use windbg to peek at what's in the managed
heap: http://msdn.microsoft.com/en-us/magazine/cc534993.aspx

Michael



-----Original Message-----
From: Digy [mailto:digydigy@gmail.com] 
Sent: Wednesday, October 15, 2008 2:41 PM
To: lucene-net-dev@incubator.apache.org
Subject: RE: Search with Sort

Yes, there is a MemoryLeakage-bug while searching+sorting in versions 2.1 &
2.3.1 
and as far as I can reproduce the bug, I prepared two patches
("WeakHashTable+FieldCacheImpl.rar" for v2.1 and "Paches for v2.3.1.rar" for
v2.3.1 (https://issues.apache.org/jira/browse/LUCENENET-106) ).

Although there are still many complains about this issue, there isn't(or I
don't have) any test case to track it down.

DIGY.



-----Original Message-----
From: Bruno Matos [mailto:bruno.matos@paradigmaxis.pt] 
Sent: Wednesday, October 15, 2008 8:34 PM
To: lucene-net-dev@incubator.apache.org
Subject: Search with Sort

Hello,

I'm with a problem with memory when I use a Sort. Can anyone help me 
with this?
I sow a lot of people posting about this, but I didn't sow any answers.

Thank you.

-- 
Bruno Matos
ParadigmaXis - Engenharia e Arquitectura de Software, S.A.
Avenida da Boavista, 1043
4100-129 Porto
Portugal

Tel: (+351) 22 605 48 72
Fax: (+351) 22 605 48 74

Mime
View raw message