lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Svensson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENENET-488) Can't open IndexReader, get OutOFMemory Exception
Date Tue, 01 May 2012 11:23:50 GMT

    [ https://issues.apache.org/jira/browse/LUCENENET-488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13265747#comment-13265747
] 

Simon Svensson commented on LUCENENET-488:
------------------------------------------

The 1.5 GiB limit sounds like you're executing a 32bit application. Is this correct? 

Does it work if you're calling the overload of IndexReader.Open which accepts a termInfosIndexDivisor
directly? (You can pass null for deletion policy to use the default deletion policy.) The
default termInfosIndexDivisor is one, increasing it will decrease the amount of memory required.
This will slow down some term-related operations against the index, but it sounds better than
not being able to open it at all.

There are some information about what data is loaded into memory at http://blog.mikemccandless.com/2010/07/lucenes-ram-usage-for-searching.html
                
> Can't open IndexReader, get OutOFMemory Exception
> -------------------------------------------------
>
>                 Key: LUCENENET-488
>                 URL: https://issues.apache.org/jira/browse/LUCENENET-488
>             Project: Lucene.Net
>          Issue Type: Bug
>          Components: Lucene.Net Core
>    Affects Versions: Lucene.Net 2.9.4g
>         Environment: Windows server 2008R2
>            Reporter: Steven
>
> Have build a large database with ~1Bn records (2 items per document) it has size 200GB
on disk. I managed to write the indexe by chunking into 100,000 blocks as I ended up with
some threading issues (another bug submission). Anyway the index is built but I can't open
it and get a memory exception (process explorer gets to 1.5GB allocated before it dies but
not sure how reliable that is, but do know there is plenty more RAM left on the box).
> Stack trace below:
> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was
>  thrown.
>    at Lucene.Net.Index.TermInfosReader..ctor(Directory dir, String seg, FieldInf
> os fis, Int32 readBufferSize, Int32 indexDivisor)
>    at Lucene.Net.Index.SegmentReader.CoreReaders..ctor(SegmentReader origInstanc
> e, Directory dir, SegmentInfo si, Int32 readBufferSize, Int32 termsIndexDivisor)
>    at Lucene.Net.Index.SegmentReader.Get(Boolean readOnly, Directory dir, Segmen
> tInfo si, Int32 readBufferSize, Boolean doOpenStores, Int32 termInfosIndexDiviso
> r)
>    at Lucene.Net.Index.SegmentReader.Get(Boolean readOnly, SegmentInfo si, Int32
>  termInfosIndexDivisor)
>    at Lucene.Net.Index.DirectoryReader..ctor(Directory directory, SegmentInfos s
> is, IndexDeletionPolicy deletionPolicy, Boolean readOnly, Int32 termInfosIndexDi
> visor)
>    at Lucene.Net.Index.DirectoryReader.<>c__DisplayClass1.<Open>b__0(String
segm
> entFileName)
>    at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit)
>    at Lucene.Net.Index.DirectoryReader.Open(Directory directory, IndexDeletionPo
> licy deletionPolicy, IndexCommit commit, Boolean readOnly, Int32 termInfosIndexD
> ivisor)
>    at Lucene.Net.Index.IndexReader.Open(String path, Boolean readOnly)
>    at Lucene.Net.Demo.SearchFiles.Main(String[] args)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message