lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrei Alecu <and...@tachyon-labs.com>
Subject Re: Lucene + Windows 2008 R2 performance issues
Date Fri, 19 Feb 2010 15:08:32 GMT
Our servers are virtualized with Hyper-V, the iSCSI SAN LUN is exposed 
only to the host the "clients" don't know about its existence, they're 
just plain hard drives as far as they're concerned.

If you look at the "Cached" value in Task Manager on R2, you'll see that 
it is not as big as the whole available physical RAM, in some cases not 
expanding nearly as much as it should.

For example, we have a smaller lucene index of 1GB and the "Cached" 
field never goes beyond 250 MB, and the virtual machine is dedicated to 
this Lucene index only, so it's the only thing running on it.

On vanilla Windows 2008, the same field would grow to at least 1GB+ and 
the full index would be accessed by lucene as if it were a RAMDirectory.

The workaround we made for now was to load up the whole index into a 
RAMDirectory because 1GB is pretty small. But we have other indexes of 
4+ GBs which are not feasible to load up in RAM.

I'm going to work on the MMapDirectory implementation as a private 
project and release it if I end up with anything useful.

On 2/18/2010 7:41 PM, Michael Garski wrote:
> Andrei,
>
> This is interesting.  I have not observed this behavior on 2008 R2 and
> we cache quite a bit in memory (filters, field cache), but we're using
> local disk as opposed to an iSCSI SAN.  Could there be some change that
> affects how those files are cached as opposed to local disk?
>
> Michael
>
> -----Original Message-----
> From: Andrei Alecu [mailto:andrei@tachyon-labs.com]
> Sent: Thursday, February 18, 2010 3:25 AM
> To: lucene-net-dev@incubator.apache.org
> Subject: Lucene + Windows 2008 R2 performance issues
>
> It seems like Microsoft changed the way the memory manager works in the
> new Windows 7 kernel (which is what Windows 2008 R2 uses), not letting
> the system cache grows to use up all available RAM any more. (at least
> not in a predictable manner)
>
> Here's a reference: http://support.microsoft.com/kb/976618
>
>      "The memory management algorithms were updated to address this
>      problem in Windows 7 and Windows Server 2008 R2 operating systems.
>      Therefore, we do not recommend that you use the provided functions
>      or the Microsoft Windows Dynamic Cache Service in computers that are
>      running Windows 7 or Windows Server 2008 R2."
>
>
> This is in some circumstances a good thing, but it also means that
> things like Lucene now need to access the hard drive more than before.
> We're storing a lucene index on an iSCSI SAN, which used to work great
> pre W2K8 R2 because it eventually found its way into the system cache,
> and it wouldn't hit the network any more. This doesn't happen any more
> with R2 and we see some query time degradation.
>
>
> The solution to this issue would be to use Memory Mapped I/O, but I see
> that MMapDirectory is not implemented, are there any plans to implement
> it?
>
> There is first class support for Memory Mapped Files in .NET 4.0 which
> is still some time away, but until then the Win32 API could be used.
>
>
>
>    


Mime
View raw message