lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "George Aroush" <geo...@aroush.net>
Subject RE: Remote searches with Lucene
Date Tue, 22 Aug 2006 01:48:52 GMT
Hi Jeff,

If you want to contribute the code, I am sure many can benefit from it.  I
can make it part of the "contribute" code base of Lucene.Net and share it
here:
https://svn.apache.org/repos/asf/incubator/lucene.net/trunk/C%23/contrib/

Regards,

-- George 

-----Original Message-----
From: Jeff Rodenburg [mailto:jeff.rodenburg@gmail.com] 
Sent: Monday, August 21, 2006 12:11 PM
To: lucene-net-dev@incubator.apache.org
Cc: Elena Demidova
Subject: Re: Remote searches with Lucene

Hello all -

I've been watching this thread to follow the direction and thought I might
be able to offer some assistance.  I run a search system that involves 4
separate search servers -- 3 serving search objects via RemoteSearchable,
and a 4th that serves in an index updating role.

The codebase for Lucene.Net provides all the library routines one needs to
provide distributed search capabilities, but does not provide facilities for
distributed search operation -- nor should it.  The ideas presented here are
certainly possible; I've implemented a working operation without requiring
the changes described here.  I'm confident in our implementation; for the
calendar year, our uptime/availability of search services is 99.99%.  Our
only outage was related to network hardware, otherwise we're sitting solid
at 100%.

I've been authorized to provide our operational code for distributed search
under Lucene.Net to the community at large.  Some of the code is customized
to our operation, but for the most part it's rather generic.  We started the
project under Lucene v1.4.3, but the operational aspect still applies under
v1.9.

The system consists of a LuceneServer, which provides searchability against
indexes as defined in XML configuration files.  In addition, an
IndexUpdateServer provides master index updating, master/slave index
replication and automated index maintenance.  Integration with our web site
ensures the index stays available, updated and current.  There's a great
deal of applied knowledge and learned behavior of many of the underlying
sub-system components that distributed search under Lucene.Net makes use of
-- .Net remoting, garbage collection, etc.

If anyone has interest, please reply.  Contributing this code requires a
little cleanup of our customization work, so my response may not be
immediate but I would make efforts to release the code in short order.

thanks,
jeff r.



On 8/19/06, Robert Boulanger <robert@boulanger.at> wrote:
>
> Hi Elena, hi Rest,
>
> > Dear All,
> >
> > The application I am working on is intended to make use of the 
> > distributed search capabilities of the Lucene library. While trying 
> > to work with the Lucene's RemoteSearchable class, I faced some 
> > problems cased by the current Lucene implementation. In following 
> > I'll try to describe them, as well as the possible ways of their 
> > solution, I identified. The most important question for me is, if 
> > these changes have a chance to be integrated in the coming Lucene 
> > versions, such that remote searches would really become feasible. I 
> > would appreciate any feedback.
>
> Same problem for me and I found some more issues which I explain below:
>
> >
> > The first problem concerns the construction of the RemoteSearchable 
> > object. .Net framework allows for both, server and client activation 
> > models of the remote objects. Currently, RemoteSearchable class 
> > possesses only one constructor that requires knowledge of a local 
> > Searchable object:
> >
> > public RemoteSearchable(Lucene.Net.Search.Searchable local)
> >
> I just added a new constructor to RemoteSearchable public 
> RemoteSearchable(): base() { this.local = this.local; }
>
> not the fine method but for me it works so far.
>
> > Since this "local" object is located on the server, knowledge of the 
> > server's index paths is needed for its creation. However, there are 
> > at least some scenarios where only the server, but not the client, 
> > knows where the indexes are stored on the server side. I think this 
> > problem could be solved by extending RemoteSearchable class with a 
> > standard constructor that reads the names of the indexes to be 
> > published out of a configuration file on the server side.
> >
> My "Server" now implements a Class which inherits directly from Remote 
> Searchable.
> in the parameterless constructor there I read the server sided 
> configfile which contains the index location , create a new 
> IndexReader and pass it as Argument to MyBase.New() See sample below.
>
> > 2. Bug in Term construction
> [snip]
>
> This whole chapter was very useful and I can commit everything works 
> fine from there on.
>
> But there is still a bug in FieldDocSortedHitQueue line 130 and below:
> I figured out that the castings are not working when the system is 
> running in a non english globalization context.
> The String in docAFields[i] which might be for example 1.345678 is 
> casted to 1345678.0 since the decimal sign is misinterpreted in German 
> systems as it seems.
> So the casting results in an overflow.
>
> So I changed it as follows:
>
> case SortField.SCORE:
> float r1 = (float)Convert.ToSingle(docA.fields[i],
> System.Globalization.NumberFormatInfo.InvariantInfo);
> float r2 = (float)Convert.ToSingle(docA.fields[i],
> System.Globalization.NumberFormatInfo.InvariantInfo);
> if (r1 > r2)
> c = - 1;
> if (r1 < r2)
> c = 1;
> break;
>
> Same in line 172 and 174:
>
> float f1 = (float)Convert.ToSingle(docA.fields[i],
> System.Globalization.NumberFormatInfo.InvariantInfo);
> //UPGRADE_TODO: The equivalent in .NET for method 
> 'java.lang.Float.floatValue' may return a different value.
>
>
"ms-help://MS.VSCC.v80/dv_commoner/local/redirect.htm?index='!DefaultContext
WindowIndex'&keyword='jlca1043'"
> float f2 = (float)Convert.ToSingle(docB.fields[i],
> System.Globalization.NumberFormatInfo.InvariantInfo);
>
>
>
> A tiny Client Server Solution now looks like this (Here in VB.NET)
> SERVER:
> Public Class RemoteQuery
> Inherits RemoteSearchable
> Public Sub New()
> MyBase.New(New IndexSearcher("C:\lucene\index")) End Sub Public Sub 
> New(ByVal local As Searchable)
> MyBase.New(local)
> End Sub
>
> End Class
>
> Module Module1
> Public Sub Main(ByVal args As System.String()) Dim chnl As New 
> HttpChannel(8888) ChannelServices.RegisterChannel(chnl, False) Dim 
> indexName As System.String = Nothing 
> RemotingConfiguration.RegisterWellKnownServiceType(GetType(RemoteQuery
> ), "Searchable", WellKnownObjectMode.Singleton)
> System.Console.ReadLine()
> End Sub
> End Module
> CLIENT
> Sub Main()
> Dim searchables As Lucene.Net.Search.Searchable() = New
> Lucene.Net.Search.Searchable() {LookupRemote()} Dim searcher As 
> Searcher = New MultiSearcher(searchables) Dim sort As New 
> Lucene.Net.Search.Sort
> sort.SetSort(Lucene.Net.Search.SortField.FIELD_SCORE)
> Dim query As Query = QueryParser.Parse("Harry", "body", New
> StandardAnalyzer())
> Dim result As Hits = searcher.Search(query, sort) End Sub Private 
> Function LookupRemote() As Lucene.Net.Search.Searchable Return 
> CType(Activator.GetObject(GetType(Lucene.Net.Search.Searchable),
> "http://192.168.8.7:8888/Searchable"), Lucene.Net.Search.Searchable) 
> End Function
>
> Hope this helps you and anybody else how has problems with 
> remotesearch so far.
>
> BTW: this all refers Version 1.9rc1
>
> --Robert Boulanger
>


Mime
View raw message