lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Björn Kremer <...@patorg.de>
Subject Re: [Lucene.Net] optimisation for the GermanStemmer.vb
Date Mon, 16 Jan 2012 15:04:36 GMT
Hello,

> Do you mean they like to search for Björn with "bjoern"? Indeed, this
> won't work with the current stemmer. 

My customers have a lot of documents. This documents can contain "björn" 
and/or "bjoern". They want to search for "björn" and also find documents 
that only contains "bjoern".(But with more complex search terms)

> "ue" occurs pretty often as an infix (think of *steuer*): about 1.5%
> of the words of the German aspell dictionary are affected. "ae" and
> "oe" are rather seldom. 

Yes I know this problem. Therefore I have searched a "DIN" standard("DIN 
5007-2"). Maybe this standard contains some more information to reduce 
this problem. But the detection of "Umlaute" is a difficult thing.
This Umlaut problem is the most interessting thing for all my customers. 
They rather want to loose some other normalizations than having the 
Umlaut problem. The fastest solution for this problem is one of my 
ideas. Maybe it is useful
to implement a better detection later.

Björn


Am 16.01.2012 13:27, schrieb Robert Jordan:
> Hi Björn,
>
> On 16.01.2012 11:28, Björn Kremer wrote:
>> Hello,
>>
>> of course you are right. They should have the same root. But on the over
>> hand, most of my customers are very interessted in searching "ae" as 
>> "ä".
>
> Do you mean they like to search for Björn with "bjoern"? Indeed, this
> won't work with the current stemmer.
>
>> Maybe the DIN norm 5007-2 is a solution: "ä" is converted to "a" and
>> "ae" is also converted to "a". So the words have the same root and my
>> customers can also search "ae" as "ä".
>
> "ue" occurs pretty often as an infix (think of *steuer*): about 1.5%
> of the words of the German aspell dictionary are affected. "ae" and
> "oe" are rather seldom.
>
> Still, it may be worth a try, because the stemmer doesn't work
> morphologically anyway. It doesn't really matter if "steuer" is
> stemmed as "steur" or "steu" as long as it's consistent.
>
> Robert
>
>>
>> I'm sorry I have no test data at the moment. But if you like my idea you
>> can also use/offer it for java lucene.
>>
>> Thank You
>> Björn
>>
>> Am 13.01.2012 21:55, schrieb Robert Jordan:
>>> Hi,
>>>
>>> With your patch, words like
>>>
>>> Haus
>>> Häuser
>>>
>>> will have different a root.
>>>
>>> How did you test your changes? Can you provide some statistics,
>>> like how many over- and understemming your patch will produce
>>> for a 50.000 words corpus?
>>>
>>> Anyway, such changes are likely not acceptable because they
>>> break the compatibility with Java Lucene and with existing indexes.
>>>
>>> Robert
>>>
>>> On 13.01.2012 16:27, Björn Kremer wrote:
>>>> Hello,
>>>>
>>>> I have a little optimisation for the GermanStemmer.vb (in
>>>> Contrib.Analyzers) class. At the moment the function "Substitute"
>>>> converts the german "Umlaute" "ä" in "a", "ö" in"o" and "ü" in "u". 
>>>> This
>>>> is not the correct german translation. They must be converted to "ae",
>>>> "oe" and "ue". So I can write the name "Björn" or "Bjoern" but not
>>>> "Bjorn". With this optimization a user can search for "Björn" and also
>>>> find "Bjoern".
>>>>
>>>> Here is the optimized code snippet:
>>>>
>>>> else if ( buffer[c] == 'ä' )
>>>> {
>>>> buffer[c] = 'a';
>>>> buffer.Insert(c + 1, 'e');
>>>> }
>>>> else if ( buffer[c] == 'ö' )
>>>> {
>>>> buffer[c] = 'o';
>>>> buffer.Insert(c + 1,'e');
>>>> }
>>>> else if ( buffer[c] == 'ü' )
>>>> {
>>>> buffer[c] = 'u';
>>>> buffer.Insert(c + 1,'e');
>>>> }
>>>>
>>>> Thank You
>>>> Björn
>>>>
>>>
>>>
>>
>
>

Mime
View raw message