lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Garski (JIRA)" <j...@apache.org>
Subject [jira] Updated: (LUCENENET-337) TokenAttribute for Selectively Including Tokens in Length Norm
Date Tue, 19 Jan 2010 17:39:54 GMT

     [ https://issues.apache.org/jira/browse/LUCENENET-337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael Garski updated LUCENENET-337:
-------------------------------------

    Attachment: LengthNorm.patch

Patch is attached.  Note that the updated project file has been converted to VS2008 version,
so if you are using 2005, you will want to skip updating the project file and manually add
the two new files to the project:

Analysis\Tokenattributes\LengthNormAttribute.cs
Analysis\Tokenattributes\LengthNormAttributeImpl.cs

> TokenAttribute for Selectively Including Tokens in Length Norm
> --------------------------------------------------------------
>
>                 Key: LUCENENET-337
>                 URL: https://issues.apache.org/jira/browse/LUCENENET-337
>             Project: Lucene.Net
>          Issue Type: Improvement
>            Reporter: Michael Garski
>            Priority: Minor
>         Attachments: LengthNorm.patch
>
>
> This patch adds functionality to Lucene.Net that allow a TokenFilter to mark a Token
as not to be included in the length norm calculation through the use of a new TokenAttribute
interface LengthNormAttribute and a corresponding implementation LengthNormAttributeImpl.
 This functionality is useful to prevent the increase of the length norm during synonym injection,
particularly in cases where there are a large number of synonyms in relation to the number
of original tokens.
> Following is an example of how to use the new attribute.
> Within your custom TokenFilter, define a field to persist a reference to the attribute
and set it's value in the constructor.  When a the stream advances to a new Token within the
call to IncrementToken() the value of the IncludeInLengthNorm property of the attribute is
set to false for Tokens which should not be included in the length norm calculation.  It defaults
to true and is reset to true after each Token is consumed within DocInverterPerField.ProcessFields.
> {code:title=CustomTokenFilter.cs|borderStyle=solid}
> public class CustomTokenFilter : TokenFilter
> {
> 	private LengthNormAttribute lnAttribute;
> 	
> 	public CustomTokenFilter(TokenStream input) : base(input)
> 	{
> 		this.lnAttribute = (LengthNormAttribute)AddAttribute(typeof(LengthNormAttribute));
> 	}
> 		
> 	public override bool IncrementToken()
> 	{
> 		if (input.IncrementToken())
> 		{
> 			// make determination that the token is not to be 
> 			// included in the length norm value
> 			// this example marks all tokens to not be 
> 			// included in the length norm value
> 			this.lnAttribute.IncludeInLengthNorm = false;
> 			return true;
> 		}
> 		else
> 		{
> 			return false;
> 		}
> 	}    
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message