lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir (JIRA)" <j...@apache.org>
Subject [Lucene.Net] [jira] [Created] (LUCENENET-428) How to do that the results are displayed in the first original tokens and them with synonyms?
Date Tue, 28 Jun 2011 11:22:18 GMT
How to do that the results are displayed in the first original tokens and them with synonyms?
---------------------------------------------------------------------------------------------

                 Key: LUCENENET-428
                 URL: https://issues.apache.org/jira/browse/LUCENENET-428
             Project: Lucene.Net
          Issue Type: Task
          Components: Lucene.Net Core
    Affects Versions: Lucene.Net 2.9.4
         Environment: .net 4.0
            Reporter: Vladimir


How to do that the results are displayed in the first original tokens and them with synonyms?
My Analyzer(part) :

public override TokenStream TokenStream(string fieldName, TextReader reader)
        {
            TokenStream result = new StandardTokenizer(reader);
            result = new LowerCaseFilter(result);
			result = new StopFilter(result, stoptable);
            result = new SynonymFilter(result, synonymEngine); 
            result = new ExtendedRussianStemFilter(result, charset);
            return result;
        }
My SynonymFilter :

internal class SynonymFilter : TokenFilter
    {
        private readonly ISynonymEngine engine;

        private readonly Queue<Token> synonymTokenQueue
            = new Queue<Token>();

        public SynonymFilter(TokenStream tokenStream, ISynonymEngine engine) : base(tokenStream)
        {
            this.engine = engine;
        }

        public override Token Next()
        {
            if (synonymTokenQueue.Count > 0)
            {
                return synonymTokenQueue.Dequeue();
            }
            
            Token t = input.Next();
            
            if (t == null)
                return null;

            if (t.Type() == "<SYNONYM>")
                return t;
            
            IEnumerable<string> synonyms = engine.GetSynonyms(t.TermText());
            
            if (synonyms == null)
            {
                return t;
            }
            
            foreach (string syn in synonyms)
            {
                if (!t.TermText().Equals(syn))
                {
                    var synToken = new Token(syn, t.StartOffset(),
                                             t.EndOffset(), "<SYNONYM>");
                    
                    synToken.SetPositionIncrement(0);
                    synonymTokenQueue.Enqueue(synToken);
                }
            }

            return t;
        }
    }

Thanks!


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message