lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerry Suggitt" <gerry.sugg...@leafsprout.com>
Subject Re: This may be a bug - NOPE IT'S NOT (sorry)
Date Thu, 22 Nov 2012 17:25:07 GMT
You are absolutely right. I wasn't doing a commit in my "flusher" function - I was just closing
and reopening the reader and searcher.

I kinda recall that I had performance issues when I used commit (and before I was using NRT)
when I had commit in the flusher function. After moving to NRT I noticed that everything worked
fine without the commit (because now the reader was not relying on the data on the file system).

I did however have a commit in the service Shutdown code.

But upon reviewing the logs from the Windows Upgrade reboot that occurred last Saturday at
3:00, I noticed that the services were not shutdown gracefully, so the Shutdown code and hence
the commit was not called.

Both databases that were lost contained a small number of documents, so they were probably
never committed to the file system automatically.

So sorry to have troubled you, but thanks for all your help.

The good thing is as a result of the abrupt shutdown a bug was revealed in our code. And furthermore
a recovery mechanism was devised that would protect us from another disastrous termination.

Thank-you once again. 

  ----- Original Message ----- 
  From: Simon Svensson 
  To: Gerry Suggitt 
  Cc: dev@lucenenet.apache.org 
  Sent: Thursday, November 22, 2012 12:09 AM
  Subject: Re: This may be a bug


  Hi, 

  I'm using my mail client's reply-all functionality to answer both to the developer mailing
list and to you personally, in case you're not part of the mailing list. You only need to
respond to the mailing list. (I'm not sure which functionality you use, I got several copies
of the answer below.) 

  I'll refer the explanation of the segments_N and segments.gen file to http://lucene.apache.org/core/old_versioned_docs/versions/3_0_0/fileformats.html


    The active segments in the index are stored in the segment info file, segments_N. There
may be one or more segments_Nfiles in the index; however, the one with the largest generation
is the active one (when older segments_N files are present it's because they temporarily cannot
be deleted, or, a writer is in the process of committing, or a customIndexDeletionPolicy is
in use). [...] 

    As of 2.1, there is also a file segments.gen. This file contains the current generation
(the _N in segments_N) of the index. This is used only as a fallback in case the current generation
cannot be accurately determined by directory listing alone (as is the case for some NFS clients
with time-based directory cache expiraation). [...] 

  One of the critical issues is your _PeriodicDocFlusher. Could you share it with us? Does
it call IndexWriter.Flush, as indicated by the name, or IndexWriter.Commit? Does it reopen
your nrt-reader, using either IndexWriter.GetReader or IndexReader.Reopen? If so, does it
also recreate a new IndexSearcher? 

  Everything so far sounds like you forgot to call IndexWriter.Commit. Flush will move move
memory buffers and what-not to disk, but will not commit it for others to read (only a reader
from IndexWriter.GetReader will be able to read it). All this data will be removed as a cleanup-procedure,
a kind of rollback, when another writer is opened against the directory after an application
restart. 

  // Simon 

  On 2012-11-21 21:29, Gerry Suggitt wrote: 



    Yes, I am using NRT. (Or I should say, I am trying to!). I commit within 10 seconds after
a document is added (When a document arrives I start a timer to allow more documents to come
in before making the commitment).

    And before I get into more details of the bug I reported, may I ask you a question about
NRT?

    ---------------------------------------------------------------------------------------------------------------------------------------------------------
start of NRT question

    According to the documentation (at least as it is described for Java), NRT should allow
"updates to be efficiently searched hopefully within milliseconds after an update is complete".
I have found that after an update, the document is not found until I have performed a commit.

    Here is the code that creates the reader, writer and searcher: _flusher is the 10 second
commit timer. 
        public void Start()
        {
            _logger.Info( ()=> "LuceneEngine.Start " + _pathname );
            System.IO.DirectoryInfo dir = new System.IO.DirectoryInfo( _pathname );
            _directory = FSDirectory.Open( dir, new NoLockFactory() ); // nolock is OK for
now because we have a single thread accessing the directory
            _analyzer = new PerFieldAnalyzerWrapper( new WhitespaceAnalyzer() );
            _writer = new IndexWriter( _directory, _analyzer, IndexWriter.MaxFieldLength.UNLIMITED
);
            _logger.Info( ()=> "LuceneEngine.Start begin Optimize" );
            _writer.Optimize();
            _logger.Info( ()=> "LuceneEngine.Start Optimize finished" );
            _reader = _writer.GetReader();
            _searcher = new IndexSearcher( _reader );
            _flusher = new _PeriodicDocFlusher( 10000, _maxDocsInCache, _OnFlushTimer, _logger
);
            _logger.Info( ()=> "LuceneEngine.Start " + _pathname + " - up and running"
);
        }
     And uses it: _DocAdded() just starts the commit timer if is it not already running.

        public void UpdateDocument( string id, Document doc )
        {
            _writer.UpdateDocument( new Term("id", id), doc, _analyzer );
            _DocAdded();
        }

        public TopDocs Search( Query query, int maxDocsToReturn )
        {
            lock( this )
            {    
                return _searcher.Search( query, maxDocsToReturn );
            }
        }
    To test it, after making a call to UpdateDocument with id = xxx, I looped making repeated
calls to Search where the query was id:xxx. This continually returned 0 documents until the
timer kicked in and performed the commit. And then Search returned 1 hit.

    So is this expected? I didn't think so, but maybe I just misinterpreted the documentation.

    -----------------------------------------------------------------------------------------------------------------------------------------------------
end of NRT question

    Back to the issue at hand ...

    I tried to reproduce the problem as I described and was unable to. So I am uncertain what
was happening there. 

    But I have some more information about the lost databases on our test machines. 

    When I say the databases were completely empty, the directory actually held two files:
    segments.gen
    segments_1

    On one machine we restored the data from an external backup (actually a SQL database!)
and everything worked fine from then on. We could see several files in the database directory.

    The other lab machine was untouched and here we discovered something that might be important.

    We noticed on the first machine, after restoring (which essentially performed a series
of _writer.UpdateDocument) after we stopped and started the Lucene service, the timestamp
on the segments.gen had changed and we now had a file segments_2. (I know, I know, you are
going "well, duh", but hold on a sec)

    On the second machine we had not touched it. And the time on the segments.gen file was
November 12, 12:31 PM.

    But the reboot of the machines occurred on November 17 at 3:30 PM.

    So why wasn't the timestamp updated? My guess: Because there were no index files in the
directory!

    But ... I have logs that show 500 documents being added successfully to the database AFTER
November 12, 12:31 PM. And these logs show commits being performed. 

    Furthermore, searches are returning documents.

    So it appears (and this is just my guess) that the commits were making the necessary updates
to the in-memory data structures that allowed searches to work, but the data was never saved
to the disk. No exception occurred which may have been thrown as a result of a failure to
write to the disk, so at this point I am baffled.

    Now why the data was not saved to the disk last week but are being saved this week is
beyond me.

    I know we don't have much to work with. I will continue to see if I can reproduce the
problem. If there is anything else you would like me to check, please ask.

    Thanks - Gerry




    ----- Original Message ----- 
      From: Simon Svensson 
      To: dev@lucenenet.apache.org 
      Cc: Gerry Suggitt 
      Sent: Wednesday, November 21, 2012 3:05 AM
      Subject: Re: This may be a bug


      Hi,

      This does indeed sound serious. Are you saying that you have a snapshot 
      (with committed documents) that is cleared when calling 
      IndexWriter.Optimize? Can you share it for reproduction purposes?

      Are you using near-realtime indexing? What you describe could happen if 
      you were using nrt, and never called IndexWriter.Commit. The index would 
      indeed be cleared next time an writer is opened against the directory, a 
      step in clearing out unused index files. A kind of rollback of 
      non-commited changes.

      // Simon


      On 2012-11-20 16:45, Gerry Suggitt wrote:
      > Sorry to send this email directly to the developers, but I couldn't see any other
way of entering a defect.
      >
      > My name is Gerry Suggitt and I work for Leafsprout Technologies, a company that
creates products for the Medical Information sector.
      >
      > We have created a Master Patient Index using Lucene that works very well - we are
able to perform fuzzy matching and all the nice things that you want in a MPI.
      >
      > But something terrible just happened. Fortunately this occured in our own lab -
we have not yet released the product to the field.
      >
      > Sometime over the weekend, the computers holding the Lucene database rebooted (probably
from a Windows upgrade). All of the Lucene databases were blown away! Completely empty!
      >
      > Recently, I had noticed the same thing when I was doing some testing, so it may
be related.
      >
      > We are currently using version 2.9.4.1
      >
      > What I was doing in my testing was taking a snapshot of the Lucene database files
(just a copy to another directory). I would run some tests which would affect the database,
so before continuing I would copy the snapshot back.
      >
      > When I started the Lucene service, the database was blown away! Completely empty!
      >
      > I was able to determine what was doing this. At startup, I was performing an optimize.
This seems like a good time for me: At startup we know no client is making demands on the
system. When I commented out the call to optimize, the database remained intact up startup.
      >
      > The systems that lost their databases still had the call to optimize in them.
      >
      > Please help!
      >




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message