lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shad Storhaug (JIRA)" <j...@apache.org>
Subject [jira] [Created] (LUCENENET-618) Need a Cross-OS NativeFSLock Implementation
Date Thu, 15 Aug 2019 08:00:05 GMT
Shad Storhaug created LUCENENET-618:
---------------------------------------

             Summary: Need a Cross-OS NativeFSLock Implementation
                 Key: LUCENENET-618
                 URL: https://issues.apache.org/jira/browse/LUCENENET-618
             Project: Lucene.Net
          Issue Type: Task
          Components: Lucene.Net Core
    Affects Versions: Lucene.Net 4.8.0
            Reporter: Shad Storhaug
             Fix For: Lucene.Net 4.8.0


Now that we have (finally) begun testing on platforms other than Windows, it has been discovered
that the {{NativeFSLock}} doesn't work reliably on other platforms.

Due to differences in how Java and .NET handle file system locking, we have switched from
the Lucene 4.8.0 implementation to [one that is based on {{FileStream.Lock}}|https://lucene.markmail.org/search/?q=lucenenet%20An%20alternative%20NativeFSLockFactory#query:lucenenet%20An%20alternative%20NativeFSLockFactory+page:1+mid:pdx6nwciw4rn75l6+state:results].
While this implementation does include support for .NET Framework/.NET Standard 1.x/.NET Standard
2.x, it relies on {{HResult}} error codes that exist only on Windows.

I have done some research, and it turns out that [using {{HResult}} error codes cannot be
made reliably portable across platforms|https://stackoverflow.com/a/46381756]. So, I used
the original implementation we had as a fallback when the OS is not Windows.

{code:c#}
        internal virtual Lock NewLock(string path)
        {
            if (Constants.WINDOWS)
                return new WindowsNativeFSLock(this, m_lockDir, path);

            // Fallback implementation for unknown platforms that don't rely on HResult
            return new NativeFSLock(this, m_lockDir, path);
        }
{code}

Unfortunately, we are now seeing error messages when testing on macOS and Linux, which are
a clear symptom of resource locking failures:

{quote}{{System.IO.IOException : The process cannot access the file '/tmp/LuceneTemp/index-MMapDirectory-i0xc4fz2/write.lock'
because it is being used by another process.}}{quote}
{quote}{{at Lucene.Net.Util.IOUtils.DisposeWhileHandlingException(Exception priorException,
IDisposable[] objects) in D:\a\1\s\src\Lucene.Net\Util\IOUtils.cs:line 197}}{{at Lucene.Net.Store.Directory.Copy(Directory
to, String src, String dest, IOContext context) in D:\a\1\s\src\Lucene.Net\Store\Directory.cs:line
214}}{{at Lucene.Net.Store.MockDirectoryWrapper.Copy(Directory to, String src, String dest,
IOContext context) in D:\a\1\s\src\Lucene.Net.TestFramework\Store\MockDirectoryWrapper.cs:line
1320}}{{at Lucene.Net.Store.RAMDirectory..ctor(Directory dir, Boolean closeDir, IOContext
context) in D:\a\1\s\src\Lucene.Net\Store\RAMDirectory.cs:line 106}}{{at Lucene.Net.Index.TestTermVectorsWriter.TestTermVectorCorruption()
in D:\a\1\s\src\Lucene.Net.Tests\Index\TestTermVectorsWriter.cs:line 446}}{quote}

It turns out the implementation that was used as a fallback had been [contributed back when
.NET Framework was the only target framework in Lucene.Net|https://github.com/apache/lucenenet/pull/70#issuecomment-72958293]
and it sounds like running on non-Windows platforms was not being considered in its implementation.

As a result, we only have 2 implementations that work on Windows and none that work reliably
on other platforms.

The {{NativeFSLock}} implementation does prevent the "file in use" error from happening frequently
on non-Windows platforms, but doesn't prevent it from happening completely. However, the {{WindowsNativeFSLock}}
is reliably making all of the tests pass on Windows across several dozen test runs.

IMO, the approach being used on Windows is fine, but for other operating systems we should
fallback to an implementation that is more reliable than the one we currently have.
 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Mime
View raw message