lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laimonas Simutis <lai...@gmail.com>
Subject Re: Unauthorized Access Exceptions in tests
Date Mon, 05 Jan 2015 22:15:59 GMT
There is some really good info on this in the JIRA ticket that made Lucene
group to implement fsync the way they did:

https://issues.apache.org/jira/browse/LUCENE-5588

It looks like Java version does not work for directories either and they
relied on catch(IOException) and returned if the call was for the
directory. In .NET UnauthorizedAccessException is not an IO exception and
it bubbled up to the callers. That makes me think we should go with the
route of ignoring fsync for directory altogether.


Laimonas


On Mon Jan 05 2015 at 1:09:22 PM Laimonas Simutis <laimis@gmail.com> wrote:

> Lucene implementation references a pretty good blog entry
> http://blog.httrack.com/blog/2013/11/15/everything-
> you-always-wanted-to-know-about-fsync/ in the comments. It looks like the
> goal is to make sure that the index does not get corrupted on hard crashes
> of the host system.
>
> Based on the info here, http://msdn.microsoft.com/en-
> us/library/ee474552(v=vs.110).aspx, what IOUtils is doing should be
> sufficient as it calls Flush(true). But then there is a post that
> references a bug with the behavior: http://stackoverflow.com/
> questions/383324/how-to-ensure-all-data-has-been-
> physically-written-to-disk (bug here: https://connect.
> microsoft.com/VisualStudio/feedback/details/634385/
> filestream-flush-flushtodisk-true-call-does-not-flush-the-
> buffers-to-disk#details) with no update if and when it has been fixed.
>
> Besides the FileStream.Flush(true) being sufficient, there is still a
> question of what to do with the directories, and if fsync on directories is
> necessary with how .NET handles disk IO...
>
>
> Oren, to your point, there is a check made before calling fsync on
> directory to make sure it exists, judging from the comments made here:
> https://github.com/laimis/lucenenet/blob/master/src/Lucene.Net.Core/Store/
> FSDirectory.cs#L398 Assumption being is that if the files were fsynced,
> the directory that contains files will be there. Not sure if that is what
> you were referring to, but that's something that caught my eye.
>
>
>
>
> On Mon, Jan 5, 2015 at 12:01 PM, Oren Eini (Ayende Rahien) <
> ayende@ayende.com> wrote:
>
>> Note that it is possible to get into a situation where a directory is
>> created by API call, and doesn't yet exists on Windows.
>>
>> *Hibernating Rhinos Ltd  *
>>
>> Oren Eini* l CEO l *Mobile: + 972-52-548-6969
>>
>> Office: +972-4-622-7811 *l *Fax: +972-153-4-622-7811
>>
>>
>>
>> On Mon, Jan 5, 2015 at 5:04 PM, Paul Irwin <pirwin@feature23.com> wrote:
>>
>> > I believe the code was commented out because it doesn't seem like it is
>> > needed on Windows, although my understanding of that could be wrong.
>> I've
>> > used Lucene.net code with the sync stuff commented out in apps in
>> > production for a couple years and no issues.
>> >
>> > However, that doesn't mean that there can't be a possible issue,
>> especially
>> > if you factor in Mono on Linux/OS X with different filesystems. Does
>> anyone
>> > know the actual purpose of the fsync code in Java Lucene? Is recreating
>> it
>> > even needed with System.IO?
>> >
>> >
>> > Paul Irwin
>> > Lead Software Engineer
>> > feature[23]
>> >
>> > Email: pirwin@feature23.com
>> > Cell: 863-698-9294
>> >
>> > On Mon, Jan 5, 2015 at 8:56 AM, Laimonas Simutis <laimis@gmail.com>
>> wrote:
>> >
>> > > Tests occassionally fail with Unauthorized access exception with stack
>> > > trace pointing here:
>> > >
>> > >
>> > >
>> > https://github.com/apache/lucenenet/blob/master/src/
>> Lucene.Net.Core/Util/IOUtils.cs#L444
>> > >
>> > > To understand the full issue, you can see how it is being called from
>> > here:
>> > >
>> > >
>> > >
>> > https://github.com/apache/lucenenet/blob/master/src/
>> Lucene.Net.Core/Store/FSDirectory.cs#L387
>> > >
>> > > Note how it first fsyncs the files and then if there were any that
>> were
>> > > fsynced, it fsyncs a directory containing the files. Directory part is
>> > the
>> > > one that is causing the problems.
>> > >
>> > > The issue is that fsync implementation in the IOUtils is using
>> FileStream
>> > > class to flush both files and directories. Doing so for directories
>> > throws
>> > > Access Denied exception, always. FileStream class cannot be used to
>> > "open"
>> > > directories.
>> > >
>> > >
>> > > Trying to think how to fix this. The simplest one is to catch Access
>> > Denied
>> > > thrown and ignore it. You can see how the existing implementation does
>> > this
>> > > for IOException catch branch. if dir is true, the IOException is
>> ignored
>> > > and method passes. That would be the simplest thing to do to get the
>> > tests
>> > > passing. Heck, even ignore the whole fsync if it is for directory.
>> > >
>> > > I do think the complete approach would involve falling back to native
>> > > functions (CreateFile for Windows to get directory's handle:
>> > >
>> > >
>> > http://msdn.microsoft.com/en-us/library/windows/desktop/
>> aa363858(v=vs.85).aspx
>> > > or equivalent in non Windows) and then call FlushFileBuffers or
>> > equivalent
>> > > in non-Windows. It is kind of what is present in FileSupport class (
>> > >
>> > >
>> > https://github.com/apache/lucenenet/blob/master/src/
>> Lucene.Net.Core/Support/FileSupport.cs
>> > > )
>> > > but not fully implemented. It seems like IOUtils tried to use
>> FileSupport
>> > > Sync implementation but it was commented out. Does anybody know
>> anything
>> > > about that? Why it was commented out, etc?
>> > >
>> > > Looking for advice here on how to proceed. There is a good number of
>> > tests
>> > > failing this way so it would be a nice issue to take care off.
>> Perhaps go
>> > > with the simple route and mark the full implementation for TODO?
>> > >
>> > >
>> > > Laimonas
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message