lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laimonas Simutis <lai...@gmail.com>
Subject Re: Unauthorized Access Exceptions in tests
Date Mon, 05 Jan 2015 18:09:22 GMT
Lucene implementation references a pretty good blog entry
http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/
in the comments. It looks like the goal is to make sure that the index does
not get corrupted on hard crashes of the host system.

Based on the info here,
http://msdn.microsoft.com/en-us/library/ee474552(v=vs.110).aspx, what
IOUtils is doing should be sufficient as it calls Flush(true). But then
there is a post that references a bug with the behavior:
http://stackoverflow.com/questions/383324/how-to-ensure-all-data-has-been-physically-written-to-disk
(bug here:
https://connect.microsoft.com/VisualStudio/feedback/details/634385/filestream-flush-flushtodisk-true-call-does-not-flush-the-buffers-to-disk#details)
with no update if and when it has been fixed.

Besides the FileStream.Flush(true) being sufficient, there is still a
question of what to do with the directories, and if fsync on directories is
necessary with how .NET handles disk IO...


Oren, to your point, there is a check made before calling fsync on
directory to make sure it exists, judging from the comments made here:
https://github.com/laimis/lucenenet/blob/master/src/Lucene.Net.Core/Store/FSDirectory.cs#L398
Assumption being is that if the files were fsynced, the directory that
contains files will be there. Not sure if that is what you were referring
to, but that's something that caught my eye.




On Mon, Jan 5, 2015 at 12:01 PM, Oren Eini (Ayende Rahien) <
ayende@ayende.com> wrote:

> Note that it is possible to get into a situation where a directory is
> created by API call, and doesn't yet exists on Windows.
>
> *Hibernating Rhinos Ltd  *
>
> Oren Eini* l CEO l *Mobile: + 972-52-548-6969
>
> Office: +972-4-622-7811 *l *Fax: +972-153-4-622-7811
>
>
>
> On Mon, Jan 5, 2015 at 5:04 PM, Paul Irwin <pirwin@feature23.com> wrote:
>
> > I believe the code was commented out because it doesn't seem like it is
> > needed on Windows, although my understanding of that could be wrong. I've
> > used Lucene.net code with the sync stuff commented out in apps in
> > production for a couple years and no issues.
> >
> > However, that doesn't mean that there can't be a possible issue,
> especially
> > if you factor in Mono on Linux/OS X with different filesystems. Does
> anyone
> > know the actual purpose of the fsync code in Java Lucene? Is recreating
> it
> > even needed with System.IO?
> >
> >
> > Paul Irwin
> > Lead Software Engineer
> > feature[23]
> >
> > Email: pirwin@feature23.com
> > Cell: 863-698-9294
> >
> > On Mon, Jan 5, 2015 at 8:56 AM, Laimonas Simutis <laimis@gmail.com>
> wrote:
> >
> > > Tests occassionally fail with Unauthorized access exception with stack
> > > trace pointing here:
> > >
> > >
> > >
> >
> https://github.com/apache/lucenenet/blob/master/src/Lucene.Net.Core/Util/IOUtils.cs#L444
> > >
> > > To understand the full issue, you can see how it is being called from
> > here:
> > >
> > >
> > >
> >
> https://github.com/apache/lucenenet/blob/master/src/Lucene.Net.Core/Store/FSDirectory.cs#L387
> > >
> > > Note how it first fsyncs the files and then if there were any that were
> > > fsynced, it fsyncs a directory containing the files. Directory part is
> > the
> > > one that is causing the problems.
> > >
> > > The issue is that fsync implementation in the IOUtils is using
> FileStream
> > > class to flush both files and directories. Doing so for directories
> > throws
> > > Access Denied exception, always. FileStream class cannot be used to
> > "open"
> > > directories.
> > >
> > >
> > > Trying to think how to fix this. The simplest one is to catch Access
> > Denied
> > > thrown and ignore it. You can see how the existing implementation does
> > this
> > > for IOException catch branch. if dir is true, the IOException is
> ignored
> > > and method passes. That would be the simplest thing to do to get the
> > tests
> > > passing. Heck, even ignore the whole fsync if it is for directory.
> > >
> > > I do think the complete approach would involve falling back to native
> > > functions (CreateFile for Windows to get directory's handle:
> > >
> > >
> >
> http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx
> > > or equivalent in non Windows) and then call FlushFileBuffers or
> > equivalent
> > > in non-Windows. It is kind of what is present in FileSupport class (
> > >
> > >
> >
> https://github.com/apache/lucenenet/blob/master/src/Lucene.Net.Core/Support/FileSupport.cs
> > > )
> > > but not fully implemented. It seems like IOUtils tried to use
> FileSupport
> > > Sync implementation but it was commented out. Does anybody know
> anything
> > > about that? Why it was commented out, etc?
> > >
> > > Looking for advice here on how to proceed. There is a good number of
> > tests
> > > failing this way so it would be a nice issue to take care off. Perhaps
> go
> > > with the simple route and mark the full implementation for TODO?
> > >
> > >
> > > Laimonas
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message