lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laimonas Simutis <lai...@gmail.com>
Subject Re: Reproducing random test failures
Date Fri, 22 May 2015 16:50:24 GMT
Thanks Wyatt. Yeah I am aware how to do it in code, was just making sure
that I wasn't missing some TC option that would enable me to switch the
builds / verbosity without code pushes. No big deal, I can check in code
and push and get feedback that way.

Looking at the output  now.....

On Thu, May 21, 2015 at 11:44 AM, Wyatt Barnett <wyatt.barnett@gmail.com>
wrote:

> I found a few minutes to play with things.
>
> First, the general technique to switching things to running under RELEASE
> vs DEBUG while keeping VERBOSE output on is to modify line 299 in
> LuceneTestCase.cs to be "true" -- the preprocessor directives there are
> basically tying DEBUG and VERBOSE together.
>
> Output for TestTieBreaker can be found at http://pastebin.com/VXQYF32T
> Output for TestDMQ8 can be found at http://pastebin.com/WYS07KqB
> Output for TestSort_2 can be downloaded from
>
> https://drive.google.com/file/d/0By01KKFIe0LEX1NubW1mRlVMVWM/view?usp=sharing
>
> I hope this helps get to the bottom of this one, let me know if you need
> further assistance.
>
> On Tue, May 19, 2015 at 2:02 PM Wyatt Barnett <wyatt.barnett@gmail.com>
> wrote:
>
> > The screenshot didn't come through so I'm kind of guessing here but I
> > switched to debug mode which also triggers VERBOSE to get more output.
> >
> > This confirmed one of the things I saw locally earlier -- that some of
> the
> > semantics switching from debug to release (or non-verbose to verbose come
> > to think of it) makes those tests fail for some reason. The other
> challenge
> > is that the output is so verbose that having verbose on makes the 4th
> test
> > run -- I think the Test_Merge tests -- real, real slow. Like 50+ minute
> > test run slow (
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191532&tab=buildResultsDiv&buildTypeId=LuceneNet_LuceneNetCoreFailingTests
> )
> > compared to ~1 minute test runs.
> >
> > Let me see if I can get output without triggering debug mode when I get
> > home, need to get to the right PC to make that happen.
> >
> > On Tue, May 19, 2015 at 11:56 AM Laimonas Simutis <laimis@gmail.com>
> > wrote:
> >
> >> Wyatt,
> >>
> >> Would it be possible to pass that in through the configuration? I tried
> >> passing in verbose flag through but did not appear to work. The same
> with
> >> configuration. Here is the properties section I am referring to:
> >>
> >> [image: Inline image 1]
> >>
> >> Do you know if it something that is possible to have for the TC builds?
> >>
> >> On Tue, May 19, 2015 at 10:54 AM, Wyatt Barnett <
> wyatt.barnett@gmail.com>
> >> wrote:
> >>
> >>> Great. One thing just hit me -- would it be better for this to run as
> >>> DEBUG
> >>> rather than RELEASE? I can flip that switch too . . .
> >>>
> >>> On Tue, May 19, 2015 at 10:52 AM Laimonas Simutis <laimis@gmail.com>
> >>> wrote:
> >>>
> >>> > Thank you! Just kicked off the build. Let's see what it tells us :)
> >>> >
> >>> > On Tue, May 19, 2015 at 9:59 AM, Wyatt Barnett <
> >>> wyatt.barnett@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Liamonas -- you should be all set, I just added
> >>> > >
> >>> > >
> >>> >
> >>>
> http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests
> >>> > > which runs the core build with focused tests. Please ignore the
> build
> >>> > > number. Test category is more of a general setting for a build
so
> >>> there
> >>> > > isn't an easy checkbox to hit.
> >>> > >
> >>> > > If it makes more sense to re-point that at your github repo just
> say
> >>> the
> >>> > > word and I'll make it so.
> >>> > >
> >>> > >
> >>> > > On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <laimis@gmail.com
> >
> >>> > wrote:
> >>> > >
> >>> > > > Wyatt,
> >>> > > >
> >>> > > > I have a branch pushed for this named "failingtests", it
is now
> >>> > running a
> >>> > > > build on TC. Where does one specify which category of tests
to
> >>> run? I
> >>> > see
> >>> > > > in the settings tab "NUnit categories include/exclude" but
don't
> >>> see
> >>> > > where
> >>> > > > to actually modify these values. The tests I would like to
run
> >>> belong
> >>> > to
> >>> > > > category "Focus" :) Do you know where to change this?
> >>> > > >
> >>> > > > On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett <
> >>> > wyatt.barnett@gmail.com
> >>> > > >
> >>> > > > wrote:
> >>> > > >
> >>> > > > > Good to hear I checked the right box.
> >>> > > > >
> >>> > > > > I'll see what I can pull together when I get home in
terms of
> >>> debug
> >>> > > > output.
> >>> > > > > In terms of testing procedure what I was thinking is
we make a
> >>> new
> >>> > > > category
> >>> > > > > -- call it "Focus" and then configure a build looking
at your
> >>> fork
> >>> > > > > filtering for just those tests. You can then push away,
fire
> off
> >>> > remote
> >>> > > > > builds and check the output yourself.
> >>> > > > >
> >>> > > > > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <
> >>> laimis@gmail.com>
> >>> > > > wrote:
> >>> > > > >
> >>> > > > > > Wyatt,
> >>> > > > > >
> >>> > > > > > I see the new options on TC, thanks for that. I
still haven't
> >>> > thought
> >>> > > > > about
> >>> > > > > > how I will go about capturing the failures exactly,
but will
> >>> give
> >>> > > you a
> >>> > > > > > shout if I need some help with TC configuration
just for
> those
> >>> > runs.
> >>> > > > > >
> >>> > > > > > If you can reproduce any of those test failures
locally, do
> you
> >>> > mind
> >>> > > > > > running them in VERBOSE mode (debug build without
any other
> >>> changes
> >>> > > > will
> >>> > > > > > do), and emailing the console output that you get?
I might be
> >>> too
> >>> > > > > > optimistic, but perhaps something there will stand
out.
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > Thanks again!
> >>> > > > > >
> >>> > > > > > Laimis
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett
<
> >>> > > > wyatt.barnett@gmail.com>
> >>> > > > > > wrote:
> >>> > > > > >
> >>> > > > > > > For TestSort_2 -- It appears to be passing
based on data at
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
> >>> > > > > > > ; I am having locally reproducable problems
on the others
> >>> though.
> >>> > > > > > >
> >>> > > > > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett
<
> >>> > > > > wyatt.barnett@gmail.com>
> >>> > > > > > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Done -- you should now see a run button
when you visit
> >>> > > > > > > >
> >>> > http://teamcity.codebetter.com/project.html?projectId=LuceneNet
> >>> > > > > > > >
> >>> > > > > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas
Simutis <
> >>> > > > laimis@gmail.com>
> >>> > > > > > > > wrote:
> >>> > > > > > > >
> >>> > > > > > > >> Wyatt,
> >>> > > > > > > >>
> >>> > > > > > > >> Could you add me to the lucene.net
group on TC? I have
> a
> >>> > login
> >>> > > > > there,
> >>> > > > > > > >> username: laimis.
> >>> > > > > > > >>
> >>> > > > > > > >>
> >>> > > > > > > >> Thanks!
> >>> > > > > > > >>
> >>> > > > > > > >> On Sat, May 16, 2015 at 6:15 PM,
Wyatt Barnett <
> >>> > > > > > wyatt.barnett@gmail.com
> >>> > > > > > > >
> >>> > > > > > > >> wrote:
> >>> > > > > > > >>
> >>> > > > > > > >> > Sounds good Laimis. You will
need to setup a login to
> >>> the
> >>> > > > > CodeBetter
> >>> > > > > > > >> > teamcity server and get added
to the lucene.net group
> >>> if
> >>> > you
> >>> > > > > > haven't
> >>> > > > > > > >> > already. Let me know if you
need help there too.
> >>> > > > > > > >> >
> >>> > > > > > > >> > On Sat, May 16, 2015 at 4:52
PM Laimonas Simutis <
> >>> > > > > laimis@gmail.com>
> >>> > > > > > > >> wrote:
> >>> > > > > > > >> >
> >>> > > > > > > >> > > Wyatt,
> >>> > > > > > > >> > >
> >>> > > > > > > >> > > Sweet, I will let you know
once I have a branch out
> >>> with
> >>> > > > > > additional
> >>> > > > > > > >> > logging
> >>> > > > > > > >> > > and separate category for
tests that you can
> >>> configure to
> >>> > > run.
> >>> > > > > > > >> > >
> >>> > > > > > > >> > > Re: release mode, tried
that and was able to fix a
> few
> >>> > bugs
> >>> > > > > after
> >>> > > > > > > >> > switching
> >>> > > > > > > >> > > to it. They were in that
PR with debug.assert
> >>> changes. Who
> >>> > > > > knows,
> >>> > > > > > > the
> >>> > > > > > > >> > > remaining failures might
still be related to that,
> but
> >>> > can't
> >>> > > > > > > >> reproduce it
> >>> > > > > > > >> > > locally.
> >>> > > > > > > >> > >
> >>> > > > > > > >> > > Laimis
> >>> > > > > > > >> > > On May 16, 2015 4:34 PM,
"Wyatt Barnett" <
> >>> > > > > wyatt.barnett@gmail.com
> >>> > > > > > >
> >>> > > > > > > >> > wrote:
> >>> > > > > > > >> > >
> >>> > > > > > > >> > > > Sorry about the blank
one -- getting used to
> google
> >>> > inbox
> >>> > > > here
> >>> > > > > > > and I
> >>> > > > > > > >> > > > misclicked.
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > > > Anyhow, I have a repro
or at least a rhyme and
> >>> reason --
> >>> > > > > > TeamCity
> >>> > > > > > > is
> >>> > > > > > > >> > > > running in release
mode and I think we have
> >>> difffering
> >>> > > > > behavior
> >>> > > > > > > >> there.
> >>> > > > > > > >> > If
> >>> > > > > > > >> > > > you switch your copy
of visual studio to release
> >>> mode
> >>> > you
> >>> > > > will
> >>> > > > > > get
> >>> > > > > > > >> the
> >>> > > > > > > >> > > same
> >>> > > > > > > >> > > > failures we are seeing
in TeamCity. Does that help
> >>> > narrow
> >>> > > it
> >>> > > > > > down
> >>> > > > > > > a
> >>> > > > > > > >> > bit?
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > > > On Sat, May 16, 2015
at 4:26 PM Wyatt Barnett <
> >>> > > > > > > >> wyatt.barnett@gmail.com
> >>> > > > > > > >> > >
> >>> > > > > > > >> > > > wrote:
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > > > >
> >>> > > > > > > >> > > > >
> >>> > > > > > > >> > > > > On Sat, May 16,
2015 at 3:22 PM Wyatt Barnett <
> >>> > > > > > > >> > wyatt.barnett@gmail.com
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > > > > wrote:
> >>> > > > > > > >> > > > >
> >>> > > > > > > >> > > > >> I agree with
Itamar -- it feels environmental.
> >>> I'll
> >>> > do
> >>> > > > some
> >>> > > > > > > >> digging
> >>> > > > > > > >> > > into
> >>> > > > > > > >> > > > >> the teamcity
output but I think the plan of
> >>> setting
> >>> > up
> >>> > > > some
> >>> > > > > > > extra
> >>> > > > > > > >> > > > verbose
> >>> > > > > > > >> > > > >> logging here
would make sense. I can set you up
> >>> with
> >>> > a
> >>> > > > > > separate
> >>> > > > > > > >> > build
> >>> > > > > > > >> > > > >> pointed at
your fork if that helps -- it will
> >>> keep
> >>> > the
> >>> > > > > > feedback
> >>> > > > > > > >> > cycle
> >>> > > > > > > >> > > > >> tighter.
The other thing we could do is
> >>> categorize
> >>> > the
> >>> > > > > tests
> >>> > > > > > > and
> >>> > > > > > > >> > focus
> >>> > > > > > > >> > > > that
> >>> > > > > > > >> > > > >> build at
running only that category so you
> don't
> >>> need
> >>> > > to
> >>> > > > > wait
> >>> > > > > > > on
> >>> > > > > > > >> the
> >>> > > > > > > >> > > > whole
> >>> > > > > > > >> > > > >> suite to
get responses. Let me know if you want
> >>> me to
> >>> > > > > proceed
> >>> > > > > > > >> there.
> >>> > > > > > > >> > > > >>
> >>> > > > > > > >> > > > >>
> >>> > > > > > > >> > > > >>
> >>> > > > > > > >> > > > >> On Sat, May
16, 2015 at 3:10 PM, Itamar
> >>> Syn-Hershko <
> >>> > > > > > > >> > > itamar@code972.com
> >>> > > > > > > >> > > > >
> >>> > > > > > > >> > > > >> wrote:
> >>> > > > > > > >> > > > >>
> >>> > > > > > > >> > > > >>> Yes,
that would be the best way to do this. On
> >>> Java
> >>> > > > > Lucene,
> >>> > > > > > > the
> >>> > > > > > > >> > > > >>> randomized
> >>> > > > > > > >> > > > >>> tests
framework allows you to re-use the
> random
> >>> seed
> >>> > > > > > > associated
> >>> > > > > > > >> > with
> >>> > > > > > > >> > > > the
> >>> > > > > > > >> > > > >>> failure,
but we are not there yet. Either
> way, I
> >>> > > suspect
> >>> > > > > > this
> >>> > > > > > > >> to be
> >>> > > > > > > >> > > an
> >>> > > > > > > >> > > > >>> environment
issue rather than a code path one.
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > > >>> --
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > > >>> Itamar
Syn-Hershko
> >>> > > > > > > >> > > > >>> http://code972.com
| @synhershko <
> >>> > > > > > > >> https://twitter.com/synhershko>
> >>> > > > > > > >> > > > >>> Freelance
Developer & Consultant
> >>> > > > > > > >> > > > >>> Lucene.NET
committer and PMC member
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > > >>> On Sat,
May 16, 2015 at 10:06 PM, Laimonas
> >>> Simutis <
> >>> > > > > > > >> > laimis@gmail.com
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > > > >>> wrote:
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > > >>> >
There are three tests that consistently fail
> >>> on TC
> >>> > > but
> >>> > > > > no
> >>> > > > > > > >> matter
> >>> > > > > > > >> > > how
> >>> > > > > > > >> > > > >>> many
> >>> > > > > > > >> > > > >>> >
times I try, I can't reproduce it locally.
> >>> These
> >>> > > tests
> >>> > > > > > are:
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
TestFuzzyQuery.TestTieBreaker
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > >
> >>> > > > > > > >> >
> >>> > > > > > > >>
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
TestSimpleExplanations.TestDMQ8
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > >
> >>> > > > > > > >> >
> >>> > > > > > > >>
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
TestTopDocsMerge.TestSort_2
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > >
> >>> > > > > > > >> >
> >>> > > > > > > >>
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
I would fix them if I could reproduce it --
> >>> and I
> >>> > am
> >>> > > > > > running
> >>> > > > > > > >> out
> >>> > > > > > > >> > of
> >>> > > > > > > >> > > > >>> ideas
> >>> > > > > > > >> > > > >>> >
how to do it. Even if I put them in a loop
> >>> running
> >>> > > > > > hundreds
> >>> > > > > > > of
> >>> > > > > > > >> > > > times, I
> >>> > > > > > > >> > > > >>> >
can't trigger the failure.
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
Anyone have any ideas how to go about
> >>> reproducing
> >>> > > it?
> >>> > > > I
> >>> > > > > am
> >>> > > > > > > >> > thinking
> >>> > > > > > > >> > > > to
> >>> > > > > > > >> > > > >>> push
> >>> > > > > > > >> > > > >>> >
very verbose code in a separate branch that
> >>> logs
> >>> > the
> >>> > > > > input
> >>> > > > > > > >> > values /
> >>> > > > > > > >> > > > >>> random
> >>> > > > > > > >> > > > >>> >
values that are used and see what happens.
> >>> > Checking
> >>> > > if
> >>> > > > > > > anyone
> >>> > > > > > > >> has
> >>> > > > > > > >> > > any
> >>> > > > > > > >> > > > >>> other
> >>> > > > > > > >> > > > >>> >
suggestions.
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
Thanks,
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>> >
Laimis
> >>> > > > > > > >> > > > >>> >
> >>> > > > > > > >> > > > >>>
> >>> > > > > > > >> > > > >>
> >>> > > > > > > >> > > > >>
> >>> > > > > > > >> > > >
> >>> > > > > > > >> > >
> >>> > > > > > > >> >
> >>> > > > > > > >>
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message