lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wyatt Barnett <wyatt.barn...@gmail.com>
Subject Re: Reproducing random test failures
Date Tue, 19 May 2015 14:54:09 GMT
Great. One thing just hit me -- would it be better for this to run as DEBUG
rather than RELEASE? I can flip that switch too . . .

On Tue, May 19, 2015 at 10:52 AM Laimonas Simutis <laimis@gmail.com> wrote:

> Thank you! Just kicked off the build. Let's see what it tells us :)
>
> On Tue, May 19, 2015 at 9:59 AM, Wyatt Barnett <wyatt.barnett@gmail.com>
> wrote:
>
> > Liamonas -- you should be all set, I just added
> >
> >
> http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests
> > which runs the core build with focused tests. Please ignore the build
> > number. Test category is more of a general setting for a build so there
> > isn't an easy checkbox to hit.
> >
> > If it makes more sense to re-point that at your github repo just say the
> > word and I'll make it so.
> >
> >
> > On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <laimis@gmail.com>
> wrote:
> >
> > > Wyatt,
> > >
> > > I have a branch pushed for this named "failingtests", it is now
> running a
> > > build on TC. Where does one specify which category of tests to run? I
> see
> > > in the settings tab "NUnit categories include/exclude" but don't see
> > where
> > > to actually modify these values. The tests I would like to run belong
> to
> > > category "Focus" :) Do you know where to change this?
> > >
> > > On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett <
> wyatt.barnett@gmail.com
> > >
> > > wrote:
> > >
> > > > Good to hear I checked the right box.
> > > >
> > > > I'll see what I can pull together when I get home in terms of debug
> > > output.
> > > > In terms of testing procedure what I was thinking is we make a new
> > > category
> > > > -- call it "Focus" and then configure a build looking at your fork
> > > > filtering for just those tests. You can then push away, fire off
> remote
> > > > builds and check the output yourself.
> > > >
> > > > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <laimis@gmail.com>
> > > wrote:
> > > >
> > > > > Wyatt,
> > > > >
> > > > > I see the new options on TC, thanks for that. I still haven't
> thought
> > > > about
> > > > > how I will go about capturing the failures exactly, but will give
> > you a
> > > > > shout if I need some help with TC configuration just for those
> runs.
> > > > >
> > > > > If you can reproduce any of those test failures locally, do you
> mind
> > > > > running them in VERBOSE mode (debug build without any other changes
> > > will
> > > > > do), and emailing the console output that you get? I might be too
> > > > > optimistic, but perhaps something there will stand out.
> > > > >
> > > > >
> > > > > Thanks again!
> > > > >
> > > > > Laimis
> > > > >
> > > > >
> > > > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <
> > > wyatt.barnett@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > For TestSort_2 -- It appears to be passing based on data at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
> > > > > > ; I am having locally reproducable problems on the others though.
> > > > > >
> > > > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <
> > > > wyatt.barnett@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Done -- you should now see a run button when you visit
> > > > > > >
> http://teamcity.codebetter.com/project.html?projectId=LuceneNet
> > > > > > >
> > > > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis <
> > > laimis@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Wyatt,
> > > > > > >>
> > > > > > >> Could you add me to the lucene.net group on TC? I have
a
> login
> > > > there,
> > > > > > >> username: laimis.
> > > > > > >>
> > > > > > >>
> > > > > > >> Thanks!
> > > > > > >>
> > > > > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett <
> > > > > wyatt.barnett@gmail.com
> > > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Sounds good Laimis. You will need to setup a login
to the
> > > > CodeBetter
> > > > > > >> > teamcity server and get added to the lucene.net
group if
> you
> > > > > haven't
> > > > > > >> > already. Let me know if you need help there too.
> > > > > > >> >
> > > > > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas Simutis
<
> > > > laimis@gmail.com>
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > > Wyatt,
> > > > > > >> > >
> > > > > > >> > > Sweet, I will let you know once I have a
branch out with
> > > > > additional
> > > > > > >> > logging
> > > > > > >> > > and separate category for tests that you
can configure to
> > run.
> > > > > > >> > >
> > > > > > >> > > Re: release mode, tried that and was able
to fix a few
> bugs
> > > > after
> > > > > > >> > switching
> > > > > > >> > > to it. They were in that PR with debug.assert
changes. Who
> > > > knows,
> > > > > > the
> > > > > > >> > > remaining failures might still be related
to that, but
> can't
> > > > > > >> reproduce it
> > > > > > >> > > locally.
> > > > > > >> > >
> > > > > > >> > > Laimis
> > > > > > >> > > On May 16, 2015 4:34 PM, "Wyatt Barnett"
<
> > > > wyatt.barnett@gmail.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> > >
> > > > > > >> > > > Sorry about the blank one -- getting
used to google
> inbox
> > > here
> > > > > > and I
> > > > > > >> > > > misclicked.
> > > > > > >> > > >
> > > > > > >> > > > Anyhow, I have a repro or at least a
rhyme and reason --
> > > > > TeamCity
> > > > > > is
> > > > > > >> > > > running in release mode and I think
we have difffering
> > > > behavior
> > > > > > >> there.
> > > > > > >> > If
> > > > > > >> > > > you switch your copy of visual studio
to release mode
> you
> > > will
> > > > > get
> > > > > > >> the
> > > > > > >> > > same
> > > > > > >> > > > failures we are seeing in TeamCity.
Does that help
> narrow
> > it
> > > > > down
> > > > > > a
> > > > > > >> > bit?
> > > > > > >> > > >
> > > > > > >> > > > On Sat, May 16, 2015 at 4:26 PM Wyatt
Barnett <
> > > > > > >> wyatt.barnett@gmail.com
> > > > > > >> > >
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > On Sat, May 16, 2015 at 3:22 PM
Wyatt Barnett <
> > > > > > >> > wyatt.barnett@gmail.com
> > > > > > >> > > >
> > > > > > >> > > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > >> I agree with Itamar -- it feels
environmental. I'll
> do
> > > some
> > > > > > >> digging
> > > > > > >> > > into
> > > > > > >> > > > >> the teamcity output but I think
the plan of setting
> up
> > > some
> > > > > > extra
> > > > > > >> > > > verbose
> > > > > > >> > > > >> logging here would make sense.
I can set you up with
> a
> > > > > separate
> > > > > > >> > build
> > > > > > >> > > > >> pointed at your fork if that
helps -- it will keep
> the
> > > > > feedback
> > > > > > >> > cycle
> > > > > > >> > > > >> tighter. The other thing we
could do is categorize
> the
> > > > tests
> > > > > > and
> > > > > > >> > focus
> > > > > > >> > > > that
> > > > > > >> > > > >> build at running only that
category so you don't need
> > to
> > > > wait
> > > > > > on
> > > > > > >> the
> > > > > > >> > > > whole
> > > > > > >> > > > >> suite to get responses. Let
me know if you want me to
> > > > proceed
> > > > > > >> there.
> > > > > > >> > > > >>
> > > > > > >> > > > >>
> > > > > > >> > > > >>
> > > > > > >> > > > >> On Sat, May 16, 2015 at 3:10
PM, Itamar Syn-Hershko <
> > > > > > >> > > itamar@code972.com
> > > > > > >> > > > >
> > > > > > >> > > > >> wrote:
> > > > > > >> > > > >>
> > > > > > >> > > > >>> Yes, that would be the
best way to do this. On Java
> > > > Lucene,
> > > > > > the
> > > > > > >> > > > >>> randomized
> > > > > > >> > > > >>> tests framework allows
you to re-use the random seed
> > > > > > associated
> > > > > > >> > with
> > > > > > >> > > > the
> > > > > > >> > > > >>> failure, but we are not
there yet. Either way, I
> > suspect
> > > > > this
> > > > > > >> to be
> > > > > > >> > > an
> > > > > > >> > > > >>> environment issue rather
than a code path one.
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> --
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> Itamar Syn-Hershko
> > > > > > >> > > > >>> http://code972.com | @synhershko
<
> > > > > > >> https://twitter.com/synhershko>
> > > > > > >> > > > >>> Freelance Developer &
Consultant
> > > > > > >> > > > >>> Lucene.NET committer and
PMC member
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> On Sat, May 16, 2015 at
10:06 PM, Laimonas Simutis <
> > > > > > >> > laimis@gmail.com
> > > > > > >> > > >
> > > > > > >> > > > >>> wrote:
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> > There are three tests
that consistently fail on TC
> > but
> > > > no
> > > > > > >> matter
> > > > > > >> > > how
> > > > > > >> > > > >>> many
> > > > > > >> > > > >>> > times I try, I can't
reproduce it locally. These
> > tests
> > > > > are:
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > TestSimpleExplanations.TestDMQ8
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > TestTopDocsMerge.TestSort_2
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > I would fix them if
I could reproduce it -- and I
> am
> > > > > running
> > > > > > >> out
> > > > > > >> > of
> > > > > > >> > > > >>> ideas
> > > > > > >> > > > >>> > how to do it. Even
if I put them in a loop running
> > > > > hundreds
> > > > > > of
> > > > > > >> > > > times, I
> > > > > > >> > > > >>> > can't trigger the
failure.
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > Anyone have any ideas
how to go about reproducing
> > it?
> > > I
> > > > am
> > > > > > >> > thinking
> > > > > > >> > > > to
> > > > > > >> > > > >>> push
> > > > > > >> > > > >>> > very verbose code
in a separate branch that logs
> the
> > > > input
> > > > > > >> > values /
> > > > > > >> > > > >>> random
> > > > > > >> > > > >>> > values that are used
and see what happens.
> Checking
> > if
> > > > > > anyone
> > > > > > >> has
> > > > > > >> > > any
> > > > > > >> > > > >>> other
> > > > > > >> > > > >>> > suggestions.
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > Thanks,
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>> > Laimis
> > > > > > >> > > > >>> >
> > > > > > >> > > > >>>
> > > > > > >> > > > >>
> > > > > > >> > > > >>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message