lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wyatt Barnett <wyatt.barn...@gmail.com>
Subject Re: Reproducing random test failures
Date Tue, 19 May 2015 18:02:43 GMT
The screenshot didn't come through so I'm kind of guessing here but I
switched to debug mode which also triggers VERBOSE to get more output.

This confirmed one of the things I saw locally earlier -- that some of the
semantics switching from debug to release (or non-verbose to verbose come
to think of it) makes those tests fail for some reason. The other challenge
is that the output is so verbose that having verbose on makes the 4th test
run -- I think the Test_Merge tests -- real, real slow. Like 50+ minute
test run slow (
http://teamcity.codebetter.com/viewLog.html?buildId=191532&tab=buildResultsDiv&buildTypeId=LuceneNet_LuceneNetCoreFailingTests)
compared to ~1 minute test runs.

Let me see if I can get output without triggering debug mode when I get
home, need to get to the right PC to make that happen.

On Tue, May 19, 2015 at 11:56 AM Laimonas Simutis <laimis@gmail.com> wrote:

> Wyatt,
>
> Would it be possible to pass that in through the configuration? I tried
> passing in verbose flag through but did not appear to work. The same with
> configuration. Here is the properties section I am referring to:
>
> [image: Inline image 1]
>
> Do you know if it something that is possible to have for the TC builds?
>
> On Tue, May 19, 2015 at 10:54 AM, Wyatt Barnett <wyatt.barnett@gmail.com>
> wrote:
>
>> Great. One thing just hit me -- would it be better for this to run as
>> DEBUG
>> rather than RELEASE? I can flip that switch too . . .
>>
>> On Tue, May 19, 2015 at 10:52 AM Laimonas Simutis <laimis@gmail.com>
>> wrote:
>>
>> > Thank you! Just kicked off the build. Let's see what it tells us :)
>> >
>> > On Tue, May 19, 2015 at 9:59 AM, Wyatt Barnett <wyatt.barnett@gmail.com
>> >
>> > wrote:
>> >
>> > > Liamonas -- you should be all set, I just added
>> > >
>> > >
>> >
>> http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests
>> > > which runs the core build with focused tests. Please ignore the build
>> > > number. Test category is more of a general setting for a build so
>> there
>> > > isn't an easy checkbox to hit.
>> > >
>> > > If it makes more sense to re-point that at your github repo just say
>> the
>> > > word and I'll make it so.
>> > >
>> > >
>> > > On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <laimis@gmail.com>
>> > wrote:
>> > >
>> > > > Wyatt,
>> > > >
>> > > > I have a branch pushed for this named "failingtests", it is now
>> > running a
>> > > > build on TC. Where does one specify which category of tests to run?
>> I
>> > see
>> > > > in the settings tab "NUnit categories include/exclude" but don't see
>> > > where
>> > > > to actually modify these values. The tests I would like to run
>> belong
>> > to
>> > > > category "Focus" :) Do you know where to change this?
>> > > >
>> > > > On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett <
>> > wyatt.barnett@gmail.com
>> > > >
>> > > > wrote:
>> > > >
>> > > > > Good to hear I checked the right box.
>> > > > >
>> > > > > I'll see what I can pull together when I get home in terms of
>> debug
>> > > > output.
>> > > > > In terms of testing procedure what I was thinking is we make
a new
>> > > > category
>> > > > > -- call it "Focus" and then configure a build looking at your
fork
>> > > > > filtering for just those tests. You can then push away, fire
off
>> > remote
>> > > > > builds and check the output yourself.
>> > > > >
>> > > > > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <
>> laimis@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > > Wyatt,
>> > > > > >
>> > > > > > I see the new options on TC, thanks for that. I still haven't
>> > thought
>> > > > > about
>> > > > > > how I will go about capturing the failures exactly, but
will
>> give
>> > > you a
>> > > > > > shout if I need some help with TC configuration just for
those
>> > runs.
>> > > > > >
>> > > > > > If you can reproduce any of those test failures locally,
do you
>> > mind
>> > > > > > running them in VERBOSE mode (debug build without any other
>> changes
>> > > > will
>> > > > > > do), and emailing the console output that you get? I might
be
>> too
>> > > > > > optimistic, but perhaps something there will stand out.
>> > > > > >
>> > > > > >
>> > > > > > Thanks again!
>> > > > > >
>> > > > > > Laimis
>> > > > > >
>> > > > > >
>> > > > > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <
>> > > > wyatt.barnett@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > For TestSort_2 -- It appears to be passing based on
data at
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
>> > > > > > > ; I am having locally reproducable problems on the
others
>> though.
>> > > > > > >
>> > > > > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett <
>> > > > > wyatt.barnett@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Done -- you should now see a run button when you
visit
>> > > > > > > >
>> > http://teamcity.codebetter.com/project.html?projectId=LuceneNet
>> > > > > > > >
>> > > > > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas Simutis
<
>> > > > laimis@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > >> Wyatt,
>> > > > > > > >>
>> > > > > > > >> Could you add me to the lucene.net group on
TC? I have a
>> > login
>> > > > > there,
>> > > > > > > >> username: laimis.
>> > > > > > > >>
>> > > > > > > >>
>> > > > > > > >> Thanks!
>> > > > > > > >>
>> > > > > > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt Barnett
<
>> > > > > > wyatt.barnett@gmail.com
>> > > > > > > >
>> > > > > > > >> wrote:
>> > > > > > > >>
>> > > > > > > >> > Sounds good Laimis. You will need to
setup a login to the
>> > > > > CodeBetter
>> > > > > > > >> > teamcity server and get added to the
lucene.net group if
>> > you
>> > > > > > haven't
>> > > > > > > >> > already. Let me know if you need help
there too.
>> > > > > > > >> >
>> > > > > > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas
Simutis <
>> > > > > laimis@gmail.com>
>> > > > > > > >> wrote:
>> > > > > > > >> >
>> > > > > > > >> > > Wyatt,
>> > > > > > > >> > >
>> > > > > > > >> > > Sweet, I will let you know once
I have a branch out
>> with
>> > > > > > additional
>> > > > > > > >> > logging
>> > > > > > > >> > > and separate category for tests
that you can configure
>> to
>> > > run.
>> > > > > > > >> > >
>> > > > > > > >> > > Re: release mode, tried that and
was able to fix a few
>> > bugs
>> > > > > after
>> > > > > > > >> > switching
>> > > > > > > >> > > to it. They were in that PR with
debug.assert changes.
>> Who
>> > > > > knows,
>> > > > > > > the
>> > > > > > > >> > > remaining failures might still be
related to that, but
>> > can't
>> > > > > > > >> reproduce it
>> > > > > > > >> > > locally.
>> > > > > > > >> > >
>> > > > > > > >> > > Laimis
>> > > > > > > >> > > On May 16, 2015 4:34 PM, "Wyatt
Barnett" <
>> > > > > wyatt.barnett@gmail.com
>> > > > > > >
>> > > > > > > >> > wrote:
>> > > > > > > >> > >
>> > > > > > > >> > > > Sorry about the blank one --
getting used to google
>> > inbox
>> > > > here
>> > > > > > > and I
>> > > > > > > >> > > > misclicked.
>> > > > > > > >> > > >
>> > > > > > > >> > > > Anyhow, I have a repro or at
least a rhyme and
>> reason --
>> > > > > > TeamCity
>> > > > > > > is
>> > > > > > > >> > > > running in release mode and
I think we have
>> difffering
>> > > > > behavior
>> > > > > > > >> there.
>> > > > > > > >> > If
>> > > > > > > >> > > > you switch your copy of visual
studio to release mode
>> > you
>> > > > will
>> > > > > > get
>> > > > > > > >> the
>> > > > > > > >> > > same
>> > > > > > > >> > > > failures we are seeing in TeamCity.
Does that help
>> > narrow
>> > > it
>> > > > > > down
>> > > > > > > a
>> > > > > > > >> > bit?
>> > > > > > > >> > > >
>> > > > > > > >> > > > On Sat, May 16, 2015 at 4:26
PM Wyatt Barnett <
>> > > > > > > >> wyatt.barnett@gmail.com
>> > > > > > > >> > >
>> > > > > > > >> > > > wrote:
>> > > > > > > >> > > >
>> > > > > > > >> > > > >
>> > > > > > > >> > > > >
>> > > > > > > >> > > > > On Sat, May 16, 2015 at
3:22 PM Wyatt Barnett <
>> > > > > > > >> > wyatt.barnett@gmail.com
>> > > > > > > >> > > >
>> > > > > > > >> > > > > wrote:
>> > > > > > > >> > > > >
>> > > > > > > >> > > > >> I agree with Itamar
-- it feels environmental.
>> I'll
>> > do
>> > > > some
>> > > > > > > >> digging
>> > > > > > > >> > > into
>> > > > > > > >> > > > >> the teamcity output
but I think the plan of
>> setting
>> > up
>> > > > some
>> > > > > > > extra
>> > > > > > > >> > > > verbose
>> > > > > > > >> > > > >> logging here would
make sense. I can set you up
>> with
>> > a
>> > > > > > separate
>> > > > > > > >> > build
>> > > > > > > >> > > > >> pointed at your fork
if that helps -- it will keep
>> > the
>> > > > > > feedback
>> > > > > > > >> > cycle
>> > > > > > > >> > > > >> tighter. The other
thing we could do is categorize
>> > the
>> > > > > tests
>> > > > > > > and
>> > > > > > > >> > focus
>> > > > > > > >> > > > that
>> > > > > > > >> > > > >> build at running only
that category so you don't
>> need
>> > > to
>> > > > > wait
>> > > > > > > on
>> > > > > > > >> the
>> > > > > > > >> > > > whole
>> > > > > > > >> > > > >> suite to get responses.
Let me know if you want
>> me to
>> > > > > proceed
>> > > > > > > >> there.
>> > > > > > > >> > > > >>
>> > > > > > > >> > > > >>
>> > > > > > > >> > > > >>
>> > > > > > > >> > > > >> On Sat, May 16, 2015
at 3:10 PM, Itamar
>> Syn-Hershko <
>> > > > > > > >> > > itamar@code972.com
>> > > > > > > >> > > > >
>> > > > > > > >> > > > >> wrote:
>> > > > > > > >> > > > >>
>> > > > > > > >> > > > >>> Yes, that would
be the best way to do this. On
>> Java
>> > > > > Lucene,
>> > > > > > > the
>> > > > > > > >> > > > >>> randomized
>> > > > > > > >> > > > >>> tests framework
allows you to re-use the random
>> seed
>> > > > > > > associated
>> > > > > > > >> > with
>> > > > > > > >> > > > the
>> > > > > > > >> > > > >>> failure, but we
are not there yet. Either way, I
>> > > suspect
>> > > > > > this
>> > > > > > > >> to be
>> > > > > > > >> > > an
>> > > > > > > >> > > > >>> environment issue
rather than a code path one.
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > > >>> --
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > > >>> Itamar Syn-Hershko
>> > > > > > > >> > > > >>> http://code972.com
| @synhershko <
>> > > > > > > >> https://twitter.com/synhershko>
>> > > > > > > >> > > > >>> Freelance Developer
& Consultant
>> > > > > > > >> > > > >>> Lucene.NET committer
and PMC member
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > > >>> On Sat, May 16,
2015 at 10:06 PM, Laimonas
>> Simutis <
>> > > > > > > >> > laimis@gmail.com
>> > > > > > > >> > > >
>> > > > > > > >> > > > >>> wrote:
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > > >>> > There are
three tests that consistently fail
>> on TC
>> > > but
>> > > > > no
>> > > > > > > >> matter
>> > > > > > > >> > > how
>> > > > > > > >> > > > >>> many
>> > > > > > > >> > > > >>> > times I try,
I can't reproduce it locally.
>> These
>> > > tests
>> > > > > > are:
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > >
>> > > > > > > >> > >
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > TestSimpleExplanations.TestDMQ8
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > >
>> > > > > > > >> > >
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > TestTopDocsMerge.TestSort_2
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > >
>> > > > > > > >> > >
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > I would fix
them if I could reproduce it --
>> and I
>> > am
>> > > > > > running
>> > > > > > > >> out
>> > > > > > > >> > of
>> > > > > > > >> > > > >>> ideas
>> > > > > > > >> > > > >>> > how to do
it. Even if I put them in a loop
>> running
>> > > > > > hundreds
>> > > > > > > of
>> > > > > > > >> > > > times, I
>> > > > > > > >> > > > >>> > can't trigger
the failure.
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > Anyone have
any ideas how to go about
>> reproducing
>> > > it?
>> > > > I
>> > > > > am
>> > > > > > > >> > thinking
>> > > > > > > >> > > > to
>> > > > > > > >> > > > >>> push
>> > > > > > > >> > > > >>> > very verbose
code in a separate branch that
>> logs
>> > the
>> > > > > input
>> > > > > > > >> > values /
>> > > > > > > >> > > > >>> random
>> > > > > > > >> > > > >>> > values that
are used and see what happens.
>> > Checking
>> > > if
>> > > > > > > anyone
>> > > > > > > >> has
>> > > > > > > >> > > any
>> > > > > > > >> > > > >>> other
>> > > > > > > >> > > > >>> > suggestions.
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > Thanks,
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>> > Laimis
>> > > > > > > >> > > > >>> >
>> > > > > > > >> > > > >>>
>> > > > > > > >> > > > >>
>> > > > > > > >> > > > >>
>> > > > > > > >> > > >
>> > > > > > > >> > >
>> > > > > > > >> >
>> > > > > > > >>
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message