lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wyatt Barnett <wyatt.barn...@gmail.com>
Subject Re: Reproducing random test failures
Date Thu, 21 May 2015 15:44:31 GMT
I found a few minutes to play with things.

First, the general technique to switching things to running under RELEASE
vs DEBUG while keeping VERBOSE output on is to modify line 299 in
LuceneTestCase.cs to be "true" -- the preprocessor directives there are
basically tying DEBUG and VERBOSE together.

Output for TestTieBreaker can be found at http://pastebin.com/VXQYF32T
Output for TestDMQ8 can be found at http://pastebin.com/WYS07KqB
Output for TestSort_2 can be downloaded from
https://drive.google.com/file/d/0By01KKFIe0LEX1NubW1mRlVMVWM/view?usp=sharing

I hope this helps get to the bottom of this one, let me know if you need
further assistance.

On Tue, May 19, 2015 at 2:02 PM Wyatt Barnett <wyatt.barnett@gmail.com>
wrote:

> The screenshot didn't come through so I'm kind of guessing here but I
> switched to debug mode which also triggers VERBOSE to get more output.
>
> This confirmed one of the things I saw locally earlier -- that some of the
> semantics switching from debug to release (or non-verbose to verbose come
> to think of it) makes those tests fail for some reason. The other challenge
> is that the output is so verbose that having verbose on makes the 4th test
> run -- I think the Test_Merge tests -- real, real slow. Like 50+ minute
> test run slow (
> http://teamcity.codebetter.com/viewLog.html?buildId=191532&tab=buildResultsDiv&buildTypeId=LuceneNet_LuceneNetCoreFailingTests)
> compared to ~1 minute test runs.
>
> Let me see if I can get output without triggering debug mode when I get
> home, need to get to the right PC to make that happen.
>
> On Tue, May 19, 2015 at 11:56 AM Laimonas Simutis <laimis@gmail.com>
> wrote:
>
>> Wyatt,
>>
>> Would it be possible to pass that in through the configuration? I tried
>> passing in verbose flag through but did not appear to work. The same with
>> configuration. Here is the properties section I am referring to:
>>
>> [image: Inline image 1]
>>
>> Do you know if it something that is possible to have for the TC builds?
>>
>> On Tue, May 19, 2015 at 10:54 AM, Wyatt Barnett <wyatt.barnett@gmail.com>
>> wrote:
>>
>>> Great. One thing just hit me -- would it be better for this to run as
>>> DEBUG
>>> rather than RELEASE? I can flip that switch too . . .
>>>
>>> On Tue, May 19, 2015 at 10:52 AM Laimonas Simutis <laimis@gmail.com>
>>> wrote:
>>>
>>> > Thank you! Just kicked off the build. Let's see what it tells us :)
>>> >
>>> > On Tue, May 19, 2015 at 9:59 AM, Wyatt Barnett <
>>> wyatt.barnett@gmail.com>
>>> > wrote:
>>> >
>>> > > Liamonas -- you should be all set, I just added
>>> > >
>>> > >
>>> >
>>> http://teamcity.codebetter.com/viewType.html?buildTypeId=LuceneNet_LuceneNetCoreFailingTests
>>> > > which runs the core build with focused tests. Please ignore the build
>>> > > number. Test category is more of a general setting for a build so
>>> there
>>> > > isn't an easy checkbox to hit.
>>> > >
>>> > > If it makes more sense to re-point that at your github repo just say
>>> the
>>> > > word and I'll make it so.
>>> > >
>>> > >
>>> > > On Mon, May 18, 2015 at 8:10 PM Laimonas Simutis <laimis@gmail.com>
>>> > wrote:
>>> > >
>>> > > > Wyatt,
>>> > > >
>>> > > > I have a branch pushed for this named "failingtests", it is now
>>> > running a
>>> > > > build on TC. Where does one specify which category of tests to
>>> run? I
>>> > see
>>> > > > in the settings tab "NUnit categories include/exclude" but don't
>>> see
>>> > > where
>>> > > > to actually modify these values. The tests I would like to run
>>> belong
>>> > to
>>> > > > category "Focus" :) Do you know where to change this?
>>> > > >
>>> > > > On Mon, May 18, 2015 at 10:49 AM, Wyatt Barnett <
>>> > wyatt.barnett@gmail.com
>>> > > >
>>> > > > wrote:
>>> > > >
>>> > > > > Good to hear I checked the right box.
>>> > > > >
>>> > > > > I'll see what I can pull together when I get home in terms
of
>>> debug
>>> > > > output.
>>> > > > > In terms of testing procedure what I was thinking is we make
a
>>> new
>>> > > > category
>>> > > > > -- call it "Focus" and then configure a build looking at
your
>>> fork
>>> > > > > filtering for just those tests. You can then push away, fire
off
>>> > remote
>>> > > > > builds and check the output yourself.
>>> > > > >
>>> > > > > On Sun, May 17, 2015 at 8:50 PM Laimonas Simutis <
>>> laimis@gmail.com>
>>> > > > wrote:
>>> > > > >
>>> > > > > > Wyatt,
>>> > > > > >
>>> > > > > > I see the new options on TC, thanks for that. I still
haven't
>>> > thought
>>> > > > > about
>>> > > > > > how I will go about capturing the failures exactly,
but will
>>> give
>>> > > you a
>>> > > > > > shout if I need some help with TC configuration just
for those
>>> > runs.
>>> > > > > >
>>> > > > > > If you can reproduce any of those test failures locally,
do you
>>> > mind
>>> > > > > > running them in VERBOSE mode (debug build without any
other
>>> changes
>>> > > > will
>>> > > > > > do), and emailing the console output that you get? I
might be
>>> too
>>> > > > > > optimistic, but perhaps something there will stand out.
>>> > > > > >
>>> > > > > >
>>> > > > > > Thanks again!
>>> > > > > >
>>> > > > > > Laimis
>>> > > > > >
>>> > > > > >
>>> > > > > > On Sun, May 17, 2015 at 8:18 AM, Wyatt Barnett <
>>> > > > wyatt.barnett@gmail.com>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > For TestSort_2 -- It appears to be passing based
on data at
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> http://teamcity.codebetter.com/project.html?projectId=LuceneNet&testNameId=-8365680837810961892&tab=testDetails
>>> > > > > > > ; I am having locally reproducable problems on
the others
>>> though.
>>> > > > > > >
>>> > > > > > > On Sun, May 17, 2015 at 7:55 AM, Wyatt Barnett
<
>>> > > > > wyatt.barnett@gmail.com>
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > Done -- you should now see a run button when
you visit
>>> > > > > > > >
>>> > http://teamcity.codebetter.com/project.html?projectId=LuceneNet
>>> > > > > > > >
>>> > > > > > > > On Sat, May 16, 2015 at 8:02 PM, Laimonas
Simutis <
>>> > > > laimis@gmail.com>
>>> > > > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > >> Wyatt,
>>> > > > > > > >>
>>> > > > > > > >> Could you add me to the lucene.net group
on TC? I have a
>>> > login
>>> > > > > there,
>>> > > > > > > >> username: laimis.
>>> > > > > > > >>
>>> > > > > > > >>
>>> > > > > > > >> Thanks!
>>> > > > > > > >>
>>> > > > > > > >> On Sat, May 16, 2015 at 6:15 PM, Wyatt
Barnett <
>>> > > > > > wyatt.barnett@gmail.com
>>> > > > > > > >
>>> > > > > > > >> wrote:
>>> > > > > > > >>
>>> > > > > > > >> > Sounds good Laimis. You will need
to setup a login to
>>> the
>>> > > > > CodeBetter
>>> > > > > > > >> > teamcity server and get added to
the lucene.net group
>>> if
>>> > you
>>> > > > > > haven't
>>> > > > > > > >> > already. Let me know if you need
help there too.
>>> > > > > > > >> >
>>> > > > > > > >> > On Sat, May 16, 2015 at 4:52 PM Laimonas
Simutis <
>>> > > > > laimis@gmail.com>
>>> > > > > > > >> wrote:
>>> > > > > > > >> >
>>> > > > > > > >> > > Wyatt,
>>> > > > > > > >> > >
>>> > > > > > > >> > > Sweet, I will let you know once
I have a branch out
>>> with
>>> > > > > > additional
>>> > > > > > > >> > logging
>>> > > > > > > >> > > and separate category for tests
that you can
>>> configure to
>>> > > run.
>>> > > > > > > >> > >
>>> > > > > > > >> > > Re: release mode, tried that
and was able to fix a few
>>> > bugs
>>> > > > > after
>>> > > > > > > >> > switching
>>> > > > > > > >> > > to it. They were in that PR
with debug.assert
>>> changes. Who
>>> > > > > knows,
>>> > > > > > > the
>>> > > > > > > >> > > remaining failures might still
be related to that, but
>>> > can't
>>> > > > > > > >> reproduce it
>>> > > > > > > >> > > locally.
>>> > > > > > > >> > >
>>> > > > > > > >> > > Laimis
>>> > > > > > > >> > > On May 16, 2015 4:34 PM, "Wyatt
Barnett" <
>>> > > > > wyatt.barnett@gmail.com
>>> > > > > > >
>>> > > > > > > >> > wrote:
>>> > > > > > > >> > >
>>> > > > > > > >> > > > Sorry about the blank one
-- getting used to google
>>> > inbox
>>> > > > here
>>> > > > > > > and I
>>> > > > > > > >> > > > misclicked.
>>> > > > > > > >> > > >
>>> > > > > > > >> > > > Anyhow, I have a repro
or at least a rhyme and
>>> reason --
>>> > > > > > TeamCity
>>> > > > > > > is
>>> > > > > > > >> > > > running in release mode
and I think we have
>>> difffering
>>> > > > > behavior
>>> > > > > > > >> there.
>>> > > > > > > >> > If
>>> > > > > > > >> > > > you switch your copy of
visual studio to release
>>> mode
>>> > you
>>> > > > will
>>> > > > > > get
>>> > > > > > > >> the
>>> > > > > > > >> > > same
>>> > > > > > > >> > > > failures we are seeing
in TeamCity. Does that help
>>> > narrow
>>> > > it
>>> > > > > > down
>>> > > > > > > a
>>> > > > > > > >> > bit?
>>> > > > > > > >> > > >
>>> > > > > > > >> > > > On Sat, May 16, 2015 at
4:26 PM Wyatt Barnett <
>>> > > > > > > >> wyatt.barnett@gmail.com
>>> > > > > > > >> > >
>>> > > > > > > >> > > > wrote:
>>> > > > > > > >> > > >
>>> > > > > > > >> > > > >
>>> > > > > > > >> > > > >
>>> > > > > > > >> > > > > On Sat, May 16, 2015
at 3:22 PM Wyatt Barnett <
>>> > > > > > > >> > wyatt.barnett@gmail.com
>>> > > > > > > >> > > >
>>> > > > > > > >> > > > > wrote:
>>> > > > > > > >> > > > >
>>> > > > > > > >> > > > >> I agree with Itamar
-- it feels environmental.
>>> I'll
>>> > do
>>> > > > some
>>> > > > > > > >> digging
>>> > > > > > > >> > > into
>>> > > > > > > >> > > > >> the teamcity output
but I think the plan of
>>> setting
>>> > up
>>> > > > some
>>> > > > > > > extra
>>> > > > > > > >> > > > verbose
>>> > > > > > > >> > > > >> logging here would
make sense. I can set you up
>>> with
>>> > a
>>> > > > > > separate
>>> > > > > > > >> > build
>>> > > > > > > >> > > > >> pointed at your
fork if that helps -- it will
>>> keep
>>> > the
>>> > > > > > feedback
>>> > > > > > > >> > cycle
>>> > > > > > > >> > > > >> tighter. The other
thing we could do is
>>> categorize
>>> > the
>>> > > > > tests
>>> > > > > > > and
>>> > > > > > > >> > focus
>>> > > > > > > >> > > > that
>>> > > > > > > >> > > > >> build at running
only that category so you don't
>>> need
>>> > > to
>>> > > > > wait
>>> > > > > > > on
>>> > > > > > > >> the
>>> > > > > > > >> > > > whole
>>> > > > > > > >> > > > >> suite to get responses.
Let me know if you want
>>> me to
>>> > > > > proceed
>>> > > > > > > >> there.
>>> > > > > > > >> > > > >>
>>> > > > > > > >> > > > >>
>>> > > > > > > >> > > > >>
>>> > > > > > > >> > > > >> On Sat, May 16,
2015 at 3:10 PM, Itamar
>>> Syn-Hershko <
>>> > > > > > > >> > > itamar@code972.com
>>> > > > > > > >> > > > >
>>> > > > > > > >> > > > >> wrote:
>>> > > > > > > >> > > > >>
>>> > > > > > > >> > > > >>> Yes, that
would be the best way to do this. On
>>> Java
>>> > > > > Lucene,
>>> > > > > > > the
>>> > > > > > > >> > > > >>> randomized
>>> > > > > > > >> > > > >>> tests framework
allows you to re-use the random
>>> seed
>>> > > > > > > associated
>>> > > > > > > >> > with
>>> > > > > > > >> > > > the
>>> > > > > > > >> > > > >>> failure, but
we are not there yet. Either way, I
>>> > > suspect
>>> > > > > > this
>>> > > > > > > >> to be
>>> > > > > > > >> > > an
>>> > > > > > > >> > > > >>> environment
issue rather than a code path one.
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > > >>> --
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > > >>> Itamar Syn-Hershko
>>> > > > > > > >> > > > >>> http://code972.com
| @synhershko <
>>> > > > > > > >> https://twitter.com/synhershko>
>>> > > > > > > >> > > > >>> Freelance
Developer & Consultant
>>> > > > > > > >> > > > >>> Lucene.NET
committer and PMC member
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > > >>> On Sat, May
16, 2015 at 10:06 PM, Laimonas
>>> Simutis <
>>> > > > > > > >> > laimis@gmail.com
>>> > > > > > > >> > > >
>>> > > > > > > >> > > > >>> wrote:
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > > >>> > There
are three tests that consistently fail
>>> on TC
>>> > > but
>>> > > > > no
>>> > > > > > > >> matter
>>> > > > > > > >> > > how
>>> > > > > > > >> > > > >>> many
>>> > > > > > > >> > > > >>> > times
I try, I can't reproduce it locally.
>>> These
>>> > > tests
>>> > > > > > are:
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > TestFuzzyQuery.TestTieBreaker
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > >
>>> > > > > > > >> > >
>>> > > > > > > >> >
>>> > > > > > > >>
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-6371662534320583798
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > TestSimpleExplanations.TestDMQ8
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > >
>>> > > > > > > >> > >
>>> > > > > > > >> >
>>> > > > > > > >>
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId5725706748293106127
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > TestTopDocsMerge.TestSort_2
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > >
>>> > > > > > > >> > >
>>> > > > > > > >> >
>>> > > > > > > >>
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> http://teamcity.codebetter.com/viewLog.html?buildId=191298&tab=buildResultsDiv&buildTypeId=LuceneNet_Core#testNameId-8365680837810961892
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > I would
fix them if I could reproduce it --
>>> and I
>>> > am
>>> > > > > > running
>>> > > > > > > >> out
>>> > > > > > > >> > of
>>> > > > > > > >> > > > >>> ideas
>>> > > > > > > >> > > > >>> > how to
do it. Even if I put them in a loop
>>> running
>>> > > > > > hundreds
>>> > > > > > > of
>>> > > > > > > >> > > > times, I
>>> > > > > > > >> > > > >>> > can't
trigger the failure.
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > Anyone
have any ideas how to go about
>>> reproducing
>>> > > it?
>>> > > > I
>>> > > > > am
>>> > > > > > > >> > thinking
>>> > > > > > > >> > > > to
>>> > > > > > > >> > > > >>> push
>>> > > > > > > >> > > > >>> > very
verbose code in a separate branch that
>>> logs
>>> > the
>>> > > > > input
>>> > > > > > > >> > values /
>>> > > > > > > >> > > > >>> random
>>> > > > > > > >> > > > >>> > values
that are used and see what happens.
>>> > Checking
>>> > > if
>>> > > > > > > anyone
>>> > > > > > > >> has
>>> > > > > > > >> > > any
>>> > > > > > > >> > > > >>> other
>>> > > > > > > >> > > > >>> > suggestions.
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > Thanks,
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>> > Laimis
>>> > > > > > > >> > > > >>> >
>>> > > > > > > >> > > > >>>
>>> > > > > > > >> > > > >>
>>> > > > > > > >> > > > >>
>>> > > > > > > >> > > >
>>> > > > > > > >> > >
>>> > > > > > > >> >
>>> > > > > > > >>
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message