lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wyatt Barnett <wyatt.barn...@gmail.com>
Subject Re: Hanging TC Builds
Date Thu, 12 Jan 2017 14:37:55 GMT
I've been on the road for the last few weeks so I haven't had time to look
into this much. I've got consecutive 3 day weekends coming up so I should
be able to do some tire kicking shortly.

That said, JetBrains did get back to me:

Hi, I tried to rerun successfull build #765
<https://teamcity-support.jetbrains.com/hc/requests/765> (https://teamcity
.jetbrains.com/viewLog.html?buildId=950000&buildTypeId=LuceneNet_LuceneNetNightly&tab=buildResultsDiv)
and go the execution exception (https://teamcity
.jetbrains.com/viewLog.html?buildId=956627&buildTypeId=LuceneNet_LuceneNetNightly&tab=buildResultsDiv).
Comparing parameters gave nothing to me, but after comparing logs I found
this:

[16:31:32] : [Step 3/5] 2016-12-28 13:31:32,116 [824] ERROR JetBrains.
TeamCity.NUnitLauncher.ErrorLoggerImpl - Unhandled exception in remote
appdomain: System.Exception: Expected: 1
[16:31:32] : [Step 3/5] But was: 0
[16:31:32] : [Step 3/5] ---> NUnit.Framework.AssertionException: Expected: 1
[16:31:32] : [Step 3/5] But was: 0
[16:31:32] : [Step 3/5]
[16:31:32] : [Step 3/5] at NUnit.Framework.Assert.That(Object actual,
IResolveConstraint expression, String message, Object[] args)
[16:31:32] : [Step 3/5] at NUnit.Framework.Assert.AreEqual(Int32 expected,
Int32 actual)
[16:31:32] : [Step 3/5] at
Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocument(Term
id, IEnumerable`1 doc) in
C:\BuildAgent\work\5f08f6c5ab8fbc83\src\Lucene.Net.Tests\core\Search\TestControlledRealTimeReopenThread.cs:line
219
[16:31:32] : [Step 3/5] at
Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run()
in
After that point we started ramming into the timeout repeatedly.

My suspicion is a test fails and manages to lock up the disk in a way that
kills performance, making subsequent tests timeout and eventually leading
to the whole suite timing out. One thing to keep in mind is that cloud
stuff is comparable to your desktop in most cases except disk I/O where,
unless one is getting very expensive dedicated SSDs, your PC is probably
orders of magnitude better.

Anyhow, as a way forward I think it might help if someone could walk
through the listed lines above and see if something jumps out at you. I'm
going to try and stand something up at home to simulate poor cloud disk i/o
and see if I can deterministically recreate the slowdowns as well.

Happy new year!

On Mon, Jan 2, 2017 at 7:42 AM Shad Storhaug <shad@shadstorhaug.com> wrote:

> FYI, I am having similar issues on my branch of #191. The following tests
> are causing NUnit to either crash or not finish. Perhaps these same tests
> are causing issues in master. Excluding them from running during the build
> might be a temporary workaround.
>
> - Lucene.Net.Analysis.Th.TestThaiAnalyzer.TestRandomHugeStrings
> - Lucene.Net.Index.TestByteSlices
> - Lucene.Net.Search.TestSearchAfter.TestQueries
> - Lucene.Net.Util.Fst.TestFSTs.TestRandomWords
>
> Thanks,
> Shad Storhaug (NightOwl888)
>
> -----Original Message-----
> From: itamar.synhershko@gmail.com [mailto:itamar.synhershko@gmail.com] On
> Behalf Of Itamar Syn-Hershko
> Sent: Thursday, December 29, 2016 11:47 PM
> To: dev@lucenenet.apache.org
> Subject: Re: Hanging TC Builds
>
> Hopefully it's the tests code that locks up and not the core - waiting to
> hear news.. Thanks!
>
> --
>
> Itamar Syn-Hershko
> Freelance Developer & Consultant
> Elasticsearch Consulting Partner
> Microsoft MVP | Lucene.NET PMC
> http://code972.com | @synhershko <https://twitter.com/synhershko>
> http://BigDataBoutique.co.il/
>
> On Sat, Dec 24, 2016 at 5:12 AM, Wyatt Barnett <wyatt.barnett@gmail.com>
> wrote:
>
> > Hi guys -- just to let everyone know I'm aware that the builds seem to
> > be hanging on the CI server. I'm working on figuring out which way is
> > up. From what I can tell it seems like something locks up and the
> > tests start timing out until the entire stack melts down.
> >
> > Will update when I have updates. Happy holidays!
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message