lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shad Storhaug (Jira)" <>
Subject [jira] [Commented] (LUCENENET-627) Port RandomizedTesting Test Runner
Date Tue, 03 Dec 2019 14:25:00 GMT


Shad Storhaug commented on LUCENENET-627:

After updating the test framework to utilize the {{NUnit.Framework.TestContext.CurrentContext.Random}}
property, I noticed that random test failures now repeat reliably until the next build. Whether
or not there is a way to lock in that behavior from one build to the next, this is probably
adequate enough to avoid porting the entire RandomizedTesting framework.

That said, it would probably make sense to create a new library with the random generators
so they can be shared without a reference to our test framework and all of the dependencies
that come with it. We should also migrate the extension methods for the {{Random}} class.

> Port RandomizedTesting Test Runner
> ----------------------------------
>                 Key: LUCENENET-627
>                 URL:
>             Project: Lucene.Net
>          Issue Type: Task
>          Components: Lucene.Net.TestFramework
>    Affects Versions: Lucene.Net 4.8.0
>            Reporter: Shad Storhaug
>            Priority: Major
>              Labels: up-for-grabs
>   Original Estimate: 150h
>  Remaining Estimate: 150h
> While the test framework can run with the standard NUnit test runner and we have managed
to make it work, it is missing the ability to reproduce a random test failure, which is crucial
for debugging some of the more complicated random tests. Lucene uses a custom JUnit test runner
called [RandomizedTesting|] to accomplish
> It also includes some other nice features
> # Ensure when a test fails, the random seeds are included in error messages and logs
> # Code analysis to ensure the tests are setup properly
> # Run tests in a random order to ensure they are not dependent upon each other
> Preliminary analysis shows that the API for NUnit allows building custom test runners
and it is a close enough match to implement the functionality. Most likely, this will require
a custom adapter as well, so the test runner can integrate into Visual Studio and dotnet test/mstest.
> Do note that without doing some pretty heavy refactoring on the Codec, Directory, and
LuceneTestCase classes, it is not possible to run tests in parallel within the same AppDomain
because the codecs use a static variable to turn codec impersonation on/off. For now, it would
probably be best to run tests serially.

This message was sent by Atlassian Jira

View raw message