lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [lucenenet] NightOwl888 opened a new issue #288: Fix Random Seed Functionality in TestFramework
Date Fri, 15 May 2020 00:20:40 GMT

NightOwl888 opened a new issue #288:
URL: https://github.com/apache/lucenenet/issues/288


   The random seed that is set to `StringHelper.GOOD_FAST_HASH_SEED` through the `tests.seed`
system property in Java currently is the same as the seed that NUnit uses in `NUnit.Framework.TestContext.CurrentContext.Random`.
If we continue using NUnit's `Random` property, more research is needed to figure out how
to set that seed the way the NUnit designers intended so it can be documented for end users.
   
   During testing, I noticed that NUnit does set the seed to a consistent pseudo-random value,
so the same test can be run over and over with exactly the same random setup. However, the
seed is re-generated when the code in the project under test is changed. I noted that NUnit
writes the seed to a file named `nunit_random_seed.tmp` in the `bin` folder of the test project
when run in Visual Studio.
   
   ### A Typical Testing Scenario in Lucene
   
   1. Test fails
   2. Test failure generates an error message that includes the random seed as a hexadecimal
string (that represents a long)
   3. Developer sets the `tests.seed` system property to the same hexadecimal string that
caused the test to fail
   4. Debugging can take place because the pseudo-randomly generated conditions are exactly
the same as they were when the test failed
   5. A fix is devised
   
    ### Testing in .NET
   
   All we have been able to do is get the same test to fail multiple times, but that breaks
as soon as any code is changed. We could probably revert back if we save a copy of the `nunit_random_seed.tmp`,
but this is very complicated to do in addition to debugging, and not very intuitive.
   
   We are missing:
   
   1. The test framework reporting the seed that it was using when the test failed
   2. The ability to set the random seed
   
   I suspect that although setting the seed cannot be done publicly, we can hack it by using
.NET Reflection to set an internal field. There was some [discussion about setting it through
an attribute](https://github.com/nunit/nunit/issues/1461), but it doesn't sound like that
has been implemented.
   
   I haven't looked into what it will take to extend NUnit to attach the random seed to the
test message that is reported. One option (but not a very good one) would be to change the
`Lucene.Net.TestFramework.Assert` class to add the random seed to all of test messages. The
main thing we need is to read the seed value. The only place it seems to be exposed is in
the `bin/nunit_random_seed.tmp` file, but it would be preferable to read it through a property
within the test session.
   
   ### Testing Lucene.NET Itself
   
   For end users, the above would solve all of the issues. However, for Lucene.NET we often
need to generate the same conditions as Java to determine where the execution paths diverge.
This is a problem because the NUnit Random class (probably) isn't the same implementation
as the Java Random class. Also, in Java the random seed is a `long`, but in .NET, it is an
`int`.
   
   We have ported the Java Random class in `J2N`, named [Randomizer](https://github.com/NightOwl888/J2N/blob/master/src/J2N/Randomizer.cs).
Ideally, we would use this implementation during testing. However, NUnit doesn't seem to have
a way to inject a custom implementation of `Random` nor does it have a way to read the seed
it uses to seed a new instance of `Randomizer` for the test framework or a way to override
that setting manually during debugging.
   
   ### Ideal Solution
   
   1. By default, the test framework uses a seed generated by the same hook that NUnit generates
their seed. Alternatively, we could use `System.DateTime.UtcNow.Ticks`.
   2. The seed generated would be a `long`.
   3. The generated seed would be used to seed the `J2N.Randomizer` class, which the getter
of the `LuceneTestFramework.Random` property would provide and cache.
   4. The test framework would ensure the seed is always output along with failure test messages,
as a hexadecimal string.
   5. The "System Properties" feature allows `tests:seed` to be set to a hexadecimal string,
which if set would override the auto-generated random seed used in step 3.
   
   Note that the ideal solution doesn't necessarily involve NUnit in the equation. With Java's
Random class ported and "System Properties" solved in a way that doesn't involve hacking the
system's environment variables when debugging, we are much closer to fixing this to make it
compatible with Java Lucene so we can test apples to apples.
   
   The main thing we are missing is writing the random seed out into the test message with
hexadecimal formatting (the same string that is used in Lucene). Other than that, setting
up the system property to override the automatically generated seed should be fairly straightforward.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message