lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shad Storhaug <s...@shadstorhaug.com>
Subject RE: API Work/Stabilization Update
Date Mon, 10 Apr 2017 20:38:36 GMT
Ø  Having me take a stab at it sounds good, when would you be expecting turnaround?

ASAP. I want to try to get this beta out as quickly as we can. I really need to wrap this up and find some paid work! But if you aren’t available today, I will give it a shot – I was just hoping to take a little of this off my plate. If you do have time for this today/tonight (wherever you are), let me know – there are plenty of other tasks to work on.

I have the script pretty much all transferred to PSake – it cut the amount of code almost in half. It looks like there already is a NuGet Publish task built into TeamCity, so I don’t see much point to keeping it in the script.

Ok, I will change the parameters of the script to utilize the build number and still allow the entire string to be passed in another parameter. It seems like control of the versioning is ultimately something that should be in the repository where people without TeamCity access can modify it. Only downside is that the “version” displayed in TeamCity will only be the incrementing number instead of what is on the NuGet packages.


From: Wyatt Barnett [mailto:wyatt.barnett@gmail.com]
Sent: Tuesday, April 11, 2017 2:46 AM
To: Shad Storhaug
Cc: Connie Yau; dev@lucenenet.apache.org
Subject: Re: API Work/Stabilization Update

Sorry for the confusion. Having me take a stab at it sounds good, when would you be expecting turnaround?

Totally agree on one true, consistent version stream. I would hit it slightly differently though -- I'd let TC supply the build number and then calculate the rest of the string and not depend on parsing in the script. It is a bit easier and cleaner to me but that might just be me.

On Mon, Apr 10, 2017 at 3:13 PM Shad Storhaug <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>> wrote:
Ok, now I am confused again ☺. But since you said the scripts are separated enough now to set it up, could you set up the initial framework of the builds and dependencies so they all work from the same commit? It should be pretty straightforward to figure out what the expectations are and build the script around them after that step is done.

It would also help if I understood how you want to setup the versioning. I can’t imagine a simpler way to do it than having the build server specify the entire version string and having the script work out how to chop it up and put it where it needs to, but maybe there is a better way that I haven’t considered.

We certainly want to keep the same version scheme between MyGet and NuGet so we aren’t locked into “we can’t fix this release because we can’t change the version” like what happened in the past. It really doesn’t make much difference if we promote to NuGet from TeamCity or from MyGet (although if we use MyGet we don’t have to configure anything extra to do it).

From: Wyatt Barnett [mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>]
Sent: Tuesday, April 11, 2017 1:39 AM

To: Shad Storhaug
Cc: Connie Yau; dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
Subject: Re: API Work/Stabilization Update

Sorry -- I knew I left something off. Yes -- we would have a final "package" step that depends on all the other steps in the chain passing for it to complete and push to myget. Note we can pass artifacts down the chain so the preceding steps could do most of the build work for the final one to push to myget. Sorry if I didn't make that clear.



On Mon, Apr 10, 2017 at 2:34 PM Shad Storhaug <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>> wrote:
Wyatt,

That helps.

But one thing to note is that the end product is a single NuGet package with both .NET 451 and .NET Core 1.0 DLLs in it. That kind of breaks the idea of having 2 different builds – unless we have a 3rd build that somehow depends on the other two that pushes the NuGet package out to MyGet. Or set one of the builds up to fail the other if it fails. Any ideas?

Thanks,
Shad Storhaug (NightOwl888)

From: Wyatt Barnett [mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>]
Sent: Tuesday, April 11, 2017 1:26 AM

To: Shad Storhaug
Cc: Connie Yau; dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
Subject: Re: API Work/Stabilization Update

Hi Shad -- yeah it looks like a lot of progress. Still under a rock at the day job.

To answer a question from the previous note -- that "no compatible agents are available" seems to be a side-effect of how they are using on-demand cloud build agents. I think it really translates to "nobody is online who is capable of doing that but we are spinning up a new one, hang tight."

My thought was to have a few separate, chained builds so we would do distinct build/test runs against each framework. To get there the build script needs to understand "build and test for net451" and "build and test for netcore1.0" distinctly. It looks like the components are now there -- you are running those separately, just in the same build configuration. Because they are separate builds we should be OK on xml result files overwriting. And they probably can be parallelized on multiple agents. Does that help demystify things a bit?



On Mon, Apr 10, 2017 at 1:28 PM Shad Storhaug <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>> wrote:
Wyatt,

I have managed to get the test failures under control and started working on a PSake script, the first task being "test". I have also set it up so NUnit will output TeamCity service messages, and made separate TeamCity tasks for each framework.

However, instrumentation really hasn't improved much from when it all ran in one task (at least the last test I ran without the service messages). So, I am not sure exactly what you had in mind to make it easy to spot which framework caused the build failure. If you want to make some edits to the build configuration to show me what you had in mind (or just explain your ideas) that would be great.

One thing I could do (if it helps) is to give the XML files different names per framework or put them into different folders per framework. Right now we just have one framework overwriting the XML files from the last one.

Thanks,
Shad Storhaug (NightOwl888)

-----Original Message-----
From: Shad Storhaug [mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>]
Sent: Saturday, April 8, 2017 11:18 PM
To: Wyatt Barnett
Cc: Connie Yau; dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
Subject: RE: API Work/Stabilization Update

I finally got a build to run all the way through on TeamCity a couple of days ago, and now I am trying to fix tests that are failing on the command line that didn't fail in Visual Studio. A lot of them were failing because the embedded resources didn't end up in the same place when using the CLI. So I added a set of extension methods that do an aggressive match by trying various combinations of assembly name/partial namespace, partial namespace only, and partial assembly name only. It seems Micrsoft still hasn't worked out all of the bugs with how embedded resources behave, and this is the only way to "fix it once".

Anyway, I am trying to start another build to see how many tests are left to fix, but it seems that there are no longer any connected compatible agents that can run it. Wyatt, can you check into this please?

Thanks,
Shad Storhaug (NightOwl888)

-----Original Message-----
From: Shad Storhaug [mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>]
Sent: Friday, April 7, 2017 3:06 AM
To: Wyatt Barnett
Cc: Connie Yau; dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
Subject: RE: API Work/Stabilization Update

Well, PSake runs on top of Powershell – it just makes Powershell act like MSBuild in that you can have multiple tasks that can be run individually and can be dependent on other tasks. But since we got most of the process setup in PowerShell already it would be simpler to transfer the existing functionality if we stick with PowerShell.

FYI – I modified the README page already (https://github.com/apache/lucenenet/tree/api-work). If anyone else has something to add or wants to make a change I will accept requests/pull requests.

I think we should add a paragraph similar to this one: https://github.com/snikch/jquery.dirtyforms/blob/master/CONTRIBUTING.md#code-contributions to our contributions page (especially those linked articles, they are a good read). And of course the status there needs updating.

I ended up manually failing all of the ThaiAnalyzer tests in .NET Core, since there is no way to catch an AccessViolationException in .NET Core. That seems to have fixed the stability problems with the test runner (I need to do some more testing to be sure). But, tomorrow I will start testing with TeamCity to see if I can get the tests to run all the way through.



From: Wyatt Barnett [mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>]
Sent: Friday, April 7, 2017 1:30 AM
To: Shad Storhaug
Cc: Connie Yau; dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
Subject: Re: API Work/Stabilization Update

Good question Shad. I think I de-escaped things properly -- the powershell command it is running is ./runbuild.ps1 -RunTests -FrameworksToTest @("net451", "netcoreapp1.0")

I wholeheartedly agree that this build is complicated enough that we should get something a little beefier than powershell in place. I was looking at FAKE before but PSake could work too. I can't claim to have significant seat-time with either but that is why one does these open source projects.

Anyhow more coming later this week/weekend when I can find some time to write things up . . .

On Thu, Apr 6, 2017 at 11:37 AM Shad Storhaug <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>> wrote:
Thanks.

Hmm…Did you play with the escaping of quotes? What I pasted was the raw command to run from CMD, but that might not be what will run in TeamCity.

I am not entirely certain why it sometimes prompts for that (even when you don’t specify to run a push), or why there are even required parameters. These are some of the reasons why I am suggesting that PSake might be better, but since I don’t know for certain if you can exit the script and re-enter the same target without having to execute all of the target’s dependencies all over again, I would like to run a few tests in TeamCity. Worst case, we don’t specify any dependent targets for “test” so we can run it multiple times without triggering the restore and build processes over and over (which is probably harmless, but takes up lots of time). But it helps if you have separate targets you can call in the script without having to set lots of different Boolean flags to tell it what to run and what not to.

From: Wyatt Barnett [mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>>]
Sent: Thursday, April 6, 2017 6:20 PM

To: Shad Storhaug
Cc: Connie Yau; dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>
Subject: Re: API Work/Stabilization Update

Hi Shad -- you now have teamcity permissions. Please be a bit careful in editing things -- it is a house of cards without great documentation as to why things are doing what and there isn't an easy rollback button like with source controlled code. Let me know if you want a tour of what is what and we can find a time to do some screen sharing.

That command won't run for me on the server -- the error is:

[14:14:38]W:   [Step 1/2] C:\BuildAgent\work\dc5a565ec74d072b\runbuild.ps1 : Cannot process command because of one or more missing mandatory
[14:14:38]W:   [Step 1/2] parameters: NuGetApiKey UploadPackages.
[14:14:38]W:   [Step 1/2]     + CategoryInfo          : InvalidArgument: (:) [runbuild.ps1], ParentContainsErrorRecordException
[14:14:38]W:   [Step 1/2]     + FullyQualifiedErrorId : MissingMandatoryParameter,runbuild.ps1

Is there a way to run it without trying to upload the packages -- that is a fairly mechanical thing we can certainly make work once we get build, test and packaging the right things going.

I will need a bit more time than I can find on weekdays to respond to your other notes -- lots of good thoughts in there, they need cogent responses.

On Wed, Apr 5, 2017 at 6:19 PM Shad Storhaug <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>> wrote:
Wyatt,

It turns out that adding the 4.5.1 framework to project.json was all that was required to get the tests to run, but dotnet.exe seems to crash whenever an exception is thrown from a test on that framework (saw 3 different exception types - NullReferenceException, AccessViolationException, IndexOutOfRangeException and they all brought it to a halt). So, I gave the NUnit build runner a shot and it seems to be more stable (and much faster as well). I wasn't able to get it working on .NET core, but I haven't yet seen it crash when running the .NET core tests.

Give it a shot to see if this will run all the way through without crashing in TeamCity.

powershell -ExecutionPolicy Bypass -Command "& .\runbuild.ps1 -PackageVersion 4.8.0.1000-beta2 -RunTests -FrameworksToTest @(\"net451\", \"netcoreapp1.0\")"

Then we can work on getting this setup so tests can be run as separate steps.

Thanks,
Shad Storhaug (NightOwl888)


-----Original Message-----
From: Shad Storhaug [mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>]
Sent: Wednesday, April 5, 2017 8:42 PM
To: Wyatt Barnett
Cc: Connie Yau; dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>
Subject: RE: API Work/Stabilization Update

I found one issue that is preventing the tests from running - the runner isn't running .NET 4.5.1 framework tests at all. That framework is missing from project.json in the test projects altogether, so it seems this has never been run. I tried putting that section in, but I am getting an error that the "dotnet-test-nunit" target is missing on .NET 4.5.1 (and tracking down the cause of this doesn't seem to be plausible amongst all of the advice telling to "upgrade"). The build works, so I don't think it would be wise to upgrade just for the sake of testing.

So, it seems we are back to square 1 - you are onto something with installing the NUnit console runner for this. That is also easier said than done - for some reason both Visual Studio and dotnet restore are ignoring the fact that I added the NuGet package to the project and are not downloading the binaries. I guess worst case scenario I can just check the binaries into the repo, but it would be more difficult to upgrade if we did that, so I am trying to find a fix to the NuGet download issue first.

The good news is in .NET core I was able to successfully run all tests in the Lucene.Net.Tests project.

As for splitting this up, I am thinking that starting a script with a true build runner like PSake is probably a good idea for the long term. Perhaps we could just take the test part out of the current script so we could set it up in TeamCity like:

1. Restore/Build step using runbuild.ps1 2. Test (using a different script that can be called multiple times) 3. Package/Push using runbuild.ps1 (side-effect of going through the Restore/Build steps all over again)


Also, allow the raw "where" clause to be passed directly to the command line tool rather than doing all of the building that is done currently. Then the tests could be setup in different steps however you like. To ensure we can add projects to the setup without having them ignored by default, I suggest:

1. Test Lucene.Net (50% of the tests)
2. Test Lucene.Net.Analysis.Common (25% of the tests) 3. Test everything but Lucene.Net and Lucene.Net.Analysis.Common (25% of the tests)

And repeat for each framework.

Or...wait a minute. I just noticed that one of the packages that downloads with NUnit.Console is a TeamCity event listener. Would it make more sense just to skip a test script for this (at least for .NET 4.5.1) and set it up using TeamCity tools?

Thanks,
Shad Storhaug (NightOwl888)

-----Original Message-----
From: Shad Storhaug [mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>]
Sent: Wednesday, April 5, 2017 1:10 PM
To: Wyatt Barnett
Cc: Connie Yau; dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>
Subject: RE: API Work/Stabilization Update

Wyatt,

I have been getting that randomly, too (but I thought it was mainly due to Visual Studio locking the file). Re-running has always made it recover here. Basically, I have set it up so it backs up the project.json files before making edits so it can restore them at the end. This is primarily to prevent someone running locally from accidentally checking in the changes the build makes. However, if it cannot backup the files for some reason (a permission error perhaps) it won’t be able to do this step.

So, looks like we need to add an option to override the backup/restore file process on the build server, which will work around any windows permission issues that prevent that from working. I will also add a check if the file exists before attempting to restore to make it more robust.

The modified process does one very important step – it names the main NuGet package Lucene.Net instead of Lucene.Net.Core. I tried making the script compensate for the fact that dotnet.exe doesn’t support an option for changing the package name to be different than the folder name, but that became very complicated – it was simpler just to rename the folder from Lucene.Net.Core to Lucene.Net.

As per your suggestion, we should probably break up the restore, build, test, and pack steps so they work separately. Unfortunately, this script wasn’t built with a task runner with separate entry points, so that is going to be a bit complicated. Do we need the part of the script that pushes to NuGet/MyGet, or is that something you can setup on TeamCity as a separate step without a script?

What specific handling do you need on the version string? I set it up so you can pass the PackageVersion (of the NuGet package) separate from the Version (of the assembly). It also sets the PackageVersion as the AssemblyInformationalVersion of the assembly and also takes care of setting all of the versions for inter-package dependencies between NuGet packages. Basically, this makes it simple to specify only 1 parameter for version and the script will take care of the rest.

Keep in mind, however we set it up that this process is temporary. We will likely change it up again when Microsoft finally gets .csproj working right. So, we should focus on the *interface* of the build process (that is, how TeamCity will interact with the script) so we can change the script around later without breaking the build. Perhaps we should use a task runner like PSake so we have separate entry points: https://github.com/psake/psake? I have always used it in conjunction with PowerShell in the past (although I have to admit, I never needed any of the other entry points – but then, I never had 2 hours of testing to do). That would eliminate the need to specify a lot of different switches to turn on and off the right stuff for the current task. i.e (Invoke-psake .\runbuild.ps1 Test -framework 4.0x64 -properties @{ “framework”=”netcore1.0” }) would just run the tests for .NET Core and do nothing else (see the Spatial4n script: https://github.com/NightOwl888/Spatial4n/blob/master/.build/build.ps1).

Build.bat is primarily to make the synatax of building simpler than what is required to call powershell when building locally. (build –pv:4.8.0.891-beta2  rather than powershell –Command “& .\runbuild.ps1 –CreatePackages –PackageVersion 4.8.0.891-beta2”). It could be expanded with more parameters and/or environment variables if necessary – I find it simpler to maintain and test if there is both a way to manually build and automatically build that run through the same script (the manual part should be as simple as possible, the automated part only as complex as it needs to be for the build server to do its magic).

I tried running all of the tests last night from the command line and it crashed during the run. Fortunately, I have a stack trace that might help track it down.

Culture: vai-Latn-LR
Time Zone: (UTC-05:00) Bogota, Lima, Quito, Rio Branco Default Codec: Lucene41 (Lucene.Net.Codecs.Lucene41.Lucene41RWCodec)
Default Similarity: RandomSimilarityProvider(queryNorm=False,coord=yes):
=> Lucene.Net.Search.TestConstantScoreQuery.TestQueryWrapperFilter
Passed
Culture: vai-Latn-LR
Time Zone: (UTC-05:00) Bogota, Lima, Quito, Rio Branco Default Codec: Lucene41 (Lucene.Net.Codecs.Lucene41.Lucene41RWCodec)
Default Similarity: RandomSimilarityProvider(queryNorm=False,coord=yes): [field, DFR G3(800)] => Lucene.Net.Search.TestConstantScoreQuery.TestWrapped2Times
Passed
Culture: he-IL
Time Zone: (UTC-04:00) Asuncion
Default Codec: Lucene3x (Lucene.Net.Codecs.Lucene3x.PreFlexRWCodec)
Default Similarity: DefaultSimilarity
: hit exc
System.NullReferenceException: Object reference not set to an instance of an object.
   at Lucene.Net.Util.IOUtils.ReThrow(Exception th) in F:\Projects\lucenenet\src\Lucene.Net\Util\IOUtils.cs:line 413
   at Lucene.Net.Index.StoredFieldsProcessor.Flush(SegmentWriteState state) in F:\Projects\lucenenet\src\Lucene.Net\Index\StoredFieldsProcessor.cs:line 91
   at Lucene.Net.Index.TwoStoredFieldsConsumers.Flush(SegmentWriteState state) in F:\Projects\lucenenet\src\Lucene.Net\Index\TwoStoredFieldsConsumers.cs:line 47
   at Lucene.Net.Index.DocFieldProcessor.Flush(SegmentWriteState state) in F:\Projects\lucenenet\src\Lucene.Net\Index\DocFieldProcessor.cs:line 84
   at Lucene.Net.Index.DocumentsWriterPerThread.Flush() in F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriterPerThread.cs:line 573
   at Lucene.Net.Index.DocumentsWriter.DoFlush(DocumentsWriterPerThread flushingDWPT) in F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 638
   at Lucene.Net.Index.DocumentsWriter.PreUpdate() in F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 461
   at Lucene.Net.Index.DocumentsWriter.UpdateDocuments(IEnumerable`1 docs, Analyzer analyzer, Term delTerm) in F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 513
   at Lucene.Net.Index.IndexWriter.UpdateDocuments(Term delTerm, IEnumerable`1 docs, Analyzer analyzer) in F:\Projects\lucenenet\src\Lucene.Net\Index\IndexWriter.cs:line 1562
   at Lucene.Net.Index.TrackingIndexWriter.UpdateDocuments(Term t, IEnumerable`1 docs) in F:\Projects\lucenenet\src\Lucene.Net\Index\TrackingIndexWriter.cs:line 103
   at Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocuments(Term id, IEnumerable`1 docs) in F:\Projects\lucenenet\src\Lucene.Net.Tests\Search\TestControlledRealTimeReopenThread.cs:line 116
   at Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run() in F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Index\ThreadedIndexingAndSearchingTestCase.cs:line 276
   at Lucene.Net.Util.IOUtils.ReThrow(Exception th) in F:\Projects\lucenenet\src\Lucene.Net\Util\IOUtils.cs:line 413
   at Lucene.Net.Index.StoredFieldsProcessor.Flush(SegmentWriteState state) in F:\Projects\lucenenet\src\Lucene.Net\Index\StoredFieldsProcessor.cs:line 91
   at Lucene.Net.Index.TwoStoredFieldsConsumers.Flush(SegmentWriteState state) in F:\Projects\lucenenet\src\Lucene.Net\Index\TwoStoredFieldsConsumers.cs:line 47
   at Lucene.Net.Index.DocFieldProcessor.Flush(SegmentWriteState state) in F:\Projects\lucenenet\src\Lucene.Net\Index\DocFieldProcessor.cs:line 84
  index=_0(4.2):C10/1:delGen=1 _1(4.2):C10 _2(4.2):C10 _3(4.2):C5 _4(4.2):C1IW 8 [5/4/2017 01:48:05; ]: merging _1(4.2):C10 _2(4.2):C10 _0(4.2):C10/1:delGen=1 _3(4.2):C5 _4(4.2):C1IW 8 [5/4/2017 01:48:05; ]: seg=_1(4.2):C10 no deletesIW 8 [5/4/2017 01:48:05; ]: seg=_2(4.2):C10 no deletesIW 8 [5/4/2017 01:48:05; ]: seg=_0(4.2):C10/1:delGen=1 delCount=1IW 8 [5/4/2017 01:48:05; ]: seg=_3(4.2):C5 no deletesIW 8 [5/4/2017 01:48:05; ]: seg=_4(4.2):C1 no deletesSM 8 [5/4/2017 01:48:05; ]: merge store matchedCount=5 vs 5SM 8 [5/4/2017 01:48:05; ]: 0 msec to merge stored fields [35 docs]SM 8 [5/4/2017 01:48:05; ]: 3 msec to merge postings [35 docs]SM 8 [5/4/2017 01:48:05; ]: 5 msec to merge doc values [35 docs]SM 8 [5/4/2017 01:48:05; ]: 0 msec to merge norms [35 docs]SM 8 [5/4/2017 01:48:05; ]: 1 msec to merge vectors [35 docs]IW 8 [5/4/2017 01:48:05; ]: merge codec=Lucene46: [[trieLong, PostingsFormat(name=Memory doPackFST= False)], [content2, PostingsFormat(name=Memory doPackFST= True)], [id, PostingsFormat(name=Memory doPackFST= True)], [content5, FST41], [autf8, PostingsFormat(name=Direct)], [trieInt, PostingsFormat(name=Memory doPackFST= False)], [utf8, PostingsFormat(name=Memory doPackFST= False)], [fieΓ▒╖ld, FST41], [content3, PostingsFormat(name=Direct)], [content, PostingsFormat(name=Memory doPackFST= True)], [content6, PostingsFormat(name=Direct)]], docValues:[[dvSortedSet, DocValuesFormat(name=Disk)], [dvPacked, DocValuesFormat(name=Disk)], [dvBytesStraightVar, DocValuesFormat(name=Asserting)], [dvLong, DocValuesFormat(name=Disk)], [dvInt, DocValuesFormat(name=Memory)], [dvBytesDerefFixed, DocValuesFormat(name=Memory)], [dvFloat, DocValuesFormat(name=Asserting)], [dvBytesSortedFixed, DocValuesFormat(name=SimpleText)], [dvBytesStraightFixed, DocValuesFormat(name=Memory)], [dvBytesSortedVar, DocValuesFormat(name=Disk)], [dvDouble, DocValuesFormat(name=Disk)], [dvBytesDerefVar, DocValuesFormat(name=Memory)], [dvByte, DocValuesFormat(name=Disk)], [dvShort, DocValuesFormat(name=Disk)]] docCount=35; merged segment has vectors; norms; docValues; prox; freqsIW 8 [5/4/2017 01:48:05; ]: merged segment size=%.3f MB vs estimate=%.3f MBTP 8 [5/4/2017 01:48:05; ]: startCommitMergeIW 8 [5/4/2017 01:48:05; ]: commitMerge: _1(4.2):C10 _2(4.2):C10 _0(4.2):C10/1:delGen=1 _3(4.2):C5 _4(4.2):C1 index=_0(4.2):C10/1:delGen=1 _1(4.2):C10 _2(4.2):C10 _3(4.2):C5 _4(4.2):C1TP 8 [5/4/2017 01:48:05; ]: startCommitMergeDeletesIW 8 [5/4/2017 01:48:05; ]: commitMergeDeletes _1(4.2):C10 _2(4.2):C10 _0(4.2):C10/1:delGen=1 _3(4.2):C5 _4(4.2):C1IW 8 [5/4/2017 01:48:05; ]: no new deletes or field updates since merge startedIFD 8 [5/4/2017 01:48:05; ]: now checkpoint "_5(4.8):C35" [1 segments ; isCommit = False]IFD 8 [5/4/2017 01:48:05; ]: 0 msec to checkpointIW 8 [5/4/2017 01:48:05; ]: after commitMerge: _5(4.8):C35UPGMP 8 [5/4/2017 01:48:05; ]: findForcedMerges: segmentsToUpgrade=System.Collections.Generic.Dictionary`2[Lucene.Net.Index.SegmentCommitInfo,System.Nullable`1[System.Boolean]]IW 8 [5/4/2017 01:48:05; ]: merge time 0 msec for 35 docsIndexUpgrader 8 [5/4/2017 01:48:05; ]: All segments upgraded to version 4.8CMS 8 [5/4/2017 01:48:05; ]:   merge thread: doneIW 8 [5/4/2017 01:48:05; ]: now flush at close waitForMerges=TrueTP 8 [5/4/2017 01:48:05; ]: startDoFlushIW 8 [5/4/2017 01:48:05; ]:   start flush: applyAllDeletes=TrueIW 8 [5/4/2017 01:48:05; ]:   index before flush _5(4.8):C35DW 8 [5/4/2017 01:48:05; ]: startFullFlushDW 8 [5/4/2017 01:48:05; ]: anyChanges? numDocsInRam=0 deletes=False hasTickets:False pendingChangesInFullFlush: FalseDW 8 [5/4/2017 01:48:05; ]:  finishFullFlush success=TrueIW 8 [5/4/2017 01:48:05; ]: apply all deletes during flushBD 8 [5/4/2017 01:48:05; ]: applyDeletes: no deletes; skippingBD 8 [5/4/2017 01:48:05; ]: prune sis=Lucene.Net.Index.SegmentInfos minGen=0 packetCount=0CMS 8 [5/4/2017 01:48:05; ]: now mergeCMS 8 [5/4/2017 01:48:05; ]:   index: _5(4.8):C35CMS 8 [5/4/2017 01:48:05; ]:   no more merges pending; now returnIW 8 [5/4/2017 01:48:05; ]: waitForMergesIW 8 [5/4/2017 01:48:05; ]: waitForMerges doneIW 8 [5/4/2017 01:48:05; ]: now call final commit()IW 8 [5/4/2017 01:48:05; ]: commit: startIW 8 [5/4/2017 01:48:05; ]: commit: enter lockIW 8 [5/4/2017 01:48:05; ]: commit: now prepareIW 8 [5/4/2017 01:48:05; ]: prepareCommit: flushIW 8 [5/4/2017 01:48:05; ]:   index before flush _5(4.8):C35TP 8 [5/4/2017 01:48:05; ]: startDoFlushDW 8 [5/4/2017 01:48:05; ]: startFullFlushDW 8 [5/4/2017 01:48:05; ]: anyChanges? numDocsInRam=0 deletes=False hasTickets:False pendingChangesInFullFlush: FalseIW 8 [5/4/2017 01:48:05; ]: apply all deletes during flushBD 8 [5/4/2017 01:48:05; ]: applyDeletes: no deletes; skippingBD 8 [5/4/2017 01:48:05; ]: prune sis=Lucene.Net.Index.SegmentInfos minGen=0 packetCount=0DW 8 [5/4/2017 01:48:05; ]:  finishFullFlush success=TrueTP 8 [5/4/2017 01:48:05; ]: startStartCommitIW 8 [5/4/2017 01:48:05; ]: StartCommit(): startIW 8 [5/4/2017 01:48:05; ]: startCommit index=_5(4.8):C35 changeCount=2TP 8 [5/4/2017 01:48:05; ]: midStartCommitTP 8 [5/4/2017 01:48:05; ]: midStartCommit2IW 8 [5/4/2017 01:48:05; ]: done all syncs: _5.fdx, _5.fdt, _5_Direct_0.doc, _5_Direct_0.pos, _5_Direct_0.pay, _5_Direct_0.tim, _5_Direct_0.tip, _5_Memory_0.ram, _5_FST41_0.doc, _5_FST41_0.pos, _5_FST41_0.pay, _5_FST41_0.tmp, _5_Memory_1.ram, _5_Disk_0.dvdd, _5_Disk_0.dvdm, _5_Memory_0.mdvd, _5_Memory_0.mdvm, _5_SimpleText_0.dat, _5_Asserting_0.dvd, _5_Asserting_0.dvm, _5.nvd, _5.nvm, _5.tvx, _5.tvd, _5.fnm, _5.siTP 8 [5/4/2017 01:48:05; ]: midStartCommitSuccessTP 8 [5/4/2017 01:48:05; ]: finishStartCommitIW 8 [5/4/2017 01:48:05; ]: commit: pendingCommit != nullIW 8 [5/4/2017 01:48:05; ]: commit: wrote segments file "segments_4"IFD 8 [5/4/2017 01:48:05; ]: now checkpoint "_5(4.8):C35" [1 segments ; isCommit = True]IFD 8 [5/4/2017 01:48:05; ]: deleteCommits: now decRef commit "segments_3"IFD 8 [5/4/2017 01:48:05; ]: delete "segments_3"IFD 8 [5/4/2017 01:48:05; ]: delete "_0.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_Lucene42_0.dvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_Lucene41_0.pos"IFD 8 [5/4/2017 01:48:05; ]: delete "_0.tvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_Lucene42_0.dvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_0.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_Lucene41_0.pay"IFD 8 [5/4/2017 01:48:05; ]: delete "_0.tvx"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete "_0.nvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_0.fdx"IFD 8 [5/4/2017 01:48:05; ]: delete "_0.si<http://0.si><http://0.si>"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05; ]: delete "_0.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_Lucene41_0.tip"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_1.del"IFD 8 [5/4/2017 01:48:05; ]: delete "_1.tvx"IFD 8 [5/4/2017 01:48:05; ]: delete "_1.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_1_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete "_1_Lucene42_0.dvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_1_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05; ]: delete "_1.nvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_1_Lucene41_0.tip"IFD 8 [5/4/2017 01:48:05; ]: delete "_1_Lucene42_0.dvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_1.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete "_1_Lucene41_0.pos"IFD 8 [5/4/2017 01:48:05; ]: delete "_1.fdx"IFD 8 [5/4/2017 01:48:05; ]: delete "_1_Lucene41_0.pay"IFD 8 [5/4/2017 01:48:05; ]: delete "_1.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete "_1.si<http://1.si><http://1.si>"IFD 8 [5/4/2017 01:48:05; ]: delete "_1.tvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.si<http://2.si><http://2.si>"IFD 8 [5/4/2017 01:48:05; ]: delete "_2_Lucene41_0.pos"IFD 8 [5/4/2017 01:48:05; ]: delete "_2_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete "_2_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete "_2_Lucene41_0.tip"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.fdx"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.tvx"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.tvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_2_Lucene42_0.dvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_2_Lucene41_0.pay"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.nvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_2_Lucene42_0.dvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.tvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_3_Lucene41_0.pay"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.fdx"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.tvx"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.nvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_3_Lucene41_0.pos"IFD 8 [5/4/2017 01:48:05; ]: delete "_3_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete "_3_Lucene41_0.tip"IFD 8 [5/4/2017 01:48:05; ]: delete "_3_Lucene42_0.dvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.si<http://3.si><http://3.si>"IFD 8 [5/4/2017 01:48:05; ]: delete "_3_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_3_Lucene42_0.dvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_4.fdx"IFD 8 [5/4/2017 01:48:05; ]: delete "_4.nvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_4_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete "_4.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete "_4.si<http://4.si><http://4.si>"IFD 8 [5/4/2017 01:48:05; ]: delete "_4.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete "_4_Lucene41_0.tip"IFD 8 [5/4/2017 01:48:05; ]: delete "_4.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_4_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05; ]: 0 msec to checkpointIW 8 [5/4/2017 01:48:05; ]: commit: doneIW 8 [5/4/2017 01:48:05; ]: at close: _5(4.8):C35
   at Lucene.Net.Index.DocumentsWriterPerThread.Flush() in F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriterPerThread.cs:line 573
   at Lucene.Net.Index.DocumentsWriter.DoFlush(DocumentsWriterPerThread flushingDWPT) in F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 638
   at Lucene.Net.Index.DocumentsWriter.PreUpdate() in F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 461
   at Lucene.Net.Index.DocumentsWriter.UpdateDocuments(IEnumerable`1 docs, Analyzer analyzer, Term delTerm) in F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 513
   at Lucene.Net.Index.IndexWriter.UpdateDocuments(Term delTerm, IEnumerable`1 docs, Analyzer analyzer) in F:\Projects\lucenenet\src\Lucene.Net\Index\IndexWriter.cs:line 1562
   at Lucene.Net.Index.TrackingIndexWriter.UpdateDocuments(Term t, IEnumerable`1 docs) in F:\Projects\lucenenet\src\Lucene.Net\Index\TrackingIndexWriter.cs:line 103
   at Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocuments(Term id, IEnumerable`1 docs) in F:\Projects\lucenenet\src\Lucene.Net.Tests\Search\TestControlledRealTimeReopenThread.cs:line 116 Unhandled Exception: System.Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at Lucene.Net.Util.IOUtils.ReThrow(Exception th) in F:\Projects\lucenenet\src\Lucene.Net\Util\IOUtils.cs:line 413
   at Lucene.Net.Index.StoredFieldsProcessor.Flush(SegmentWriteState state) in F:\Projects\lucenenet\src\Lucene.Net\Index\StoredFieldsProcessor.cs:line 91
   at Lucene.Net.Index.TwoStoredFieldsConsumers.Flush(SegmentWriteState state) in F:\Projects\lucenenet\src\Lucene.Net\Index\TwoStoredFieldsConsumers.cs:line 47
   at Lucene.Net.Index.DocFieldProcessor.Flush(SegmentWriteState state) in F:\Projects\lucenenet\src\Lucene.Net\Index\DocFieldProcessor.cs:line 84
   at Lucene.Net.Index.DocumentsWriterPerThread.Flush() in F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriterPerThread.cs:line 573
   at Lucene.Net.Index.DocumentsWriter.DoFlush(DocumentsWriterPerThread flushingDWPT) in F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 638
   at Lucene.Net.Index.DocumentsWriter.PreUpdate() in F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 461
   at Lucene.Net.Index.DocumentsWriter.UpdateDocuments(IEnumerable`1 docs, Analyzer analyzer, Term delTerm) in F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 513
   at Lucene.Net.Index.IndexWriter.UpdateDocuments(Term delTerm, IEnumerable`1 docs, Analyzer analyzer) in F:\Projects\lucenenet\src\Lucene.Net\Index\IndexWriter.cs:line 1562
   at Lucene.Net.Index.TrackingIndexWriter.UpdateDocuments(Term t, IEnumerable`1 docs) in F:\Projects\lucenenet\src\Lucene.Net\Index\TrackingIndexWriter.cs:line 103
   at Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocuments(Term id, IEnumerable`1 docs) in F:\Projects\lucenenet\src\Lucene.Net.Tests\Search\TestControlledRealTimeReopenThread.cs:line 116
   at Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run() in F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Index\ThreadedIndexingAndSearchingTestCase.cs:line 276 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Lucene.Net.Util.IOUtils.ReThrow(Exception th)
   at Lucene.Net.Index.StoredFieldsProcessor.Flush(SegmentWriteState state)
   at Lucene.Net.Index.TwoStoredFieldsConsumers.Flush(SegmentWriteState state)
   at Lucene.Net.Index.DocFieldProcessor.Flush(SegmentWriteState state)
   at Lucene.Net.Index.DocumentsWriterPerThread.Flush()
   at Lucene.Net.Index.DocumentsWriter.DoFlush(DocumentsWriterPerThread flushingDWPT)
   at Lucene.Net.Index.DocumentsWriter.PreUpdate()
   at Lucene.Net.Index.DocumentsWriter.UpdateDocuments(IEnumerable`1 docs, Analyzer analyzer, Term delTerm)
   at Lucene.Net.Index.IndexWriter.UpdateDocuments(Term delTerm, IEnumerable`1 docs, Analyzer analyzer)
   at Lucene.Net.Index.TrackingIndexWriter.UpdateDocuments(Term t, IEnumerable`1 docs)
   at Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocuments(Term id, IEnumerable`1 docs)
   at Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run()
   --- End of inner exception stack trace ---
   at Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run()
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run() in F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Index\ThreadedIndexingAndSearchingTestCase.cs:line 276
WARNING: Could not find TestResult.xml.
Testing [Lucene.Net.Tests.Analysis.Common] on [net451]...
dotnet.exe test --configuration Release --framework net451 --no-build Project 'F:\Projects\lucenenet\src\Lucene.Net.Tests.Analysis.Common\project.json' does not support framework: net451
WARNING: Could not find TestResult.xml.
Testing [Lucene.Net.Tests.Analysis.Common] on [netcoreapp1.0]...
dotnet.exe test --configuration Release --framework netcoreapp1.0 --no-build NUnit .NET Core Runner 3.4.0 Copyright (C) 2016 Charlie Poole Runtime Environment
    OS Platform: Windows
     OS Version: 10.0.14393
        Runtime: win10-x64
Test Files
    F:\Projects\lucenenet\src\Lucene.Net.Tests.Analysis.Common\bin\Release\netcoreapp1.0\Lucene.Net.Tests.Analysis.Common.dll
Terminate batch job (Y/N)? y

And yea, I see that it is having trouble finding TestResult.xml files, too. Perhaps that move step also needs to be made optional or taken out entirely.


Thanks,
Shad Storhaug (NightOwl888)



From: Wyatt Barnett [mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>>]
Sent: Wednesday, April 5, 2017 4:43 AM
To: Shad Storhaug
Cc: Connie Yau; dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>
Subject: Re: API Work/Stabilization Update

Update -- build was building and testing as of yesterday evening, it looks like some of the build.ps1 changes blew up whatever was working there, the build is now failing because it can't find some project.json.bak file . . .

On Tue, Apr 4, 2017 at 4:41 PM Wyatt Barnett <wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>>>> wrote:
Hi Shad -- sorry I have been trying to get the whole thing working on TeamCity for the last few days. Unfortunately test runs take upwards of an hour and I have to remember to get back to it to check so progress is slow. In any case I'm actually very close to getting the fundamentals worked out -- see https://teamcity.jetbrains.com/viewType.html?buildTypeId=LuceneNet_Vs2015LuceneNetPortable for the specific build project. At this point I can get it to build and run all the tests. The problem is reading the test results -- build.ps1 makes the nunit XML files but it moves them and for some reason TeamCity didn't like that. Is there any reason it was taking that copy step connie?

Project.json vs Project.csproj is pretty immaterial to me.

Anyhow, it does appear the build server has the pre-requisites required to do so. As for teamcity access I would create an account at https://teamcity.jetbrains.com and then let me know what the user / email is and I'll see if I can get them to patch you in.

FWIW build.bat probably doesn't help much -- we need a bit more elegant handling of the version string, and probably want to run separate tests for each target so we know what is failing.



On Tue, Apr 4, 2017 at 1:31 PM Shad Storhaug <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>>> wrote:
Update

I have made it past the hurdles with the build, and got the build working on MyGet. But since MyGet has a build timeout of 15 minutes, there is no way to run the tests there.

So, now we need to get it working on TeamCity. AFAIK I don't have access to it, so if someone could either help me out or provide access that would be great.

project.json files are NOT the new way (as I previously thought) - Microsoft has already scrapped this idea. I guess back when Connie was working on it things were still up in the air: http://fizzylogic.nl/2016/05/11/project-json-is-going-away-and-it-s-a-good-thing/ - and actually they still are. So, we might have to keep it this way for now and then change it around at some later point when Microsoft officially announces that multi-targeting support is released, then we can go back to a single solution and single project file per project.

We need to have the .NET Core SDK installed on the build server. Although it is possible to download the binaries, they are more than 100MB when unpacked. So, here is the list or prerequisites for the build server.

PowerShell version 3.0 or higher
Git for Windows
.NET Core 1.1 with SDK Preview 2.1 build 3177 (https://github.com/dotnet/core/blob/master/release-notes/download-archive.md)

To run the build, execute "build.bat".

Environment Variables:

%PackageVersion% - Entire version string including pre-release tag %RunAllTests% - Pass "true" to bypass the 4 parameters that you would otherwise have to set on runbuild.ps1 in order to run all of the tests for both .NET Framework and .NET Core


Thanks,
Shad Storhaug (NightOwl888)


-----Original Message-----
From: Shad Storhaug [mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>>]
Sent: Sunday, April 2, 2017 4:10 PM
To: wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>>>
Cc: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>>; Connie Yau
Subject: RE: API Work/Stabilization Update

Update

I tried running the tests with verbosity on and NUnit is misbehaving in ways I didn't previously realize. There is apparently some resource leak that is causing a buildup over several tests and then once it reaches a certain threshold all of the tests will fail with an OutOfMemoryException. For now, I am going to consider fixing verbosity low priority - instead we should disable it for the time being and focus on the build/test runs.

Looking through the build script there are few things of note:

1. It skips long running tests and tests that are marked with a timeout by default 2. It runs in Release by default 3. Connie has listed Visual Studio 2015 Update 3 as a prerequisite

Clearly, this isn't what we want. We shouldn't be skipping any tests on the build server except for those that have been manually ignored. Verbosity is being ignored in the build script by default, which explains why the verbosity issue is not causing the script to crash.

I am also trying to work out if there is a way to run without Visual Studio. The script depends on dotnet.exe, which (I think) is installed by the .NET Core SDK. I am attempting to find out if we really need the SDK or if the executable is all we need. Either way, I don't see this leaving the ground without having .NET Framework 4.5.1 and (possibly) .NET Core 1.1 installed on the server.

I guess this leaves a few questions:

1. Is it safe to assume the .NET Framework 4.5.1 and .NET Core 1.1 runtimes will be installed on the server?
2. Can we install the .NET Core SDK on the server, or is our only option to find out if the tools we need are portable (I can't imagine they are not, but you never know)?


Thanks,
Shad Storhaug (NightOwl888)

-----Original Message-----
From: Wyatt Barnett [mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>>>]
Sent: Saturday, April 1, 2017 7:52 AM
To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>>
Subject: Re: API Work/Stabilization Update

I wanted to walk back to one thing you asked about the tests Shad:

"Worst case, we can just turn off verbosity on the build, but it might be helpful to leave it on if something goes wrong"

I *think* I'm turning the verbosity off in the tests -- they should be running in release mode. They would not even run in debug if I recall correctly. Let me know how the switch works and I'll make sure I'm throwing it on the build server side.

On Fri, Mar 31, 2017 at 6:56 PM Shad Storhaug <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>>> wrote:

> I don't necessarily need ownership access, but he did give me
> ownership to the https://www.nuget.org/packages/Spatial4n.Core/ package.
> Alternatively, Itamar can enter his keys on
> https://www.myget.org/feed/Security/spatial4n - he already has ownership.
> Once the keys are added there, I will be able to push.
>
> -----Original Message-----
> From: Prescott Nasser
> [mailto:geobmx540@hotmail.com<mailto:geobmx540@hotmail.com><mailto:geobmx540@hotmail.com<mailto:geobmx540@hotmail.com>><mailto:geobmx540@hotmail.com<mailto:geobmx540@hotmail.com><mailto:geobmx540@hotmail.com<mailto:geobmx540@hotmail.com>>>]
> Sent: Saturday, April 1, 2017 5:49 AM
> To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>>
> Subject: RE: API Work/Stabilization Update
>
> Access like ownership access? Paging Itamar..
>
>
>
> -----Original Message-----
> From: Shad Storhaug
> [mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>>]
> Sent: Friday, March 31, 2017 3:38 PM
> To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>>
> Subject: RE: API Work/Stabilization Update
>
> BTW - the contrib NuGet package is now dead - the functionality has
> been moved into new sub-projects just like in Lucene.
>
> But, that reminds me - I still don't have access to
> https://www.nuget.org/packages/Spatial4n.Core.NTS/, so I have been
> unable to update it to the same version that
> https://www.nuget.org/packages/Spatial4n.Core/ is on. NTS is not
> required by Lucene.Net, but it is required for anyone who needs to run the tests.
>
> -----Original Message-----
> From: Shad Storhaug
> [mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>>]
> Sent: Saturday, April 1, 2017 5:31 AM
> To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>>
> Subject: RE: API Work/Stabilization Update
>
> Wyatt,
>
> Great. Yea, actually after I sent my last email I was starting to
> wonder if Connie ever tested the build on a box without Visual Studio installed.
> We might need to determine what the prerequisites for the build are so
> we know what to put on the build server.
>
> I haven't yet had a chance to determine what tests are causing the
> build to crash (although, I know I can reliably reproduce it). I plan
> on getting that straightened out shortly. Worst case, we can just turn
> off verbosity on the build, but it might be helpful to leave it on if
> something goes wrong. I'll also look into the build dependencies and the script itself.
>
> I am just going through now and deleting all of the old 3.x source
> files that are now just causing noise in the project and resurrecting
> the tests for the support classes. But I am nearly finished. If you
> are available, I would like to try to get the beta2 release done over
> the weekend (at least to the point where it is on MyGet). We need to
> get the README and CONTRIBUTING files updated with the latest status
> (it would be helpful to sync the wiki page as well:
> https://cwiki.apache.org/confluence/display/LUCENENET/Current+Status -
> I don't think I have access for that).
>
> Several issues regarding the main readme need to be addressed:
>
> 1. There is no link to the wiki
> 2. There is no link to the issue tracker 3. It is not clear what the
> status is 4. There are no build instructions 5. There is no mention
> that we need help to finish 6. There is no link to the license 7. The
> documentation links are out of date 8. The top part of the repo
> already lists the files, do we really need to describe them again?
>
> Itamar mentioned he would be working on a new web site - I am not sure
> what the status of that is, but either way these issues are causing
> friction with anyone who is willing to help, but can't navigate
> through these hurdles. People who are used to working with GitHub, its
> issue tracker, and wiki don't find it natural to go looking somewhere
> else to these tools. For beta testing, we definitely need to make it
> crystal clear how to report bugs.
>
> Also, the "known issues" list is getting short enough so I can start
> adding them to the issue tracker.
>
> I have been using the downtime while running tests to fix up the
> documentation comments in Lucene.Net.Core, but it's still only about
> 50% done. I could use some help getting them fixed so they are at
> least visible in Visual Studio intellisense. See the latest status on
> #203. And then of course we can use them to generate new documents.
>
>
> Thanks,
> Shad Storhaug (NightOwl888)
>
>
> -----Original Message-----
> From: Wyatt Barnett
> [mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>>>]
> Sent: Saturday, April 1, 2017 4:22 AM
> To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>>
> Subject: Re: API Work/Stabilization Update
>
> Shad -- definitely makes sense.
>
> Json files are fine -- functionally this is a bit too fancy to use
> Teamcity's automagic nuget package generation so as long as we've got
> a file to edit we are fine.
>
> Myget -> nuget works for me but that doesn't solve the key problem. I
> don't have it, maybe Prescott or Itamar know where it is kept but I
> can't claim to have ever seen it. I joined this party after the last nuget push.
>
> It is a bit foggy but I think I ran into the nunit-console issue with
> the
> Build.ps1 script. Remember that with build servers the pre-requisites
> often need to be embedded in the project for things to work properly.
>
> Anyhow, let me know when you are in a good place with your branch to
> start slogging through getting the new build working. In the interests
> of full disclosure I'm working an event the last week and a half of
> April and will be completely out of pocket then. But I'm about otherwise.
>
> On Sat, Mar 25, 2017 at 10:04 AM Shad Storhaug
> <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>>>
> wrote:
>
> > Wyatt,
> >
> > > We will probably want to build out .nuspec files to get all the
> > > nuget
> > stuff right for these projects -- I don't think the generation will
> > work for us to get things quite right.
> >
> > Connie has set us up to use .json files instead of .nuspec files to
> > generate the NuGet packages (the new way instead of the old way).
> > The build script Build.ps1 does it all (it even has help
> > documentation), but it is missing an option to override versioning.
> > Ideally we would be able to override the version that is in the
> > .json file with an environment variable (which you can pass from
> > TeamCity), and be able to override that on the CLI for local
> > "one-off" builds.  See the build
> instructions on #191:
> > https://github.com/apache/lucenenet/pull/191
> >
> > > Regarding the nuget key -- that plan works for me, the trick is I
> > > don't
> > have the key to add to myget.
> >
> > I don't know what order the infrastructure was setup in on your end,
> > but my thought was that if someone had previously pushed from MyGet
> > to NuGet the key is probably already configured there. But yea, you
> > would need access to MyGet to confirm.
> >
> > > I would love to start beating on that a bit but the .net core
> > > version
> > seems to want NUnit 3.5+ which needs to be added to the project to run.
> >
> > I will take a look at your pull request, but I think this is a
> > symptom of trying to run using the older tooling. The Build.ps1
> > script already has the ability to test, and all of the tooling is
> > there to do it (I think - maybe I should do a fresh clone to be
> > sure). It does have some prerequisites, though (see #191). It builds
> > both the .NET Framework and .NET Core versions and packages them into NuGet.
> >
> > Per #191: Hopefully Lucene.Net.sln can be removed in the future
> > because the .NET Core projects compile for .NET 4.5.1 already.
> >
> > So I think the aim is to eventually eliminate those .csproj files
> > (and for that matter .nuspec files) and use strictly .json files for
> > project configuration going forward.
> >
> >
> > Thanks,
> > Shad Storhaug (NightOwl888)
> >
> > -----Original Message-----
> > From: Wyatt Barnett
> > [mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>>>]
> > Sent: Saturday, March 25, 2017 5:00 AM
> > To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>>
> > Subject: Re: API Work/Stabilization Update
> >
> > Shad -- the overall plan sounds good. We will probably want to build
> > out .nuspec files to get all the nuget stuff right for these
> > projects
> > -- I don't think the generation will work for us to get things quite
> right.
> >
> > Regarding the nuget key -- that plan works for me, the trick is I
> > don't have the key to add to myget. Come to think of it I don't
> > think I have the proverbial keys to the myget page either but I
> > think Martin can help us out there.
> >
> > Buffers could be the issue on the tests -- I've long suspected that
> > or I/O causing the meltdown, I just haven't been able to reproduce.
> > I would love to start beating on that a bit but the .net core
> > version seems to want NUnit 3.5+ which needs to be added to the
> > project to run. If you get that added I can start beating on the
> > test problems a
> bit more.
> >
> > Thanks for all your hard work putting this together, let me know how
> > I can help you get it out the proverbial door.
> >
> > On Fri, Mar 24, 2017 at 9:34 AM Shad Storhaug
> > <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>>>
> > wrote:
> >
> > > Wyatt,
> > >
> > > Thanks. Actually, I was thinking this should go in a few steps
> > > instead of
> > > one:
> > >
> > > 1. Merge #203.
> > > 2. Change the pre-release label to "beta2" and work out any issues
> > > to build/push to MyGet (might take a few tries) 3. Update the
> > > README and CONTRIBUTING pages 4. Push the package to NuGet
> > >
> > > I have always just used the control panel at MyGet to push
> > > upstream to NuGet, and it is capable of storing someone's key so
> > > the person who pushes it doesn't actually need it.
> > >
> > > As far as the tests burning down are concerned, I discovered that
> > > some of them write so much "verbose" data that they overflow
> > > NUnit's buffer and cause it to crash (sometimes this even causes
> > > Visual Studio to crash locally). I think I have found all of the
> > > tests in the Core that were causing this and hard-coded them to
> > > set verbose off (with the ability to manually override), but I
> > > noticed that there are still tests in Analysis.Common that can cause it to crash.
> > > I haven't investigated if there is a setting in NUnit to increase
> > > the buffer size, which might be a better fix, but I could probably
> > > track down the rest of the tests that are causing this before the
> merge.
> > >
> > > Thanks,
> > > Shad Storhaug (NightOwl888)
> > >
> > > -----Original Message-----
> > > From: Wyatt Barnett
> > > [mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com><mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>>>]
> > > Sent: Friday, March 24, 2017 8:14 PM
> > > To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>>
> > > Subject: Re: API Work/Stabilization Update
> > >
> > > I'm about and happy to update when you are ready.
> > >
> > > I think the build should start working for 203 if you add the
> > > nunit-console nuget package. At least work in the sense of the
> > > build will start. I'm still chasing those build time bugbears I
> > > haven't been able to slay yet.
> > >
> > > As for getting to nuget.org<http://nuget.org><http://nuget.org><http://nuget.org> -- who has the key?
> > > I've never had access to it so I'm not sure we can update what is out there.
> > >
> > >
> > > On Fri, Mar 24, 2017 at 09:10 Shad Storhaug
> > > <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com><mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>>>
> > wrote:
> > >
> > > > Wyatt, Prescott, Itamar, anybody? We've almost got all of the
> > > > tests passing on #203 and would like to merge to master and
> > > > release it with the new pre-release label "beta2".
> > > >
> > > > If there is nobody available to get the build running after
> > > > merging, could someone give me access to TeamCity to work on it?
> > > >
> > > > Thanks,
> > > > Shad Storhaug (NightOwl888)
> > > >
> > > > From: Shad Storhaug
> > > > Sent: Tuesday, March 21, 2017 5:25 AM
> > > > To: 'dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org><mailto:dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>>>'
> > > > Subject: API Work/Stabilization Update
> > > >
> > > > I am getting very close to getting #203 merged. I wouldn't go so
> > > > far as to say that the API is finished, but the most significant
> > > > of the breaking API changes are now behind us.
> > > >
> > > > BUILD/VERSIONING
> > > >
> > > > I just wanted to be sure there is someone available to help get
> > > > the build working after the merge. I think it would be
> > > > appropriate to change the pre-release label from "beta" to
> > > > "beta2" (without resetting the build number, since that is actually what NuGet uses).
> > > > This would be primarily because of a major breaking API change,
> > > > but also to indicate another advancement toward release.
> > > >
> > > > We should probably also get this onto NuGet as soon as possible
> > > > to
> > > > (hopefully) make it easier to recruit help to stabilize and
> > > > create some integration packages for popular Microsoft frameworks.
> > > >
> > > > KNOWN ISSUES
> > > >
> > > >
> > > > 1.       The QueryParser.Flexible custom localized message
> > functionality
> > > > is currently not implemented for .NET core, so those tests are
> > > > now
> > > failing.
> > > >
> > > > 2.       The implementation of Lucene.Net.Expressions currently reads
> > > data
> > > > from the configuration file. This is not how modern libraries
> > > > are supposed to be built - instead we want any configuration to
> > > > be pushed in from the application that uses Lucene.Net. Reading
> > > > from the configuration file directly means no opportunity to use
> > > > dependency injection. There is also a namespace
> > > > Support/Configuration that can and should be removed after the
> > > > implementation is refactored to be DI-friendly (see
> > > > http://blog.ploeh.dk/2014/05/19/di-friendly-framework/). I
> > > > haven't yet worked out how the implementation was done in .NET -
> > > > in Java, the defaults were read from an embedded resource file
> > > > and could be overridden by passing in a ClassLoader (similar to
> > > > .NET's Assembly
> > > class) - if anyone has any information on how the "auto generated"
> > > C# code was generated, please share.
> > > >
> > > > 3.       The Collation functionality in Analysis.Common doesn't work
> > with
> > > > icu-dotnet, and has been excluded from compilation using the
> > > > constant FEATURE_COLLATION. I am now convinced after reading the
> > > > docs that it would be better to port the similar functionality
> > > > from Analysis.ICU because it was designed to work with icu4j and
> > > > is therefore more likely to work with icu-dotnet.
> > > >
> > > > 4.       The Highlighter PostingsHighlight and VectorHighlight
> > > > functionality relies on icu-dotnet, which doesn't have a close
> > > > match for the BreakIterator in the JRE, so there are likely some
> > > > big differences in the functionality. Several hacks were put in
> > > > to make the tests pass, but these are not likely to fix all of
> > > > the issues in the
> > > wild.
> > > >
> > > > 5.       There are several namespaces in Lucene.Net.Core and
> > > > Lucene.Net.Codecs that have broken documentation comments.
> > > >
> > > > 6.       There are some concurrency and performance issues (as
> pointed
> > > out
> > > > by Vincent Van Den Burghe):
> > > > http://git.net/ml/general/2017-02/msg00168.html
> > > >
> > > > 7.       We have around 2 dozen tests that fail during randomization
> > > > (averaging about 17 broken per run), and 8 tests that fail
> > > > all/most of the time.
> > > >
> > > > RESOLVED ISSUES (in addition to API refactoring)
> > > >
> > > >
> > > > 1.       Finished implementing the randomization of Codecs, Culture,
> > Time
> > > > Zone, and InfoStream in the TestFramework.
> > > >
> > > > 2.       Added factories for Codec, DocValuesFormat, and
> PostingsFormat
> > > so
> > > > custom implementations can be provided via dependency injection
> > > > instead of using the Java-ish NamedSPILoader class. The name
> > > > must now be provided by an attribute (or by class naming
> > > > convention) rather than via constructor, so it can be read
> > > > without creating an instance of
> > > the class.
> > > >
> > > > 3.       Fixed several of the codecs in Lucene.Net.Codecs that were
> > still
> > > > not functioning (and not being tested because of the unfinished
> > > > RandomCodec class and test mocks).
> > > >
> > > > 4.       Reviewed all catch blocks in Lucene.Net.Core to ensure the
> > right
> > > > type of exceptions are being caught and the right type re-thrown.
> > > >
> > > > 5.       Fixed culture-sensitive comparison and sort order issues
> when
> > > > using strings in Lucene.Net.Core and Lucene.Net.Codecs.
> > > >
> > > > 6.       Merged similar functionality in Support into the same class
> > and
> > > > deleted several unused Support classes.
> > > >
> > > > 7.       Made the API CLS compliant, so it now works with all .NET
> > > > languages.
> > > >
> > > >
> > > > Shad Storhaug (NightOwl888)
> > > >
> > >
> >
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message