lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wyatt Barnett <wyatt.barn...@gmail.com>
Subject Re: API Work/Stabilization Update
Date Thu, 06 Apr 2017 18:30:16 GMT
Good question Shad. I think I de-escaped things properly -- the powershell
command it is running is ./runbuild.ps1 -RunTests -FrameworksToTest
@("net451", "netcoreapp1.0")

I wholeheartedly agree that this build is complicated enough that we should
get something a little beefier than powershell in place. I was looking at
FAKE before but PSake could work too. I can't claim to have significant
seat-time with either but that is why one does these open source projects.

Anyhow more coming later this week/weekend when I can find some time to
write things up . . .

On Thu, Apr 6, 2017 at 11:37 AM Shad Storhaug <shad@shadstorhaug.com> wrote:

> Thanks.
>
>
>
> Hmm…Did you play with the escaping of quotes? What I pasted was the raw
> command to run from CMD, but that might not be what will run in TeamCity.
>
>
>
> I am not entirely certain why it sometimes prompts for that (even when you
> don’t specify to run a push), or why there are even required parameters.
> These are some of the reasons why I am suggesting that PSake might be
> better, but since I don’t know for certain if you can exit the script and
> re-enter the same target without having to execute all of the target’s
> dependencies all over again, I would like to run a few tests in TeamCity.
> Worst case, we don’t specify any dependent targets for “test” so we can run
> it multiple times without triggering the restore and build processes over
> and over (which is probably harmless, but takes up lots of time). But it
> helps if you have separate targets you can call in the script without
> having to set lots of different Boolean flags to tell it what to run and
> what not to.
>
>
>
> *From:* Wyatt Barnett [mailto:wyatt.barnett@gmail.com]
> *Sent:* Thursday, April 6, 2017 6:20 PM
>
>
> *To:* Shad Storhaug
> *Cc:* Connie Yau; dev@lucenenet.apache.org
> *Subject:* Re: API Work/Stabilization Update
>
>
>
> Hi Shad -- you now have teamcity permissions. Please be a bit careful in
> editing things -- it is a house of cards without great documentation as to
> why things are doing what and there isn't an easy rollback button like with
> source controlled code. Let me know if you want a tour of what is what and
> we can find a time to do some screen sharing.
>
>
>
> That command won't run for me on the server -- the error is:
>
>
>
> [14:14:38]W:   [Step 1/2]
> C:\BuildAgent\work\dc5a565ec74d072b\runbuild.ps1 : Cannot process command
> because of one or more missing mandatory
>
> [14:14:38]W:   [Step 1/2] parameters: NuGetApiKey UploadPackages.
>
> [14:14:38]W:   [Step 1/2]     + CategoryInfo          : InvalidArgument:
> (:) [runbuild.ps1], ParentContainsErrorRecordException
>
> [14:14:38]W:   [Step 1/2]     + FullyQualifiedErrorId :
> MissingMandatoryParameter,runbuild.ps1
>
>
>
> Is there a way to run it without trying to upload the packages -- that is
> a fairly mechanical thing we can certainly make work once we get build,
> test and packaging the right things going.
>
>
>
> I will need a bit more time than I can find on weekdays to respond to your
> other notes -- lots of good thoughts in there, they need cogent responses.
>
>
>
> On Wed, Apr 5, 2017 at 6:19 PM Shad Storhaug <shad@shadstorhaug.com>
> wrote:
>
> Wyatt,
>
> It turns out that adding the 4.5.1 framework to project.json was all that
> was required to get the tests to run, but dotnet.exe seems to crash
> whenever an exception is thrown from a test on that framework (saw 3
> different exception types - NullReferenceException,
> AccessViolationException, IndexOutOfRangeException and they all brought it
> to a halt). So, I gave the NUnit build runner a shot and it seems to be
> more stable (and much faster as well). I wasn't able to get it working on
> .NET core, but I haven't yet seen it crash when running the .NET core tests.
>
> Give it a shot to see if this will run all the way through without
> crashing in TeamCity.
>
> powershell -ExecutionPolicy Bypass -Command "& .\runbuild.ps1
> -PackageVersion 4.8.0.1000-beta2 -RunTests -FrameworksToTest @(\"net451\",
> \"netcoreapp1.0\")"
>
> Then we can work on getting this setup so tests can be run as separate
> steps.
>
> Thanks,
> Shad Storhaug (NightOwl888)
>
>
> -----Original Message-----
> From: Shad Storhaug [mailto:shad@shadstorhaug.com]
> Sent: Wednesday, April 5, 2017 8:42 PM
> To: Wyatt Barnett
> Cc: Connie Yau; dev@lucenenet.apache.org
> Subject: RE: API Work/Stabilization Update
>
> I found one issue that is preventing the tests from running - the runner
> isn't running .NET 4.5.1 framework tests at all. That framework is missing
> from project.json in the test projects altogether, so it seems this has
> never been run. I tried putting that section in, but I am getting an error
> that the "dotnet-test-nunit" target is missing on .NET 4.5.1 (and tracking
> down the cause of this doesn't seem to be plausible amongst all of the
> advice telling to "upgrade"). The build works, so I don't think it would be
> wise to upgrade just for the sake of testing.
>
> So, it seems we are back to square 1 - you are onto something with
> installing the NUnit console runner for this. That is also easier said than
> done - for some reason both Visual Studio and dotnet restore are ignoring
> the fact that I added the NuGet package to the project and are not
> downloading the binaries. I guess worst case scenario I can just check the
> binaries into the repo, but it would be more difficult to upgrade if we did
> that, so I am trying to find a fix to the NuGet download issue first.
>
> The good news is in .NET core I was able to successfully run all tests in
> the Lucene.Net.Tests project.
>
> As for splitting this up, I am thinking that starting a script with a true
> build runner like PSake is probably a good idea for the long term. Perhaps
> we could just take the test part out of the current script so we could set
> it up in TeamCity like:
>
> 1. Restore/Build step using runbuild.ps1 2. Test (using a different script
> that can be called multiple times) 3. Package/Push using runbuild.ps1
> (side-effect of going through the Restore/Build steps all over again)
>
>
> Also, allow the raw "where" clause to be passed directly to the command
> line tool rather than doing all of the building that is done currently.
> Then the tests could be setup in different steps however you like. To
> ensure we can add projects to the setup without having them ignored by
> default, I suggest:
>
> 1. Test Lucene.Net (50% of the tests)
> 2. Test Lucene.Net.Analysis.Common (25% of the tests) 3. Test everything
> but Lucene.Net and Lucene.Net.Analysis.Common (25% of the tests)
>
> And repeat for each framework.
>
> Or...wait a minute. I just noticed that one of the packages that downloads
> with NUnit.Console is a TeamCity event listener. Would it make more sense
> just to skip a test script for this (at least for .NET 4.5.1) and set it up
> using TeamCity tools?
>
> Thanks,
> Shad Storhaug (NightOwl888)
>
> -----Original Message-----
> From: Shad Storhaug [mailto:shad@shadstorhaug.com]
> Sent: Wednesday, April 5, 2017 1:10 PM
> To: Wyatt Barnett
> Cc: Connie Yau; dev@lucenenet.apache.org
> Subject: RE: API Work/Stabilization Update
>
> Wyatt,
>
> I have been getting that randomly, too (but I thought it was mainly due to
> Visual Studio locking the file). Re-running has always made it recover
> here. Basically, I have set it up so it backs up the project.json files
> before making edits so it can restore them at the end. This is primarily to
> prevent someone running locally from accidentally checking in the changes
> the build makes. However, if it cannot backup the files for some reason (a
> permission error perhaps) it won’t be able to do this step.
>
> So, looks like we need to add an option to override the backup/restore
> file process on the build server, which will work around any windows
> permission issues that prevent that from working. I will also add a check
> if the file exists before attempting to restore to make it more robust.
>
> The modified process does one very important step – it names the main
> NuGet package Lucene.Net instead of Lucene.Net.Core. I tried making the
> script compensate for the fact that dotnet.exe doesn’t support an option
> for changing the package name to be different than the folder name, but
> that became very complicated – it was simpler just to rename the folder
> from Lucene.Net.Core to Lucene.Net.
>
> As per your suggestion, we should probably break up the restore, build,
> test, and pack steps so they work separately. Unfortunately, this script
> wasn’t built with a task runner with separate entry points, so that is
> going to be a bit complicated. Do we need the part of the script that
> pushes to NuGet/MyGet, or is that something you can setup on TeamCity as a
> separate step without a script?
>
> What specific handling do you need on the version string? I set it up so
> you can pass the PackageVersion (of the NuGet package) separate from the
> Version (of the assembly). It also sets the PackageVersion as the
> AssemblyInformationalVersion of the assembly and also takes care of setting
> all of the versions for inter-package dependencies between NuGet packages.
> Basically, this makes it simple to specify only 1 parameter for version and
> the script will take care of the rest.
>
> Keep in mind, however we set it up that this process is temporary. We will
> likely change it up again when Microsoft finally gets .csproj working
> right. So, we should focus on the *interface* of the build process (that
> is, how TeamCity will interact with the script) so we can change the script
> around later without breaking the build. Perhaps we should use a task
> runner like PSake so we have separate entry points:
> https://github.com/psake/psake? I have always used it in conjunction with
> PowerShell in the past (although I have to admit, I never needed any of the
> other entry points – but then, I never had 2 hours of testing to do). That
> would eliminate the need to specify a lot of different switches to turn on
> and off the right stuff for the current task. i.e (Invoke-psake
> .\runbuild.ps1 Test -framework 4.0x64 -properties @{
> “framework”=”netcore1.0” }) would just run the tests for .NET Core and do
> nothing else (see the Spatial4n script:
> https://github.com/NightOwl888/Spatial4n/blob/master/.build/build.ps1).
>
> Build.bat is primarily to make the synatax of building simpler than what
> is required to call powershell when building locally. (build
> –pv:4.8.0.891-beta2  rather than powershell –Command “& .\runbuild.ps1
> –CreatePackages –PackageVersion 4.8.0.891-beta2”). It could be expanded
> with more parameters and/or environment variables if necessary – I find it
> simpler to maintain and test if there is both a way to manually build and
> automatically build that run through the same script (the manual part
> should be as simple as possible, the automated part only as complex as it
> needs to be for the build server to do its magic).
>
> I tried running all of the tests last night from the command line and it
> crashed during the run. Fortunately, I have a stack trace that might help
> track it down.
>
> Culture: vai-Latn-LR
> Time Zone: (UTC-05:00) Bogota, Lima, Quito, Rio Branco Default Codec:
> Lucene41 (Lucene.Net.Codecs.Lucene41.Lucene41RWCodec)
> Default Similarity: RandomSimilarityProvider(queryNorm=False,coord=yes):
> => Lucene.Net.Search.TestConstantScoreQuery.TestQueryWrapperFilter
> Passed
> Culture: vai-Latn-LR
> Time Zone: (UTC-05:00) Bogota, Lima, Quito, Rio Branco Default Codec:
> Lucene41 (Lucene.Net.Codecs.Lucene41.Lucene41RWCodec)
> Default Similarity: RandomSimilarityProvider(queryNorm=False,coord=yes):
> [field, DFR G3(800)] =>
> Lucene.Net.Search.TestConstantScoreQuery.TestWrapped2Times
> Passed
> Culture: he-IL
> Time Zone: (UTC-04:00) Asuncion
> Default Codec: Lucene3x (Lucene.Net.Codecs.Lucene3x.PreFlexRWCodec)
> Default Similarity: DefaultSimilarity
> : hit exc
> System.NullReferenceException: Object reference not set to an instance of
> an object.
>    at Lucene.Net.Util.IOUtils.ReThrow(Exception th) in
> F:\Projects\lucenenet\src\Lucene.Net\Util\IOUtils.cs:line 413
>    at Lucene.Net.Index.StoredFieldsProcessor.Flush(SegmentWriteState
> state) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\StoredFieldsProcessor.cs:line 91
>    at Lucene.Net.Index.TwoStoredFieldsConsumers.Flush(SegmentWriteState
> state) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\TwoStoredFieldsConsumers.cs:line
> 47
>    at Lucene.Net.Index.DocFieldProcessor.Flush(SegmentWriteState state) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocFieldProcessor.cs:line 84
>    at Lucene.Net.Index.DocumentsWriterPerThread.Flush() in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriterPerThread.cs:line
> 573
>    at Lucene.Net.Index.DocumentsWriter.DoFlush(DocumentsWriterPerThread
> flushingDWPT) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 638
>    at Lucene.Net.Index.DocumentsWriter.PreUpdate() in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 461
>    at Lucene.Net.Index.DocumentsWriter.UpdateDocuments(IEnumerable`1 docs,
> Analyzer analyzer, Term delTerm) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 513
>    at Lucene.Net.Index.IndexWriter.UpdateDocuments(Term delTerm,
> IEnumerable`1 docs, Analyzer analyzer) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\IndexWriter.cs:line 1562
>    at Lucene.Net.Index.TrackingIndexWriter.UpdateDocuments(Term t,
> IEnumerable`1 docs) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\TrackingIndexWriter.cs:line 103
>    at
> Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocuments(Term
> id, IEnumerable`1 docs) in
> F:\Projects\lucenenet\src\Lucene.Net.Tests\Search\TestControlledRealTimeReopenThread.cs:line
> 116
>    at
> Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run()
> in
> F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Index\ThreadedIndexingAndSearchingTestCase.cs:line
> 276
>    at Lucene.Net.Util.IOUtils.ReThrow(Exception th) in
> F:\Projects\lucenenet\src\Lucene.Net\Util\IOUtils.cs:line 413
>    at Lucene.Net.Index.StoredFieldsProcessor.Flush(SegmentWriteState
> state) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\StoredFieldsProcessor.cs:line 91
>    at Lucene.Net.Index.TwoStoredFieldsConsumers.Flush(SegmentWriteState
> state) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\TwoStoredFieldsConsumers.cs:line
> 47
>    at Lucene.Net.Index.DocFieldProcessor.Flush(SegmentWriteState state) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocFieldProcessor.cs:line 84
>   index=_0(4.2):C10/1:delGen=1 _1(4.2):C10 _2(4.2):C10 _3(4.2):C5
> _4(4.2):C1IW 8 [5/4/2017 01:48:05; ]: merging _1(4.2):C10 _2(4.2):C10
> _0(4.2):C10/1:delGen=1 _3(4.2):C5 _4(4.2):C1IW 8 [5/4/2017 01:48:05; ]:
> seg=_1(4.2):C10 no deletesIW 8 [5/4/2017 01:48:05; ]: seg=_2(4.2):C10 no
> deletesIW 8 [5/4/2017 01:48:05; ]: seg=_0(4.2):C10/1:delGen=1 delCount=1IW
> 8 [5/4/2017 01:48:05; ]: seg=_3(4.2):C5 no deletesIW 8 [5/4/2017 01:48:05;
> ]: seg=_4(4.2):C1 no deletesSM 8 [5/4/2017 01:48:05; ]: merge store
> matchedCount=5 vs 5SM 8 [5/4/2017 01:48:05; ]: 0 msec to merge stored
> fields [35 docs]SM 8 [5/4/2017 01:48:05; ]: 3 msec to merge postings [35
> docs]SM 8 [5/4/2017 01:48:05; ]: 5 msec to merge doc values [35 docs]SM 8
> [5/4/2017 01:48:05; ]: 0 msec to merge norms [35 docs]SM 8 [5/4/2017
> 01:48:05; ]: 1 msec to merge vectors [35 docs]IW 8 [5/4/2017 01:48:05; ]:
> merge codec=Lucene46: [[trieLong, PostingsFormat(name=Memory doPackFST=
> False)], [content2, PostingsFormat(name=Memory doPackFST= True)], [id,
> PostingsFormat(name=Memory doPackFST= True)], [content5, FST41], [autf8,
> PostingsFormat(name=Direct)], [trieInt, PostingsFormat(name=Memory
> doPackFST= False)], [utf8, PostingsFormat(name=Memory doPackFST= False)],
> [fieΓ▒╖ld, FST41], [content3, PostingsFormat(name=Direct)], [content,
> PostingsFormat(name=Memory doPackFST= True)], [content6,
> PostingsFormat(name=Direct)]], docValues:[[dvSortedSet,
> DocValuesFormat(name=Disk)], [dvPacked, DocValuesFormat(name=Disk)],
> [dvBytesStraightVar, DocValuesFormat(name=Asserting)], [dvLong,
> DocValuesFormat(name=Disk)], [dvInt, DocValuesFormat(name=Memory)],
> [dvBytesDerefFixed, DocValuesFormat(name=Memory)], [dvFloat,
> DocValuesFormat(name=Asserting)], [dvBytesSortedFixed,
> DocValuesFormat(name=SimpleText)], [dvBytesStraightFixed,
> DocValuesFormat(name=Memory)], [dvBytesSortedVar,
> DocValuesFormat(name=Disk)], [dvDouble, DocValuesFormat(name=Disk)],
> [dvBytesDerefVar, DocValuesFormat(name=Memory)], [dvByte,
> DocValuesFormat(name=Disk)], [dvShort, DocValuesFormat(name=Disk)]]
> docCount=35; merged segment has vectors; norms; docValues; prox; freqsIW 8
> [5/4/2017 01:48:05; ]: merged segment size=%.3f MB vs estimate=%.3f MBTP 8
> [5/4/2017 01:48:05; ]: startCommitMergeIW 8 [5/4/2017 01:48:05; ]:
> commitMerge: _1(4.2):C10 _2(4.2):C10 _0(4.2):C10/1:delGen=1 _3(4.2):C5
> _4(4.2):C1 index=_0(4.2):C10/1:delGen=1 _1(4.2):C10 _2(4.2):C10 _3(4.2):C5
> _4(4.2):C1TP 8 [5/4/2017 01:48:05; ]: startCommitMergeDeletesIW 8 [5/4/2017
> 01:48:05; ]: commitMergeDeletes _1(4.2):C10 _2(4.2):C10
> _0(4.2):C10/1:delGen=1 _3(4.2):C5 _4(4.2):C1IW 8 [5/4/2017 01:48:05; ]: no
> new deletes or field updates since merge startedIFD 8 [5/4/2017 01:48:05;
> ]: now checkpoint "_5(4.8):C35" [1 segments ; isCommit = False]IFD 8
> [5/4/2017 01:48:05; ]: 0 msec to checkpointIW 8 [5/4/2017 01:48:05; ]:
> after commitMerge: _5(4.8):C35UPGMP 8 [5/4/2017 01:48:05; ]:
> findForcedMerges:
> segmentsToUpgrade=System.Collections.Generic.Dictionary`2[Lucene.Net.Index.SegmentCommitInfo,System.Nullable`1[System.Boolean]]IW
> 8 [5/4/2017 01:48:05; ]: merge time 0 msec for 35 docsIndexUpgrader 8
> [5/4/2017 01:48:05; ]: All segments upgraded to version 4.8CMS 8 [5/4/2017
> 01:48:05; ]:   merge thread: doneIW 8 [5/4/2017 01:48:05; ]: now flush at
> close waitForMerges=TrueTP 8 [5/4/2017 01:48:05; ]: startDoFlushIW 8
> [5/4/2017 01:48:05; ]:   start flush: applyAllDeletes=TrueIW 8 [5/4/2017
> 01:48:05; ]:   index before flush _5(4.8):C35DW 8 [5/4/2017 01:48:05; ]:
> startFullFlushDW 8 [5/4/2017 01:48:05; ]: anyChanges? numDocsInRam=0
> deletes=False hasTickets:False pendingChangesInFullFlush: FalseDW 8
> [5/4/2017 01:48:05; ]:  finishFullFlush success=TrueIW 8 [5/4/2017
> 01:48:05; ]: apply all deletes during flushBD 8 [5/4/2017 01:48:05; ]:
> applyDeletes: no deletes; skippingBD 8 [5/4/2017 01:48:05; ]: prune
> sis=Lucene.Net.Index.SegmentInfos minGen=0 packetCount=0CMS 8 [5/4/2017
> 01:48:05; ]: now mergeCMS 8 [5/4/2017 01:48:05; ]:   index: _5(4.8):C35CMS
> 8 [5/4/2017 01:48:05; ]:   no more merges pending; now returnIW 8 [5/4/2017
> 01:48:05; ]: waitForMergesIW 8 [5/4/2017 01:48:05; ]: waitForMerges doneIW
> 8 [5/4/2017 01:48:05; ]: now call final commit()IW 8 [5/4/2017 01:48:05; ]:
> commit: startIW 8 [5/4/2017 01:48:05; ]: commit: enter lockIW 8 [5/4/2017
> 01:48:05; ]: commit: now prepareIW 8 [5/4/2017 01:48:05; ]: prepareCommit:
> flushIW 8 [5/4/2017 01:48:05; ]:   index before flush _5(4.8):C35TP 8
> [5/4/2017 01:48:05; ]: startDoFlushDW 8 [5/4/2017 01:48:05; ]:
> startFullFlushDW 8 [5/4/2017 01:48:05; ]: anyChanges? numDocsInRam=0
> deletes=False hasTickets:False pendingChangesInFullFlush: FalseIW 8
> [5/4/2017 01:48:05; ]: apply all deletes during flushBD 8 [5/4/2017
> 01:48:05; ]: applyDeletes: no deletes; skippingBD 8 [5/4/2017 01:48:05; ]:
> prune sis=Lucene.Net.Index.SegmentInfos minGen=0 packetCount=0DW 8
> [5/4/2017 01:48:05; ]:  finishFullFlush success=TrueTP 8 [5/4/2017
> 01:48:05; ]: startStartCommitIW 8 [5/4/2017 01:48:05; ]: StartCommit():
> startIW 8 [5/4/2017 01:48:05; ]: startCommit index=_5(4.8):C35
> changeCount=2TP 8 [5/4/2017 01:48:05; ]: midStartCommitTP 8 [5/4/2017
> 01:48:05; ]: midStartCommit2IW 8 [5/4/2017 01:48:05; ]: done all syncs:
> _5.fdx, _5.fdt, _5_Direct_0.doc, _5_Direct_0.pos, _5_Direct_0.pay,
> _5_Direct_0.tim, _5_Direct_0.tip, _5_Memory_0.ram, _5_FST41_0.doc,
> _5_FST41_0.pos, _5_FST41_0.pay, _5_FST41_0.tmp, _5_Memory_1.ram,
> _5_Disk_0.dvdd, _5_Disk_0.dvdm, _5_Memory_0.mdvd, _5_Memory_0.mdvm,
> _5_SimpleText_0.dat, _5_Asserting_0.dvd, _5_Asserting_0.dvm, _5.nvd,
> _5.nvm, _5.tvx, _5.tvd, _5.fnm, _5.siTP 8 [5/4/2017 01:48:05; ]:
> midStartCommitSuccessTP 8 [5/4/2017 01:48:05; ]: finishStartCommitIW 8
> [5/4/2017 01:48:05; ]: commit: pendingCommit != nullIW 8 [5/4/2017
> 01:48:05; ]: commit: wrote segments file "segments_4"IFD 8 [5/4/2017
> 01:48:05; ]: now checkpoint "_5(4.8):C35" [1 segments ; isCommit = True]IFD
> 8 [5/4/2017 01:48:05; ]: deleteCommits: now decRef commit "segments_3"IFD 8
> [5/4/2017 01:48:05; ]: delete "segments_3"IFD 8 [5/4/2017 01:48:05; ]:
> delete "_0.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_Lucene42_0.dvd"IFD 8
> [5/4/2017 01:48:05; ]: delete "_0_Lucene41_0.pos"IFD 8 [5/4/2017 01:48:05;
> ]: delete "_0.tvd"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_0_Lucene42_0.dvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_0.nvm"IFD 8
> [5/4/2017 01:48:05; ]: delete "_0_Lucene41_0.pay"IFD 8 [5/4/2017 01:48:05;
> ]: delete "_0.tvx"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_0_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete "_0.nvd"IFD 8
> [5/4/2017 01:48:05; ]: delete "_0.fdx"IFD 8 [5/4/2017 01:48:05; ]: delete "_
> 0.si"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_Lucene41_0.tim"IFD 8
> [5/4/2017 01:48:05; ]: delete "_0.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_0_Lucene41_0.tip"IFD 8 [5/4/2017 01:48:05; ]: delete "_0_1.del"IFD 8
> [5/4/2017 01:48:05; ]: delete "_1.tvx"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_1.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_1_Lucene41_0.doc"IFD 8
> [5/4/2017 01:48:05; ]: delete "_1_Lucene42_0.dvm"IFD 8 [5/4/2017 01:48:05;
> ]: delete "_1_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_1.nvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_1_Lucene41_0.tip"IFD 8
> [5/4/2017 01:48:05; ]: delete "_1_Lucene42_0.dvd"IFD 8 [5/4/2017 01:48:05;
> ]: delete "_1.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_1_Lucene41_0.pos"IFD 8 [5/4/2017 01:48:05; ]: delete "_1.fdx"IFD 8
> [5/4/2017 01:48:05; ]: delete "_1_Lucene41_0.pay"IFD 8 [5/4/2017 01:48:05;
> ]: delete "_1.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete "_1.si"IFD 8
> [5/4/2017 01:48:05; ]: delete "_1.tvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_
> 2.si"IFD 8 [5/4/2017 01:48:05; ]: delete "_2_Lucene41_0.pos"IFD 8
> [5/4/2017 01:48:05; ]: delete "_2_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05;
> ]: delete "_2.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_2_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_2_Lucene41_0.tip"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.fdx"IFD 8
> [5/4/2017 01:48:05; ]: delete "_2.tvx"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_2.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.tvd"IFD 8 [5/4/2017
> 01:48:05; ]: delete "_2.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_2_Lucene42_0.dvm"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_2_Lucene41_0.pay"IFD 8 [5/4/2017 01:48:05; ]: delete "_2.nvd"IFD 8
> [5/4/2017 01:48:05; ]: delete "_2_Lucene42_0.dvd"IFD 8 [5/4/2017 01:48:05;
> ]: delete "_3.tvd"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_3_Lucene41_0.pay"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.fdt"IFD 8
> [5/4/2017 01:48:05; ]: delete "_3.fnm"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_3.fdx"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.tvx"IFD 8 [5/4/2017
> 01:48:05; ]: delete "_3.nvd"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_3_Lucene41_0.pos"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_3_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_3_Lucene41_0.tip"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_3_Lucene42_0.dvm"IFD 8 [5/4/2017 01:48:05; ]: delete "_3.si"IFD 8
> [5/4/2017 01:48:05; ]: delete "_3_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05;
> ]: delete "_3.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_3_Lucene42_0.dvd"IFD 8 [5/4/2017 01:48:05; ]: delete "_4.fdx"IFD 8
> [5/4/2017 01:48:05; ]: delete "_4.nvd"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_4_Lucene41_0.doc"IFD 8 [5/4/2017 01:48:05; ]: delete "_4.fnm"IFD 8
> [5/4/2017 01:48:05; ]: delete "_4.si"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_4.fdt"IFD 8 [5/4/2017 01:48:05; ]: delete "_4_Lucene41_0.tip"IFD 8
> [5/4/2017 01:48:05; ]: delete "_4.nvm"IFD 8 [5/4/2017 01:48:05; ]: delete
> "_4_Lucene41_0.tim"IFD 8 [5/4/2017 01:48:05; ]: 0 msec to checkpointIW 8
> [5/4/2017 01:48:05; ]: commit: doneIW 8 [5/4/2017 01:48:05; ]: at close:
> _5(4.8):C35
>    at Lucene.Net.Index.DocumentsWriterPerThread.Flush() in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriterPerThread.cs:line
> 573
>    at Lucene.Net.Index.DocumentsWriter.DoFlush(DocumentsWriterPerThread
> flushingDWPT) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 638
>    at Lucene.Net.Index.DocumentsWriter.PreUpdate() in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 461
>    at Lucene.Net.Index.DocumentsWriter.UpdateDocuments(IEnumerable`1 docs,
> Analyzer analyzer, Term delTerm) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 513
>    at Lucene.Net.Index.IndexWriter.UpdateDocuments(Term delTerm,
> IEnumerable`1 docs, Analyzer analyzer) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\IndexWriter.cs:line 1562
>    at Lucene.Net.Index.TrackingIndexWriter.UpdateDocuments(Term t,
> IEnumerable`1 docs) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\TrackingIndexWriter.cs:line 103
>    at
> Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocuments(Term
> id, IEnumerable`1 docs) in
> F:\Projects\lucenenet\src\Lucene.Net.Tests\Search\TestControlledRealTimeReopenThread.cs:line
> 116 Unhandled Exception: System.Exception: System.NullReferenceException:
> Object reference not set to an instance of an object.
>    at Lucene.Net.Util.IOUtils.ReThrow(Exception th) in
> F:\Projects\lucenenet\src\Lucene.Net\Util\IOUtils.cs:line 413
>    at Lucene.Net.Index.StoredFieldsProcessor.Flush(SegmentWriteState
> state) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\StoredFieldsProcessor.cs:line 91
>    at Lucene.Net.Index.TwoStoredFieldsConsumers.Flush(SegmentWriteState
> state) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\TwoStoredFieldsConsumers.cs:line
> 47
>    at Lucene.Net.Index.DocFieldProcessor.Flush(SegmentWriteState state) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocFieldProcessor.cs:line 84
>    at Lucene.Net.Index.DocumentsWriterPerThread.Flush() in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriterPerThread.cs:line
> 573
>    at Lucene.Net.Index.DocumentsWriter.DoFlush(DocumentsWriterPerThread
> flushingDWPT) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 638
>    at Lucene.Net.Index.DocumentsWriter.PreUpdate() in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 461
>    at Lucene.Net.Index.DocumentsWriter.UpdateDocuments(IEnumerable`1 docs,
> Analyzer analyzer, Term delTerm) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\DocumentsWriter.cs:line 513
>    at Lucene.Net.Index.IndexWriter.UpdateDocuments(Term delTerm,
> IEnumerable`1 docs, Analyzer analyzer) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\IndexWriter.cs:line 1562
>    at Lucene.Net.Index.TrackingIndexWriter.UpdateDocuments(Term t,
> IEnumerable`1 docs) in
> F:\Projects\lucenenet\src\Lucene.Net\Index\TrackingIndexWriter.cs:line 103
>    at
> Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocuments(Term
> id, IEnumerable`1 docs) in
> F:\Projects\lucenenet\src\Lucene.Net.Tests\Search\TestControlledRealTimeReopenThread.cs:line
> 116
>    at
> Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run()
> in
> F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Index\ThreadedIndexingAndSearchingTestCase.cs:line
> 276 ---> System.NullReferenceException: Object reference not set to an
> instance of an object.
>    at Lucene.Net.Util.IOUtils.ReThrow(Exception th)
>    at Lucene.Net.Index.StoredFieldsProcessor.Flush(SegmentWriteState state)
>    at Lucene.Net.Index.TwoStoredFieldsConsumers.Flush(SegmentWriteState
> state)
>    at Lucene.Net.Index.DocFieldProcessor.Flush(SegmentWriteState state)
>    at Lucene.Net.Index.DocumentsWriterPerThread.Flush()
>    at Lucene.Net.Index.DocumentsWriter.DoFlush(DocumentsWriterPerThread
> flushingDWPT)
>    at Lucene.Net.Index.DocumentsWriter.PreUpdate()
>    at Lucene.Net.Index.DocumentsWriter.UpdateDocuments(IEnumerable`1 docs,
> Analyzer analyzer, Term delTerm)
>    at Lucene.Net.Index.IndexWriter.UpdateDocuments(Term delTerm,
> IEnumerable`1 docs, Analyzer analyzer)
>    at Lucene.Net.Index.TrackingIndexWriter.UpdateDocuments(Term t,
> IEnumerable`1 docs)
>    at
> Lucene.Net.Search.TestControlledRealTimeReopenThread.UpdateDocuments(Term
> id, IEnumerable`1 docs)
>    at
> Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run()
>    --- End of inner exception stack trace ---
>    at
> Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run()
>    at System.Threading.ExecutionContext.Run(ExecutionContext
> executionContext, ContextCallback callback, Object state)
>    at
> Lucene.Net.Index.ThreadedIndexingAndSearchingTestCase.ThreadAnonymousInnerClassHelper.Run()
> in
> F:\Projects\lucenenet\src\Lucene.Net.TestFramework\Index\ThreadedIndexingAndSearchingTestCase.cs:line
> 276
> WARNING: Could not find TestResult.xml.
> Testing [Lucene.Net.Tests.Analysis.Common] on [net451]...
> dotnet.exe test --configuration Release --framework net451 --no-build
> Project
> 'F:\Projects\lucenenet\src\Lucene.Net.Tests.Analysis.Common\project.json'
> does not support framework: net451
> WARNING: Could not find TestResult.xml.
> Testing [Lucene.Net.Tests.Analysis.Common] on [netcoreapp1.0]...
> dotnet.exe test --configuration Release --framework netcoreapp1.0
> --no-build NUnit .NET Core Runner 3.4.0 Copyright (C) 2016 Charlie Poole
> Runtime Environment
>     OS Platform: Windows
>      OS Version: 10.0.14393
>         Runtime: win10-x64
> Test Files
>
> F:\Projects\lucenenet\src\Lucene.Net.Tests.Analysis.Common\bin\Release\netcoreapp1.0\Lucene.Net.Tests.Analysis.Common.dll
> Terminate batch job (Y/N)? y
>
> And yea, I see that it is having trouble finding TestResult.xml files,
> too. Perhaps that move step also needs to be made optional or taken out
> entirely.
>
>
> Thanks,
> Shad Storhaug (NightOwl888)
>
>
>
> From: Wyatt Barnett [mailto:wyatt.barnett@gmail.com]
> Sent: Wednesday, April 5, 2017 4:43 AM
> To: Shad Storhaug
> Cc: Connie Yau; dev@lucenenet.apache.org
> Subject: Re: API Work/Stabilization Update
>
> Update -- build was building and testing as of yesterday evening, it looks
> like some of the build.ps1 changes blew up whatever was working there, the
> build is now failing because it can't find some project.json.bak file . . .
>
> On Tue, Apr 4, 2017 at 4:41 PM Wyatt Barnett <wyatt.barnett@gmail.com
> <mailto:wyatt.barnett@gmail.com>> wrote:
> Hi Shad -- sorry I have been trying to get the whole thing working on
> TeamCity for the last few days. Unfortunately test runs take upwards of an
> hour and I have to remember to get back to it to check so progress is slow.
> In any case I'm actually very close to getting the fundamentals worked out
> -- see
> https://teamcity.jetbrains.com/viewType.html?buildTypeId=LuceneNet_Vs2015LuceneNetPortable
> for the specific build project. At this point I can get it to build and run
> all the tests. The problem is reading the test results -- build.ps1 makes
> the nunit XML files but it moves them and for some reason TeamCity didn't
> like that. Is there any reason it was taking that copy step connie?
>
> Project.json vs Project.csproj is pretty immaterial to me.
>
> Anyhow, it does appear the build server has the pre-requisites required to
> do so. As for teamcity access I would create an account at
> https://teamcity.jetbrains.com and then let me know what the user / email
> is and I'll see if I can get them to patch you in.
>
> FWIW build.bat probably doesn't help much -- we need a bit more elegant
> handling of the version string, and probably want to run separate tests for
> each target so we know what is failing.
>
>
>
> On Tue, Apr 4, 2017 at 1:31 PM Shad Storhaug <shad@shadstorhaug.com
> <mailto:shad@shadstorhaug.com>> wrote:
> Update
>
> I have made it past the hurdles with the build, and got the build working
> on MyGet. But since MyGet has a build timeout of 15 minutes, there is no
> way to run the tests there.
>
> So, now we need to get it working on TeamCity. AFAIK I don't have access
> to it, so if someone could either help me out or provide access that would
> be great.
>
> project.json files are NOT the new way (as I previously thought) -
> Microsoft has already scrapped this idea. I guess back when Connie was
> working on it things were still up in the air:
> http://fizzylogic.nl/2016/05/11/project-json-is-going-away-and-it-s-a-good-thing/
> - and actually they still are. So, we might have to keep it this way for
> now and then change it around at some later point when Microsoft officially
> announces that multi-targeting support is released, then we can go back to
> a single solution and single project file per project.
>
> We need to have the .NET Core SDK installed on the build server. Although
> it is possible to download the binaries, they are more than 100MB when
> unpacked. So, here is the list or prerequisites for the build server.
>
> PowerShell version 3.0 or higher
> Git for Windows
> .NET Core 1.1 with SDK Preview 2.1 build 3177 (
> https://github.com/dotnet/core/blob/master/release-notes/download-archive.md
> )
>
> To run the build, execute "build.bat".
>
> Environment Variables:
>
> %PackageVersion% - Entire version string including pre-release tag
> %RunAllTests% - Pass "true" to bypass the 4 parameters that you would
> otherwise have to set on runbuild.ps1 in order to run all of the tests for
> both .NET Framework and .NET Core
>
>
> Thanks,
> Shad Storhaug (NightOwl888)
>
>
> -----Original Message-----
> From: Shad Storhaug [mailto:shad@shadstorhaug.com<mailto:
> shad@shadstorhaug.com>]
> Sent: Sunday, April 2, 2017 4:10 PM
> To: wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>
> Cc: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>; Connie Yau
> Subject: RE: API Work/Stabilization Update
>
> Update
>
> I tried running the tests with verbosity on and NUnit is misbehaving in
> ways I didn't previously realize. There is apparently some resource leak
> that is causing a buildup over several tests and then once it reaches a
> certain threshold all of the tests will fail with an OutOfMemoryException.
> For now, I am going to consider fixing verbosity low priority - instead we
> should disable it for the time being and focus on the build/test runs.
>
> Looking through the build script there are few things of note:
>
> 1. It skips long running tests and tests that are marked with a timeout by
> default 2. It runs in Release by default 3. Connie has listed Visual Studio
> 2015 Update 3 as a prerequisite
>
> Clearly, this isn't what we want. We shouldn't be skipping any tests on
> the build server except for those that have been manually ignored.
> Verbosity is being ignored in the build script by default, which explains
> why the verbosity issue is not causing the script to crash.
>
> I am also trying to work out if there is a way to run without Visual
> Studio. The script depends on dotnet.exe, which (I think) is installed by
> the .NET Core SDK. I am attempting to find out if we really need the SDK or
> if the executable is all we need. Either way, I don't see this leaving the
> ground without having .NET Framework 4.5.1 and (possibly) .NET Core 1.1
> installed on the server.
>
> I guess this leaves a few questions:
>
> 1. Is it safe to assume the .NET Framework 4.5.1 and .NET Core 1.1
> runtimes will be installed on the server?
> 2. Can we install the .NET Core SDK on the server, or is our only option
> to find out if the tools we need are portable (I can't imagine they are
> not, but you never know)?
>
>
> Thanks,
> Shad Storhaug (NightOwl888)
>
> -----Original Message-----
> From: Wyatt Barnett [mailto:wyatt.barnett@gmail.com<mailto:
> wyatt.barnett@gmail.com>]
> Sent: Saturday, April 1, 2017 7:52 AM
> To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
> Subject: Re: API Work/Stabilization Update
>
> I wanted to walk back to one thing you asked about the tests Shad:
>
> "Worst case, we can just turn off verbosity on the build, but it might be
> helpful to leave it on if something goes wrong"
>
> I *think* I'm turning the verbosity off in the tests -- they should be
> running in release mode. They would not even run in debug if I recall
> correctly. Let me know how the switch works and I'll make sure I'm throwing
> it on the build server side.
>
> On Fri, Mar 31, 2017 at 6:56 PM Shad Storhaug <shad@shadstorhaug.com
> <mailto:shad@shadstorhaug.com>> wrote:
>
> > I don't necessarily need ownership access, but he did give me
> > ownership to the https://www.nuget.org/packages/Spatial4n.Core/ package.
> > Alternatively, Itamar can enter his keys on
> > https://www.myget.org/feed/Security/spatial4n - he already has
> ownership.
> > Once the keys are added there, I will be able to push.
> >
> > -----Original Message-----
> > From: Prescott Nasser
> > [mailto:geobmx540@hotmail.com<mailto:geobmx540@hotmail.com>]
> > Sent: Saturday, April 1, 2017 5:49 AM
> > To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
> > Subject: RE: API Work/Stabilization Update
> >
> > Access like ownership access? Paging Itamar..
> >
> >
> >
> > -----Original Message-----
> > From: Shad Storhaug
> > [mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>]
> > Sent: Friday, March 31, 2017 3:38 PM
> > To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
> > Subject: RE: API Work/Stabilization Update
> >
> > BTW - the contrib NuGet package is now dead - the functionality has
> > been moved into new sub-projects just like in Lucene.
> >
> > But, that reminds me - I still don't have access to
> > https://www.nuget.org/packages/Spatial4n.Core.NTS/, so I have been
> > unable to update it to the same version that
> > https://www.nuget.org/packages/Spatial4n.Core/ is on. NTS is not
> > required by Lucene.Net, but it is required for anyone who needs to run
> the tests.
> >
> > -----Original Message-----
> > From: Shad Storhaug
> > [mailto:shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>]
> > Sent: Saturday, April 1, 2017 5:31 AM
> > To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
> > Subject: RE: API Work/Stabilization Update
> >
> > Wyatt,
> >
> > Great. Yea, actually after I sent my last email I was starting to
> > wonder if Connie ever tested the build on a box without Visual Studio
> installed.
> > We might need to determine what the prerequisites for the build are so
> > we know what to put on the build server.
> >
> > I haven't yet had a chance to determine what tests are causing the
> > build to crash (although, I know I can reliably reproduce it). I plan
> > on getting that straightened out shortly. Worst case, we can just turn
> > off verbosity on the build, but it might be helpful to leave it on if
> > something goes wrong. I'll also look into the build dependencies and the
> script itself.
> >
> > I am just going through now and deleting all of the old 3.x source
> > files that are now just causing noise in the project and resurrecting
> > the tests for the support classes. But I am nearly finished. If you
> > are available, I would like to try to get the beta2 release done over
> > the weekend (at least to the point where it is on MyGet). We need to
> > get the README and CONTRIBUTING files updated with the latest status
> > (it would be helpful to sync the wiki page as well:
> > https://cwiki.apache.org/confluence/display/LUCENENET/Current+Status -
> > I don't think I have access for that).
> >
> > Several issues regarding the main readme need to be addressed:
> >
> > 1. There is no link to the wiki
> > 2. There is no link to the issue tracker 3. It is not clear what the
> > status is 4. There are no build instructions 5. There is no mention
> > that we need help to finish 6. There is no link to the license 7. The
> > documentation links are out of date 8. The top part of the repo
> > already lists the files, do we really need to describe them again?
> >
> > Itamar mentioned he would be working on a new web site - I am not sure
> > what the status of that is, but either way these issues are causing
> > friction with anyone who is willing to help, but can't navigate
> > through these hurdles. People who are used to working with GitHub, its
> > issue tracker, and wiki don't find it natural to go looking somewhere
> > else to these tools. For beta testing, we definitely need to make it
> > crystal clear how to report bugs.
> >
> > Also, the "known issues" list is getting short enough so I can start
> > adding them to the issue tracker.
> >
> > I have been using the downtime while running tests to fix up the
> > documentation comments in Lucene.Net.Core, but it's still only about
> > 50% done. I could use some help getting them fixed so they are at
> > least visible in Visual Studio intellisense. See the latest status on
> > #203. And then of course we can use them to generate new documents.
> >
> >
> > Thanks,
> > Shad Storhaug (NightOwl888)
> >
> >
> > -----Original Message-----
> > From: Wyatt Barnett
> > [mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>]
> > Sent: Saturday, April 1, 2017 4:22 AM
> > To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
> > Subject: Re: API Work/Stabilization Update
> >
> > Shad -- definitely makes sense.
> >
> > Json files are fine -- functionally this is a bit too fancy to use
> > Teamcity's automagic nuget package generation so as long as we've got
> > a file to edit we are fine.
> >
> > Myget -> nuget works for me but that doesn't solve the key problem. I
> > don't have it, maybe Prescott or Itamar know where it is kept but I
> > can't claim to have ever seen it. I joined this party after the last
> nuget push.
> >
> > It is a bit foggy but I think I ran into the nunit-console issue with
> > the
> > Build.ps1 script. Remember that with build servers the pre-requisites
> > often need to be embedded in the project for things to work properly.
> >
> > Anyhow, let me know when you are in a good place with your branch to
> > start slogging through getting the new build working. In the interests
> > of full disclosure I'm working an event the last week and a half of
> > April and will be completely out of pocket then. But I'm about otherwise.
> >
> > On Sat, Mar 25, 2017 at 10:04 AM Shad Storhaug
> > <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>
> > wrote:
> >
> > > Wyatt,
> > >
> > > > We will probably want to build out .nuspec files to get all the
> > > > nuget
> > > stuff right for these projects -- I don't think the generation will
> > > work for us to get things quite right.
> > >
> > > Connie has set us up to use .json files instead of .nuspec files to
> > > generate the NuGet packages (the new way instead of the old way).
> > > The build script Build.ps1 does it all (it even has help
> > > documentation), but it is missing an option to override versioning.
> > > Ideally we would be able to override the version that is in the
> > > .json file with an environment variable (which you can pass from
> > > TeamCity), and be able to override that on the CLI for local
> > > "one-off" builds.  See the build
> > instructions on #191:
> > > https://github.com/apache/lucenenet/pull/191
> > >
> > > > Regarding the nuget key -- that plan works for me, the trick is I
> > > > don't
> > > have the key to add to myget.
> > >
> > > I don't know what order the infrastructure was setup in on your end,
> > > but my thought was that if someone had previously pushed from MyGet
> > > to NuGet the key is probably already configured there. But yea, you
> > > would need access to MyGet to confirm.
> > >
> > > > I would love to start beating on that a bit but the .net core
> > > > version
> > > seems to want NUnit 3.5+ which needs to be added to the project to run.
> > >
> > > I will take a look at your pull request, but I think this is a
> > > symptom of trying to run using the older tooling. The Build.ps1
> > > script already has the ability to test, and all of the tooling is
> > > there to do it (I think - maybe I should do a fresh clone to be
> > > sure). It does have some prerequisites, though (see #191). It builds
> > > both the .NET Framework and .NET Core versions and packages them into
> NuGet.
> > >
> > > Per #191: Hopefully Lucene.Net.sln can be removed in the future
> > > because the .NET Core projects compile for .NET 4.5.1 already.
> > >
> > > So I think the aim is to eventually eliminate those .csproj files
> > > (and for that matter .nuspec files) and use strictly .json files for
> > > project configuration going forward.
> > >
> > >
> > > Thanks,
> > > Shad Storhaug (NightOwl888)
> > >
> > > -----Original Message-----
> > > From: Wyatt Barnett
> > > [mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>]
> > > Sent: Saturday, March 25, 2017 5:00 AM
> > > To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
> > > Subject: Re: API Work/Stabilization Update
> > >
> > > Shad -- the overall plan sounds good. We will probably want to build
> > > out .nuspec files to get all the nuget stuff right for these
> > > projects
> > > -- I don't think the generation will work for us to get things quite
> > right.
> > >
> > > Regarding the nuget key -- that plan works for me, the trick is I
> > > don't have the key to add to myget. Come to think of it I don't
> > > think I have the proverbial keys to the myget page either but I
> > > think Martin can help us out there.
> > >
> > > Buffers could be the issue on the tests -- I've long suspected that
> > > or I/O causing the meltdown, I just haven't been able to reproduce.
> > > I would love to start beating on that a bit but the .net core
> > > version seems to want NUnit 3.5+ which needs to be added to the
> > > project to run. If you get that added I can start beating on the
> > > test problems a
> > bit more.
> > >
> > > Thanks for all your hard work putting this together, let me know how
> > > I can help you get it out the proverbial door.
> > >
> > > On Fri, Mar 24, 2017 at 9:34 AM Shad Storhaug
> > > <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>
> > > wrote:
> > >
> > > > Wyatt,
> > > >
> > > > Thanks. Actually, I was thinking this should go in a few steps
> > > > instead of
> > > > one:
> > > >
> > > > 1. Merge #203.
> > > > 2. Change the pre-release label to "beta2" and work out any issues
> > > > to build/push to MyGet (might take a few tries) 3. Update the
> > > > README and CONTRIBUTING pages 4. Push the package to NuGet
> > > >
> > > > I have always just used the control panel at MyGet to push
> > > > upstream to NuGet, and it is capable of storing someone's key so
> > > > the person who pushes it doesn't actually need it.
> > > >
> > > > As far as the tests burning down are concerned, I discovered that
> > > > some of them write so much "verbose" data that they overflow
> > > > NUnit's buffer and cause it to crash (sometimes this even causes
> > > > Visual Studio to crash locally). I think I have found all of the
> > > > tests in the Core that were causing this and hard-coded them to
> > > > set verbose off (with the ability to manually override), but I
> > > > noticed that there are still tests in Analysis.Common that can cause
> it to crash.
> > > > I haven't investigated if there is a setting in NUnit to increase
> > > > the buffer size, which might be a better fix, but I could probably
> > > > track down the rest of the tests that are causing this before the
> > merge.
> > > >
> > > > Thanks,
> > > > Shad Storhaug (NightOwl888)
> > > >
> > > > -----Original Message-----
> > > > From: Wyatt Barnett
> > > > [mailto:wyatt.barnett@gmail.com<mailto:wyatt.barnett@gmail.com>]
> > > > Sent: Friday, March 24, 2017 8:14 PM
> > > > To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
> > > > Subject: Re: API Work/Stabilization Update
> > > >
> > > > I'm about and happy to update when you are ready.
> > > >
> > > > I think the build should start working for 203 if you add the
> > > > nunit-console nuget package. At least work in the sense of the
> > > > build will start. I'm still chasing those build time bugbears I
> > > > haven't been able to slay yet.
> > > >
> > > > As for getting to nuget.org<http://nuget.org> -- who has the key?
> > > > I've never had access to it so I'm not sure we can update what is
> out there.
> > > >
> > > >
> > > > On Fri, Mar 24, 2017 at 09:10 Shad Storhaug
> > > > <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>
> > > wrote:
> > > >
> > > > > Wyatt, Prescott, Itamar, anybody? We've almost got all of the
> > > > > tests passing on #203 and would like to merge to master and
> > > > > release it with the new pre-release label "beta2".
> > > > >
> > > > > If there is nobody available to get the build running after
> > > > > merging, could someone give me access to TeamCity to work on it?
> > > > >
> > > > > Thanks,
> > > > > Shad Storhaug (NightOwl888)
> > > > >
> > > > > From: Shad Storhaug
> > > > > Sent: Tuesday, March 21, 2017 5:25 AM
> > > > > To: 'dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>'
> > > > > Subject: API Work/Stabilization Update
> > > > >
> > > > > I am getting very close to getting #203 merged. I wouldn't go so
> > > > > far as to say that the API is finished, but the most significant
> > > > > of the breaking API changes are now behind us.
> > > > >
> > > > > BUILD/VERSIONING
> > > > >
> > > > > I just wanted to be sure there is someone available to help get
> > > > > the build working after the merge. I think it would be
> > > > > appropriate to change the pre-release label from "beta" to
> > > > > "beta2" (without resetting the build number, since that is
> actually what NuGet uses).
> > > > > This would be primarily because of a major breaking API change,
> > > > > but also to indicate another advancement toward release.
> > > > >
> > > > > We should probably also get this onto NuGet as soon as possible
> > > > > to
> > > > > (hopefully) make it easier to recruit help to stabilize and
> > > > > create some integration packages for popular Microsoft frameworks.
> > > > >
> > > > > KNOWN ISSUES
> > > > >
> > > > >
> > > > > 1.       The QueryParser.Flexible custom localized message
> > > functionality
> > > > > is currently not implemented for .NET core, so those tests are
> > > > > now
> > > > failing.
> > > > >
> > > > > 2.       The implementation of Lucene.Net.Expressions currently
> reads
> > > > data
> > > > > from the configuration file. This is not how modern libraries
> > > > > are supposed to be built - instead we want any configuration to
> > > > > be pushed in from the application that uses Lucene.Net. Reading
> > > > > from the configuration file directly means no opportunity to use
> > > > > dependency injection. There is also a namespace
> > > > > Support/Configuration that can and should be removed after the
> > > > > implementation is refactored to be DI-friendly (see
> > > > > http://blog.ploeh.dk/2014/05/19/di-friendly-framework/). I
> > > > > haven't yet worked out how the implementation was done in .NET -
> > > > > in Java, the defaults were read from an embedded resource file
> > > > > and could be overridden by passing in a ClassLoader (similar to
> > > > > .NET's Assembly
> > > > class) - if anyone has any information on how the "auto generated"
> > > > C# code was generated, please share.
> > > > >
> > > > > 3.       The Collation functionality in Analysis.Common doesn't
> work
> > > with
> > > > > icu-dotnet, and has been excluded from compilation using the
> > > > > constant FEATURE_COLLATION. I am now convinced after reading the
> > > > > docs that it would be better to port the similar functionality
> > > > > from Analysis.ICU because it was designed to work with icu4j and
> > > > > is therefore more likely to work with icu-dotnet.
> > > > >
> > > > > 4.       The Highlighter PostingsHighlight and VectorHighlight
> > > > > functionality relies on icu-dotnet, which doesn't have a close
> > > > > match for the BreakIterator in the JRE, so there are likely some
> > > > > big differences in the functionality. Several hacks were put in
> > > > > to make the tests pass, but these are not likely to fix all of
> > > > > the issues in the
> > > > wild.
> > > > >
> > > > > 5.       There are several namespaces in Lucene.Net.Core and
> > > > > Lucene.Net.Codecs that have broken documentation comments.
> > > > >
> > > > > 6.       There are some concurrency and performance issues (as
> > pointed
> > > > out
> > > > > by Vincent Van Den Burghe):
> > > > > http://git.net/ml/general/2017-02/msg00168.html
> > > > >
> > > > > 7.       We have around 2 dozen tests that fail during
> randomization
> > > > > (averaging about 17 broken per run), and 8 tests that fail
> > > > > all/most of the time.
> > > > >
> > > > > RESOLVED ISSUES (in addition to API refactoring)
> > > > >
> > > > >
> > > > > 1.       Finished implementing the randomization of Codecs,
> Culture,
> > > Time
> > > > > Zone, and InfoStream in the TestFramework.
> > > > >
> > > > > 2.       Added factories for Codec, DocValuesFormat, and
> > PostingsFormat
> > > > so
> > > > > custom implementations can be provided via dependency injection
> > > > > instead of using the Java-ish NamedSPILoader class. The name
> > > > > must now be provided by an attribute (or by class naming
> > > > > convention) rather than via constructor, so it can be read
> > > > > without creating an instance of
> > > > the class.
> > > > >
> > > > > 3.       Fixed several of the codecs in Lucene.Net.Codecs that were
> > > still
> > > > > not functioning (and not being tested because of the unfinished
> > > > > RandomCodec class and test mocks).
> > > > >
> > > > > 4.       Reviewed all catch blocks in Lucene.Net.Core to ensure the
> > > right
> > > > > type of exceptions are being caught and the right type re-thrown.
> > > > >
> > > > > 5.       Fixed culture-sensitive comparison and sort order issues
> > when
> > > > > using strings in Lucene.Net.Core and Lucene.Net.Codecs.
> > > > >
> > > > > 6.       Merged similar functionality in Support into the same
> class
> > > and
> > > > > deleted several unused Support classes.
> > > > >
> > > > > 7.       Made the API CLS compliant, so it now works with all .NET
> > > > > languages.
> > > > >
> > > > >
> > > > > Shad Storhaug (NightOwl888)
> > > > >
> > > >
> > >
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message