lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shad Storhaug <s...@shadstorhaug.com>
Subject RE: Update on dev happenings
Date Thu, 08 Aug 2019 04:37:13 GMT
Update.

I really didn't want to give up TeamCity because it is far better than Azure DevOps, but it
seems that between the fact that JetBrains won't give us any more than one build agent, the
fact it doesn't seem to be possible to get any Agents on Linux or Mac (at least none that
will run powershell), and the fact that Microsoft broke the ability to get test results into
TeamCity (I tried half a dozen configurations, but due to the other issues it didn't seem
worth the effort) that Azure DevOps is a better choice. We definitely should switch to using
Azure Artifacts regardless because of the generous amount of feed storage they allow (which
frankly was why we didn't target .NET Standard 2.0 on all of the projects on the last release).

We now have a YAML Azure DevOps pipeline in the repository. The test projects for Lucene.Net
(core) were split into smaller pieces in order to run the tests in parallel in background
jobs, and I was able to get the testing time down to about 40 minutes, which is a pretty safe
margin less than the 1 hour limit Azure DevOps imposes on how long a job can run. I set it
up so basically anyone can create a free Azure DevOps account, import the template, and it
will automatically build, version a CI build based on a counter, and test all target frameworks
on Windows, macOS and Ubuntu. Of course, it will build much faster if you setup the project
as a public project so you can get 10 parallel jobs. If you enter an Azure Artifact feed ID
(a GUID) as a variable, it will also push the packages and debug symbols into the feed (see
the documentation in azure-pipelines.yml https://github.com/apache/lucenenet/blob/master/azure-pipelines.yml#L22).

However, I have run into a couple of obstacles with actually getting this to work with the
official repo. It seems that there are INFRA issues blocking it from working as a CI integration
because Microsoft requires write repo access and Apache has a strict policy against it: https://issues.apache.org/jira/browse/INFRA-17030

Furthermore, Microsoft doesn't support YAML in any "Other Git (generic)" repositories: https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/?view=azure-devops.

Despite this, I was able to use the GitHub mirror of our repo by first setting it up on a
fork, and then switching the GitHub repo name to apache/lucenenet, running in read-only mode.
One problem solved.

I have setup what I hope will be the official Lucene.NET organization here: https://dev.azure.com/lucene-net/Lucene.NET/_build/results?buildId=1&view=ms.vss-test-web.build-test-results-tab,
but if we need to change it for any reason, nearly all of the configuration is in YAML so
it isn't a lot of work.

This configuration allows us to build a release because that is launched manually, but CI
builds are another story. Azure DevOps doesn't get triggered by a push to the Apache repo,
and they don't seem to have an option to poll for changes like we do in TeamCity. A couple
of options I have been contemplating:

1. Setup a build in TeamCity to do an API call to Azure DevOps to launch the build: https://docs.microsoft.com/en-us/rest/api/azure/devops/build/?view=azure-devops-rest-5.0#queue-a-build
2. Setup an email address and register it with commits@lucenenet.apache.org and trigger the
build whenever an email is received

However, before I hack together a solution, I wanted to check whether any of the PMC know
if it is possible to have INFRA setup a commit hook so we can do this in a more straightforward
way? Possibly by setting up an Azure Function and having Apache call it via a URL: https://github.com/MicrosoftDocs/vsts-docs/issues/2366?
I would ask them, but the docs say to ask here first...


Thanks,
Shad Storhaug (NightOwl888)


-----Original Message-----
From: Laimonas Simutis <laimis@gmail.com> 
Sent: Tuesday, March 12, 2019 8:12 PM
To: dev@lucenenet.apache.org
Subject: Re: Update on dev happenings

Quick update. I've had a successful build with azure pipelines, you can see it here: https://dev.azure.com/lsimutis/lsimutis/_build/results?buildId=5

This is off of the fork of the current repo. Will see how the tests do there next and then
perhaps we can seriously eval moving our builds off of teamcity on jetbrains. Or at the very
least get the PRs building and verified automatically while we still decide if resurrecting
teamcity setup is worth persuing.


Laimonas

On Tue, Feb 26, 2019 at 8:35 AM Laimonas Simutis <laimis@gmail.com> wrote:

> I wonder if the assumption that .net core needs to be installed is 
> outdated and we can find an agent with the .net core present.
> psake abstraction here is hard to follow and reason what's going on... 
> I will check if there are other .net core based projects that are 
> being built in there that we can try to follow.
>
> Another option is for us to migrate away from TeamCity and use 
> something else. This could be an angle worth pursuing:
> https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with
> -unlimited-ci-cd-minutes-for-open-source
> .
>
> In addition, someone mentioned other alternatives for open source 
> projects, maybe whoever mentioned could chime in again.
>
> At this point having a build/test cycle going for master + all the PRs 
> is the next big need to get this train moving.
>
>
> Laimonas
>
> On Sun, Feb 17, 2019 at 12:25 PM Simon Svensson <sisve@devhost.se> wrote:
>
>> Hi,
>>
>> I'm back after a week of districtions, with some further thoughts 
>> about TeamCity.
>>
>> I don't think we can trust the agents to ever have a known state, or 
>> even share a common state. The obvious(?) answer is docker. Is docker 
>> installed on the agent, and are we allowed to pull down docker images 
>> on the agents? Can we use windows containers? (I believe it defaults 
>> to linux containers.)
>>
>> I believe that my permission level doesn't allow me to change the 
>> existing build configuration, but I really hope there's a Docker 
>> build step...
>>
>> The following docker run command will build the project. The %CD% 
>> should expand to the lucenenet project root. I'm seeing some weird 
>> timings, it takes minutes to restore packages for a single project, 
>> but I'll be open and say that it could be something I've done to 
>> myself in my experimenting with weird stuff. (And Windows started 
>> informing me about low disk space when writing that sentence...)
>>
>> docker run ^
>>  --rm ^
>>  --env DOTNET_CLI_TELEMETRY_OPTOUT=1 ^  --volume %CD%:c:\build ^  
>> --workdir c:\build ^
>>  mcr.microsoft.com/windows/servercore:1809 ^  POWERSHELL 
>> "Import-Module .\build\psake.psm1; Invoke-psake
>> .\build\build.ps1 -properties @{backup_files='false'}"
>>
>> Are there any support forums or similar for the hosted teamcity for 
>> opensource projects? Or do we have any secret contacts that has all 
>> the answers?
>>
>> // Simon Svensson
>>
>> On 2019-02-10 22:49, Simon Svensson wrote:
>> > Hi,
>> >
>> > So, I messed something up and managed to build the master branch 
>> > instead of my sisve-debugging-build branch... and it worked. I 
>> > havn't figured out why. Probably server gnomes.
>> >
>> > It seems there are several agents/runners; and they aren't rolled 
>> > back to a clean state between builds. I triggered the build on the 
>> > real branch, and it got stuck on low disk space. After a few 
>> > minutes I forcefully stopped it. It's still attempting to clean 
>> > when I am writing this. (I wonder if I just messed up an agent...)
>> >
>> > The second attempt ran into the problem with non-working dotnet again.
>> >
>> > This is the build log from my accidentally working build. Note that 
>> > there is already an old sdk installed, and a total of four sdks 
>> > installed after dotnet-install has executed.
>> >
>> https://teamcity.jetbrains.com/viewLog.html?buildId=1960125&tab=build
>> Log&_focus=1200
>> >
>> > It scares me slightly that this agent, the one that succeeded with 
>> > the build, had builds since December 2016 in the build history...
>> >
>> https://teamcity.jetbrains.com/agentDetails.html?id=170591&tab=agentH
>> istory&page=31
>> >
>> > I will continue reading up on how TeamCity works, and the agents, 
>> > and try to figure out why we had an agent with lots of sdks and 
>> > .net runtimes installed.
>> >
>> > // Simon Svensson
>> >
>> > On 2019-02-09 19:41, Laimonas Simutis wrote:
>> >> Simon,
>> >>
>> >> You should have access now to the project configuration. Let us 
>> >> know
>> how it
>> >> goes. From the logs it looks like the project attempts to install 
>> >> .net
>> cli
>> >> and fails and restore can't run, nor the build steps.
>> >>
>> >> Good luck. I will ping here if I have any breakthroughs myself.
>> >>
>> >>
>> >> Laimonas
>> >>
>> >>
>> >> On Sat, Feb 9, 2019 at 3:52 AM Simon Svensson <sisve@devhost.se>
>> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> My JetBrains login would be the secretive "sisve@devhost.se".
>> >>>
>> >>> // Simon Svensson
>> >>>
>> >>> On 2019-02-08 17:26, Laimonas Simutis wrote:
>> >>>> Simon,
>> >>>>
>> >>>> Oh my, I jumped into investigating the next builds in the chain
>> assuming
>> >>>> that the green means all is well, but you are right. That did 
>> >>>> not
>> work.
>> >>> Let
>> >>>> me take a look at that and see what fails in base.
>> >>>>
>> >>>> What's your jetbrains login/email? I should be able to add you 
>> >>>> to the administrative portion of the build
>> >>>>
>> >>>>
>> >>>> Laimonas
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson <sisve@devhost.se>
>> wrote:
>> >>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> The main build has a green checkbox and the word "Success" next

>> >>>>> to
>> it. I
>> >>>>> would classify this as slightly inaccurate, the build totally
>> failed.
>> >>>>> The build script logged an error, but exited with a exit code

>> >>>>> of 0
>> so no
>> >>>>> error was signaled.
>> >>>>>
>> >>>>> In short, "dotnet.exe restore" failed for unknown reasons, and

>> >>>>> this continues in all builds since 1190. There are silent 
>> >>>>> failures
>> before the
>> >>>>> restore; where the build scripts should output some information
>> about
>> >>>>> build numbers, directory paths and such, it should also output

>> >>>>> "dotnet.exe --version" and "dotnet.exe --info". These worked
in
>> build
>> >>>>> 1189, but failed in 1190.
>> >>>>>
>> >>>>> A perhaps related observation is that we, starting with build

>> >>>>> 1190,
>> also
>> >>>>> need to install .NET SDK 2.0.0 every run. It could be relevant

>> >>>>> that
>> it's
>> >>>>> over 6 months between build 1189 and 1190, so there could 
>> >>>>> perhaps
>> be a
>> >>>>> change in TeamCity we need to track down.
>> >>>>>
>> >>>>> Is there an administrative ui in TeamCity that I can get access
to?
>> I
>> >>>>> believe the next step would be to debug (read: qualified 
>> >>>>> guessing)
>> the
>> >>>>> build script on a separate branch; would this be picked up
>> automatically
>> >>>>> by TeamCity?
>> >>>>>
>> >>>>> // Simon Svensson
>> >>>>>
>> >>>>> On 2019-02-06 13:57, Laimonas Simutis wrote:
>> >>>>>> Simon,
>> >>>>>>
>> >>>>>> I was able to get the base builds going but as you already
>> noticed, the
>> >>>>>> subsequent builds are failing outright. The build structure

>> >>>>>> has
>> changed
>> >>>>>> quite a bit since the last time I as in there, so it's taking

>> >>>>>> me
>> some
>> >>>>> time
>> >>>>>> to understand what's going on. I will continue to try to

>> >>>>>> untangle
>> it or
>> >>>>> set
>> >>>>>> up a new build flow in there if that fails.
>> >>>>>>
>> >>>>>> Chris, here is the project link:
>> >>>>>>
>> >>>>>
>> >>>
>> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=p
>> rojectOverview
>> >>>>>> if you want to take a look, perhaps you will see right away

>> >>>>>> what's
>> up.
>> >>>>>>
>> >>>>>> Laimonas
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini 
>> >>>>>> <chris@brass9.com>
>> >>> wrote:
>> >>>>>>
>> >>>>>>> We use Lucene.Net to do conventional search at:
>> >>>>>>>
>> >>>>>>> https://conservationx.com
>> >>>>>>>
>> >>>>>>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch,
>> Levenshtein,
>> >>> etc
>> >>>>>>>
>> >>>>>>> There was a problem mentioned about TeamCity; is there
a way 
>> >>>>>>> I can
>> >>> learn
>> >>>>>>> more in case I'd possibly be able to get it working?
Where is 
>> >>>>>>> it
>> >>> checked
>> >>>>>>> in, where is it running, etc
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>
Mime
View raw message