lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shad Storhaug <s...@shadstorhaug.com>
Subject RE: Update on dev happenings
Date Thu, 08 Aug 2019 23:03:06 GMT
Patric,

Thanks for the suggestion. I am not sure how relevant it is though because our GitHub presence
is just a mirror of the repo that is hosted at Apache. When we push commits, we push them
directly to Apache so GitHub really has no way to hook into the process unless it is somehow
based on watching for commit changes by polling.

In fact, that is why Azure DevOps doesn't "just work" - they aren't watching for commit changes
to the mirror, they are expecting the commits through the front door that is strictly read-only
by Apache. I haven't tried, but I suspect that the PR functionality of Azure DevOps will work
because those are submitted directly through GitHub. It would be a bit of a pain for every
committer to have to open a PR first before merging their own commits in order to get any
CI results, though.

I don't even have enough access to the apache/lucenenet GitHub account to merge a pull request
and I am not sure whether I can get it because I haven't asked. I have just been working around
it by merging PRs with Git and closing them via commit message.

It seems that the path of least resistance here is to use our existing setup on TeamCity to
fire an API call to Azure DevOps. TeamCity has its own built in key vault so whatever credentials
are required to call the Azure DevOps API can be stored there, and whatever task that is necessary
can be setup to make the API call. Its quite slow, though - the poll only happens every 20
minutes and then there could be another 5-10 minute wait to get a build agent. The polling
interval can be adjusted, but not sure what the minimum would be that Apache's server could
handle. Slow is still better than never or manual, though.

Thanks,
Shad Storhaug (NightOwl888)

-----Original Message-----
From: Patric Forsgard <patric@tasteful.se> 
Sent: Friday, August 9, 2019 3:26 AM
To: dev@lucenenet.apache.org
Subject: Re: Update on dev happenings

I was notified today about Github Actions (
https://developer.github.com/actions/) that also can be executed on developer computer with
ACT ( https://github.com/nektos/act/blob/master/README.md). Can this be someting to use instead?
Yes, it will be a dependency on docker in that case to be able to run everyting.

// Patric

On Thu, Aug 8, 2019, 06:37 Shad Storhaug <shad@shadstorhaug.com> wrote:

> Update.
>
> I really didn't want to give up TeamCity because it is far better than 
> Azure DevOps, but it seems that between the fact that JetBrains won't 
> give us any more than one build agent, the fact it doesn't seem to be 
> possible to get any Agents on Linux or Mac (at least none that will 
> run powershell), and the fact that Microsoft broke the ability to get 
> test results into TeamCity (I tried half a dozen configurations, but 
> due to the other issues it didn't seem worth the effort) that Azure 
> DevOps is a better choice. We definitely should switch to using Azure 
> Artifacts regardless because of the generous amount of feed storage 
> they allow (which frankly was why we didn't target .NET Standard 2.0 on all of the projects
on the last release).
>
> We now have a YAML Azure DevOps pipeline in the repository. The test 
> projects for Lucene.Net (core) were split into smaller pieces in order 
> to run the tests in parallel in background jobs, and I was able to get 
> the testing time down to about 40 minutes, which is a pretty safe 
> margin less than the 1 hour limit Azure DevOps imposes on how long a 
> job can run. I set it up so basically anyone can create a free Azure 
> DevOps account, import the template, and it will automatically build, 
> version a CI build based on a counter, and test all target frameworks 
> on Windows, macOS and Ubuntu. Of course, it will build much faster if 
> you setup the project as a public project so you can get 10 parallel 
> jobs. If you enter an Azure Artifact feed ID (a GUID) as a variable, 
> it will also push the packages and debug symbols into the feed (see 
> the documentation in azure-pipelines.yml https://github.com/apache/lucenenet/blob/master/azure-pipelines.yml#L22).
>
> However, I have run into a couple of obstacles with actually getting 
> this to work with the official repo. It seems that there are INFRA 
> issues blocking it from working as a CI integration because Microsoft 
> requires write repo access and Apache has a strict policy against it:
> https://issues.apache.org/jira/browse/INFRA-17030
>
> Furthermore, Microsoft doesn't support YAML in any "Other Git (generic)"
> repositories:
> https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/?view=az
> ure-devops
> .
>
> Despite this, I was able to use the GitHub mirror of our repo by first 
> setting it up on a fork, and then switching the GitHub repo name to 
> apache/lucenenet, running in read-only mode. One problem solved.
>
> I have setup what I hope will be the official Lucene.NET organization
> here:
> https://dev.azure.com/lucene-net/Lucene.NET/_build/results?buildId=1&v
> iew=ms.vss-test-web.build-test-results-tab,
> but if we need to change it for any reason, nearly all of the 
> configuration is in YAML so it isn't a lot of work.
>
> This configuration allows us to build a release because that is 
> launched manually, but CI builds are another story. Azure DevOps 
> doesn't get triggered by a push to the Apache repo, and they don't 
> seem to have an option to poll for changes like we do in TeamCity. A 
> couple of options I have been contemplating:
>
> 1. Setup a build in TeamCity to do an API call to Azure DevOps to 
> launch the build:
> https://docs.microsoft.com/en-us/rest/api/azure/devops/build/?view=azu
> re-devops-rest-5.0#queue-a-build 2. Setup an email address and 
> register it with commits@lucenenet.apache.org and trigger the build 
> whenever an email is received
>
> However, before I hack together a solution, I wanted to check whether 
> any of the PMC know if it is possible to have INFRA setup a commit 
> hook so we can do this in a more straightforward way? Possibly by 
> setting up an Azure Function and having Apache call it via a URL:
> https://github.com/MicrosoftDocs/vsts-docs/issues/2366? I would ask 
> them, but the docs say to ask here first...
>
>
> Thanks,
> Shad Storhaug (NightOwl888)
>
>
> -----Original Message-----
> From: Laimonas Simutis <laimis@gmail.com>
> Sent: Tuesday, March 12, 2019 8:12 PM
> To: dev@lucenenet.apache.org
> Subject: Re: Update on dev happenings
>
> Quick update. I've had a successful build with azure pipelines, you 
> can see it here:
> https://dev.azure.com/lsimutis/lsimutis/_build/results?buildId=5
>
> This is off of the fork of the current repo. Will see how the tests do 
> there next and then perhaps we can seriously eval moving our builds 
> off of teamcity on jetbrains. Or at the very least get the PRs 
> building and verified automatically while we still decide if 
> resurrecting teamcity setup is worth persuing.
>
>
> Laimonas
>
> On Tue, Feb 26, 2019 at 8:35 AM Laimonas Simutis <laimis@gmail.com> wrote:
>
> > I wonder if the assumption that .net core needs to be installed is 
> > outdated and we can find an agent with the .net core present.
> > psake abstraction here is hard to follow and reason what's going on...
> > I will check if there are other .net core based projects that are 
> > being built in there that we can try to follow.
> >
> > Another option is for us to migrate away from TeamCity and use 
> > something else. This could be an angle worth pursuing:
> > https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-wi
> > th -unlimited-ci-cd-minutes-for-open-source
> > .
> >
> > In addition, someone mentioned other alternatives for open source 
> > projects, maybe whoever mentioned could chime in again.
> >
> > At this point having a build/test cycle going for master + all the 
> > PRs is the next big need to get this train moving.
> >
> >
> > Laimonas
> >
> > On Sun, Feb 17, 2019 at 12:25 PM Simon Svensson <sisve@devhost.se>
> wrote:
> >
> >> Hi,
> >>
> >> I'm back after a week of districtions, with some further thoughts 
> >> about TeamCity.
> >>
> >> I don't think we can trust the agents to ever have a known state, 
> >> or even share a common state. The obvious(?) answer is docker. Is 
> >> docker installed on the agent, and are we allowed to pull down 
> >> docker images on the agents? Can we use windows containers? (I 
> >> believe it defaults to linux containers.)
> >>
> >> I believe that my permission level doesn't allow me to change the 
> >> existing build configuration, but I really hope there's a Docker 
> >> build step...
> >>
> >> The following docker run command will build the project. The %CD% 
> >> should expand to the lucenenet project root. I'm seeing some weird 
> >> timings, it takes minutes to restore packages for a single project, 
> >> but I'll be open and say that it could be something I've done to 
> >> myself in my experimenting with weird stuff. (And Windows started 
> >> informing me about low disk space when writing that sentence...)
> >>
> >> docker run ^
> >>  --rm ^
> >>  --env DOTNET_CLI_TELEMETRY_OPTOUT=1 ^  --volume %CD%:c:\build ^ 
> >> --workdir c:\build ^
> >>  mcr.microsoft.com/windows/servercore:1809 ^  POWERSHELL 
> >> "Import-Module .\build\psake.psm1; Invoke-psake
> >> .\build\build.ps1 -properties @{backup_files='false'}"
> >>
> >> Are there any support forums or similar for the hosted teamcity for 
> >> opensource projects? Or do we have any secret contacts that has all 
> >> the answers?
> >>
> >> // Simon Svensson
> >>
> >> On 2019-02-10 22:49, Simon Svensson wrote:
> >> > Hi,
> >> >
> >> > So, I messed something up and managed to build the master branch 
> >> > instead of my sisve-debugging-build branch... and it worked. I 
> >> > havn't figured out why. Probably server gnomes.
> >> >
> >> > It seems there are several agents/runners; and they aren't rolled 
> >> > back to a clean state between builds. I triggered the build on 
> >> > the real branch, and it got stuck on low disk space. After a few 
> >> > minutes I forcefully stopped it. It's still attempting to clean 
> >> > when I am writing this. (I wonder if I just messed up an 
> >> > agent...)
> >> >
> >> > The second attempt ran into the problem with non-working dotnet again.
> >> >
> >> > This is the build log from my accidentally working build. Note 
> >> > that there is already an old sdk installed, and a total of four 
> >> > sdks installed after dotnet-install has executed.
> >> >
> >> https://teamcity.jetbrains.com/viewLog.html?buildId=1960125&tab=bui
> >> ld
> >> Log&_focus=1200
> >> >
> >> > It scares me slightly that this agent, the one that succeeded 
> >> > with the build, had builds since December 2016 in the build history...
> >> >
> >> https://teamcity.jetbrains.com/agentDetails.html?id=170591&tab=agen
> >> tH
> >> istory&page=31
> >> >
> >> > I will continue reading up on how TeamCity works, and the agents, 
> >> > and try to figure out why we had an agent with lots of sdks and 
> >> > .net runtimes installed.
> >> >
> >> > // Simon Svensson
> >> >
> >> > On 2019-02-09 19:41, Laimonas Simutis wrote:
> >> >> Simon,
> >> >>
> >> >> You should have access now to the project configuration. Let us 
> >> >> know
> >> how it
> >> >> goes. From the logs it looks like the project attempts to 
> >> >> install .net
> >> cli
> >> >> and fails and restore can't run, nor the build steps.
> >> >>
> >> >> Good luck. I will ping here if I have any breakthroughs myself.
> >> >>
> >> >>
> >> >> Laimonas
> >> >>
> >> >>
> >> >> On Sat, Feb 9, 2019 at 3:52 AM Simon Svensson <sisve@devhost.se>
> >> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> My JetBrains login would be the secretive "sisve@devhost.se".
> >> >>>
> >> >>> // Simon Svensson
> >> >>>
> >> >>> On 2019-02-08 17:26, Laimonas Simutis wrote:
> >> >>>> Simon,
> >> >>>>
> >> >>>> Oh my, I jumped into investigating the next builds in the 
> >> >>>> chain
> >> assuming
> >> >>>> that the green means all is well, but you are right. That did

> >> >>>> not
> >> work.
> >> >>> Let
> >> >>>> me take a look at that and see what fails in base.
> >> >>>>
> >> >>>> What's your jetbrains login/email? I should be able to add
you 
> >> >>>> to the administrative portion of the build
> >> >>>>
> >> >>>>
> >> >>>> Laimonas
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson 
> >> >>>> <sisve@devhost.se>
> >> wrote:
> >> >>>>
> >> >>>>> Hi,
> >> >>>>>
> >> >>>>> The main build has a green checkbox and the word "Success"

> >> >>>>> next to
> >> it. I
> >> >>>>> would classify this as slightly inaccurate, the build totally
> >> failed.
> >> >>>>> The build script logged an error, but exited with a exit
code 
> >> >>>>> of 0
> >> so no
> >> >>>>> error was signaled.
> >> >>>>>
> >> >>>>> In short, "dotnet.exe restore" failed for unknown reasons,

> >> >>>>> and this continues in all builds since 1190. There are
silent 
> >> >>>>> failures
> >> before the
> >> >>>>> restore; where the build scripts should output some 
> >> >>>>> information
> >> about
> >> >>>>> build numbers, directory paths and such, it should also

> >> >>>>> output "dotnet.exe --version" and "dotnet.exe --info".
These 
> >> >>>>> worked in
> >> build
> >> >>>>> 1189, but failed in 1190.
> >> >>>>>
> >> >>>>> A perhaps related observation is that we, starting with
build 
> >> >>>>> 1190,
> >> also
> >> >>>>> need to install .NET SDK 2.0.0 every run. It could be 
> >> >>>>> relevant that
> >> it's
> >> >>>>> over 6 months between build 1189 and 1190, so there could

> >> >>>>> perhaps
> >> be a
> >> >>>>> change in TeamCity we need to track down.
> >> >>>>>
> >> >>>>> Is there an administrative ui in TeamCity that I can get

> >> >>>>> access
> to?
> >> I
> >> >>>>> believe the next step would be to debug (read: qualified
> >> >>>>> guessing)
> >> the
> >> >>>>> build script on a separate branch; would this be picked
up
> >> automatically
> >> >>>>> by TeamCity?
> >> >>>>>
> >> >>>>> // Simon Svensson
> >> >>>>>
> >> >>>>> On 2019-02-06 13:57, Laimonas Simutis wrote:
> >> >>>>>> Simon,
> >> >>>>>>
> >> >>>>>> I was able to get the base builds going but as you
already
> >> noticed, the
> >> >>>>>> subsequent builds are failing outright. The build structure

> >> >>>>>> has
> >> changed
> >> >>>>>> quite a bit since the last time I as in there, so it's

> >> >>>>>> taking me
> >> some
> >> >>>>> time
> >> >>>>>> to understand what's going on. I will continue to try
to 
> >> >>>>>> untangle
> >> it or
> >> >>>>> set
> >> >>>>>> up a new build flow in there if that fails.
> >> >>>>>>
> >> >>>>>> Chris, here is the project link:
> >> >>>>>>
> >> >>>>>
> >> >>>
> >> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab
> >> =p
> >> rojectOverview
> >> >>>>>> if you want to take a look, perhaps you will see right
away 
> >> >>>>>> what's
> >> up.
> >> >>>>>>
> >> >>>>>> Laimonas
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini 
> >> >>>>>> <chris@brass9.com>
> >> >>> wrote:
> >> >>>>>>
> >> >>>>>>> We use Lucene.Net to do conventional search at:
> >> >>>>>>>
> >> >>>>>>> https://conservationx.com
> >> >>>>>>>
> >> >>>>>>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch,
> >> Levenshtein,
> >> >>> etc
> >> >>>>>>>
> >> >>>>>>> There was a problem mentioned about TeamCity; is
there a 
> >> >>>>>>> way I can
> >> >>> learn
> >> >>>>>>> more in case I'd possibly be able to get it working?
Where 
> >> >>>>>>> is it
> >> >>> checked
> >> >>>>>>> in, where is it running, etc
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >
>
Mime
View raw message