lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patric Forsgard <pat...@tasteful.se>
Subject Re: Update on dev happenings
Date Thu, 08 Aug 2019 20:26:11 GMT
I was notified today about Github Actions (
https://developer.github.com/actions/) that also can be executed on
developer computer with ACT (
https://github.com/nektos/act/blob/master/README.md). Can this be someting
to use instead? Yes, it will be a dependency on docker in that case to be
able to run everyting.

// Patric

On Thu, Aug 8, 2019, 06:37 Shad Storhaug <shad@shadstorhaug.com> wrote:

> Update.
>
> I really didn't want to give up TeamCity because it is far better than
> Azure DevOps, but it seems that between the fact that JetBrains won't give
> us any more than one build agent, the fact it doesn't seem to be possible
> to get any Agents on Linux or Mac (at least none that will run powershell),
> and the fact that Microsoft broke the ability to get test results into
> TeamCity (I tried half a dozen configurations, but due to the other issues
> it didn't seem worth the effort) that Azure DevOps is a better choice. We
> definitely should switch to using Azure Artifacts regardless because of the
> generous amount of feed storage they allow (which frankly was why we didn't
> target .NET Standard 2.0 on all of the projects on the last release).
>
> We now have a YAML Azure DevOps pipeline in the repository. The test
> projects for Lucene.Net (core) were split into smaller pieces in order to
> run the tests in parallel in background jobs, and I was able to get the
> testing time down to about 40 minutes, which is a pretty safe margin less
> than the 1 hour limit Azure DevOps imposes on how long a job can run. I set
> it up so basically anyone can create a free Azure DevOps account, import
> the template, and it will automatically build, version a CI build based on
> a counter, and test all target frameworks on Windows, macOS and Ubuntu. Of
> course, it will build much faster if you setup the project as a public
> project so you can get 10 parallel jobs. If you enter an Azure Artifact
> feed ID (a GUID) as a variable, it will also push the packages and debug
> symbols into the feed (see the documentation in azure-pipelines.yml
> https://github.com/apache/lucenenet/blob/master/azure-pipelines.yml#L22).
>
> However, I have run into a couple of obstacles with actually getting this
> to work with the official repo. It seems that there are INFRA issues
> blocking it from working as a CI integration because Microsoft requires
> write repo access and Apache has a strict policy against it:
> https://issues.apache.org/jira/browse/INFRA-17030
>
> Furthermore, Microsoft doesn't support YAML in any "Other Git (generic)"
> repositories:
> https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/?view=azure-devops
> .
>
> Despite this, I was able to use the GitHub mirror of our repo by first
> setting it up on a fork, and then switching the GitHub repo name to
> apache/lucenenet, running in read-only mode. One problem solved.
>
> I have setup what I hope will be the official Lucene.NET organization
> here:
> https://dev.azure.com/lucene-net/Lucene.NET/_build/results?buildId=1&view=ms.vss-test-web.build-test-results-tab,
> but if we need to change it for any reason, nearly all of the configuration
> is in YAML so it isn't a lot of work.
>
> This configuration allows us to build a release because that is launched
> manually, but CI builds are another story. Azure DevOps doesn't get
> triggered by a push to the Apache repo, and they don't seem to have an
> option to poll for changes like we do in TeamCity. A couple of options I
> have been contemplating:
>
> 1. Setup a build in TeamCity to do an API call to Azure DevOps to launch
> the build:
> https://docs.microsoft.com/en-us/rest/api/azure/devops/build/?view=azure-devops-rest-5.0#queue-a-build
> 2. Setup an email address and register it with
> commits@lucenenet.apache.org and trigger the build whenever an email is
> received
>
> However, before I hack together a solution, I wanted to check whether any
> of the PMC know if it is possible to have INFRA setup a commit hook so we
> can do this in a more straightforward way? Possibly by setting up an Azure
> Function and having Apache call it via a URL:
> https://github.com/MicrosoftDocs/vsts-docs/issues/2366? I would ask them,
> but the docs say to ask here first...
>
>
> Thanks,
> Shad Storhaug (NightOwl888)
>
>
> -----Original Message-----
> From: Laimonas Simutis <laimis@gmail.com>
> Sent: Tuesday, March 12, 2019 8:12 PM
> To: dev@lucenenet.apache.org
> Subject: Re: Update on dev happenings
>
> Quick update. I've had a successful build with azure pipelines, you can
> see it here:
> https://dev.azure.com/lsimutis/lsimutis/_build/results?buildId=5
>
> This is off of the fork of the current repo. Will see how the tests do
> there next and then perhaps we can seriously eval moving our builds off of
> teamcity on jetbrains. Or at the very least get the PRs building and
> verified automatically while we still decide if resurrecting teamcity setup
> is worth persuing.
>
>
> Laimonas
>
> On Tue, Feb 26, 2019 at 8:35 AM Laimonas Simutis <laimis@gmail.com> wrote:
>
> > I wonder if the assumption that .net core needs to be installed is
> > outdated and we can find an agent with the .net core present.
> > psake abstraction here is hard to follow and reason what's going on...
> > I will check if there are other .net core based projects that are
> > being built in there that we can try to follow.
> >
> > Another option is for us to migrate away from TeamCity and use
> > something else. This could be an angle worth pursuing:
> > https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with
> > -unlimited-ci-cd-minutes-for-open-source
> > .
> >
> > In addition, someone mentioned other alternatives for open source
> > projects, maybe whoever mentioned could chime in again.
> >
> > At this point having a build/test cycle going for master + all the PRs
> > is the next big need to get this train moving.
> >
> >
> > Laimonas
> >
> > On Sun, Feb 17, 2019 at 12:25 PM Simon Svensson <sisve@devhost.se>
> wrote:
> >
> >> Hi,
> >>
> >> I'm back after a week of districtions, with some further thoughts
> >> about TeamCity.
> >>
> >> I don't think we can trust the agents to ever have a known state, or
> >> even share a common state. The obvious(?) answer is docker. Is docker
> >> installed on the agent, and are we allowed to pull down docker images
> >> on the agents? Can we use windows containers? (I believe it defaults
> >> to linux containers.)
> >>
> >> I believe that my permission level doesn't allow me to change the
> >> existing build configuration, but I really hope there's a Docker
> >> build step...
> >>
> >> The following docker run command will build the project. The %CD%
> >> should expand to the lucenenet project root. I'm seeing some weird
> >> timings, it takes minutes to restore packages for a single project,
> >> but I'll be open and say that it could be something I've done to
> >> myself in my experimenting with weird stuff. (And Windows started
> >> informing me about low disk space when writing that sentence...)
> >>
> >> docker run ^
> >>  --rm ^
> >>  --env DOTNET_CLI_TELEMETRY_OPTOUT=1 ^  --volume %CD%:c:\build ^
> >> --workdir c:\build ^
> >>  mcr.microsoft.com/windows/servercore:1809 ^  POWERSHELL
> >> "Import-Module .\build\psake.psm1; Invoke-psake
> >> .\build\build.ps1 -properties @{backup_files='false'}"
> >>
> >> Are there any support forums or similar for the hosted teamcity for
> >> opensource projects? Or do we have any secret contacts that has all
> >> the answers?
> >>
> >> // Simon Svensson
> >>
> >> On 2019-02-10 22:49, Simon Svensson wrote:
> >> > Hi,
> >> >
> >> > So, I messed something up and managed to build the master branch
> >> > instead of my sisve-debugging-build branch... and it worked. I
> >> > havn't figured out why. Probably server gnomes.
> >> >
> >> > It seems there are several agents/runners; and they aren't rolled
> >> > back to a clean state between builds. I triggered the build on the
> >> > real branch, and it got stuck on low disk space. After a few
> >> > minutes I forcefully stopped it. It's still attempting to clean
> >> > when I am writing this. (I wonder if I just messed up an agent...)
> >> >
> >> > The second attempt ran into the problem with non-working dotnet again.
> >> >
> >> > This is the build log from my accidentally working build. Note that
> >> > there is already an old sdk installed, and a total of four sdks
> >> > installed after dotnet-install has executed.
> >> >
> >> https://teamcity.jetbrains.com/viewLog.html?buildId=1960125&tab=build
> >> Log&_focus=1200
> >> >
> >> > It scares me slightly that this agent, the one that succeeded with
> >> > the build, had builds since December 2016 in the build history...
> >> >
> >> https://teamcity.jetbrains.com/agentDetails.html?id=170591&tab=agentH
> >> istory&page=31
> >> >
> >> > I will continue reading up on how TeamCity works, and the agents,
> >> > and try to figure out why we had an agent with lots of sdks and
> >> > .net runtimes installed.
> >> >
> >> > // Simon Svensson
> >> >
> >> > On 2019-02-09 19:41, Laimonas Simutis wrote:
> >> >> Simon,
> >> >>
> >> >> You should have access now to the project configuration. Let us
> >> >> know
> >> how it
> >> >> goes. From the logs it looks like the project attempts to install
> >> >> .net
> >> cli
> >> >> and fails and restore can't run, nor the build steps.
> >> >>
> >> >> Good luck. I will ping here if I have any breakthroughs myself.
> >> >>
> >> >>
> >> >> Laimonas
> >> >>
> >> >>
> >> >> On Sat, Feb 9, 2019 at 3:52 AM Simon Svensson <sisve@devhost.se>
> >> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> My JetBrains login would be the secretive "sisve@devhost.se".
> >> >>>
> >> >>> // Simon Svensson
> >> >>>
> >> >>> On 2019-02-08 17:26, Laimonas Simutis wrote:
> >> >>>> Simon,
> >> >>>>
> >> >>>> Oh my, I jumped into investigating the next builds in the chain
> >> assuming
> >> >>>> that the green means all is well, but you are right. That did
> >> >>>> not
> >> work.
> >> >>> Let
> >> >>>> me take a look at that and see what fails in base.
> >> >>>>
> >> >>>> What's your jetbrains login/email? I should be able to add
you
> >> >>>> to the administrative portion of the build
> >> >>>>
> >> >>>>
> >> >>>> Laimonas
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson <sisve@devhost.se>
> >> wrote:
> >> >>>>
> >> >>>>> Hi,
> >> >>>>>
> >> >>>>> The main build has a green checkbox and the word "Success"
next
> >> >>>>> to
> >> it. I
> >> >>>>> would classify this as slightly inaccurate, the build totally
> >> failed.
> >> >>>>> The build script logged an error, but exited with a exit
code
> >> >>>>> of 0
> >> so no
> >> >>>>> error was signaled.
> >> >>>>>
> >> >>>>> In short, "dotnet.exe restore" failed for unknown reasons,
and
> >> >>>>> this continues in all builds since 1190. There are silent
> >> >>>>> failures
> >> before the
> >> >>>>> restore; where the build scripts should output some information
> >> about
> >> >>>>> build numbers, directory paths and such, it should also
output
> >> >>>>> "dotnet.exe --version" and "dotnet.exe --info". These worked
in
> >> build
> >> >>>>> 1189, but failed in 1190.
> >> >>>>>
> >> >>>>> A perhaps related observation is that we, starting with
build
> >> >>>>> 1190,
> >> also
> >> >>>>> need to install .NET SDK 2.0.0 every run. It could be relevant
> >> >>>>> that
> >> it's
> >> >>>>> over 6 months between build 1189 and 1190, so there could
> >> >>>>> perhaps
> >> be a
> >> >>>>> change in TeamCity we need to track down.
> >> >>>>>
> >> >>>>> Is there an administrative ui in TeamCity that I can get
access
> to?
> >> I
> >> >>>>> believe the next step would be to debug (read: qualified
> >> >>>>> guessing)
> >> the
> >> >>>>> build script on a separate branch; would this be picked
up
> >> automatically
> >> >>>>> by TeamCity?
> >> >>>>>
> >> >>>>> // Simon Svensson
> >> >>>>>
> >> >>>>> On 2019-02-06 13:57, Laimonas Simutis wrote:
> >> >>>>>> Simon,
> >> >>>>>>
> >> >>>>>> I was able to get the base builds going but as you
already
> >> noticed, the
> >> >>>>>> subsequent builds are failing outright. The build structure
> >> >>>>>> has
> >> changed
> >> >>>>>> quite a bit since the last time I as in there, so it's
taking
> >> >>>>>> me
> >> some
> >> >>>>> time
> >> >>>>>> to understand what's going on. I will continue to try
to
> >> >>>>>> untangle
> >> it or
> >> >>>>> set
> >> >>>>>> up a new build flow in there if that fails.
> >> >>>>>>
> >> >>>>>> Chris, here is the project link:
> >> >>>>>>
> >> >>>>>
> >> >>>
> >> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=p
> >> rojectOverview
> >> >>>>>> if you want to take a look, perhaps you will see right
away
> >> >>>>>> what's
> >> up.
> >> >>>>>>
> >> >>>>>> Laimonas
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini
> >> >>>>>> <chris@brass9.com>
> >> >>> wrote:
> >> >>>>>>
> >> >>>>>>> We use Lucene.Net to do conventional search at:
> >> >>>>>>>
> >> >>>>>>> https://conservationx.com
> >> >>>>>>>
> >> >>>>>>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch,
> >> Levenshtein,
> >> >>> etc
> >> >>>>>>>
> >> >>>>>>> There was a problem mentioned about TeamCity; is
there a way
> >> >>>>>>> I can
> >> >>> learn
> >> >>>>>>> more in case I'd possibly be able to get it working?
Where is
> >> >>>>>>> it
> >> >>> checked
> >> >>>>>>> in, where is it running, etc
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message