ant-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Suereth <joshua.suer...@gmail.com>
Subject Re: Ivy slow resolution
Date Mon, 28 Apr 2014 14:36:51 GMT
On Mon, Apr 28, 2014 at 10:13 AM, Nicolas Lalevée <
nicolas.lalevee@hibnet.org> wrote:

>
> Le 28 avr. 2014 à 13:59, Josh Suereth <joshua.suereth@gmail.com> a écrit :
>
> > On a whim, I implemented parallel download of artifacts by hooking the
> > downloadArtifacts
> > method of Resolve engine.  While this can potentially speed up download
> > performance, the resolution times (getDepedencies) still dominates (of
> > course).  Parallel downloads is an often requested feature of sbt, so we
> > may still release this hook for our users, as they request it, but I
> think
> > the *really* just want faster resolution times.
> >
> > We may be doing some investigation into improving the performance of
> > getDependencies (and most importantly fetchDependencies).  I was curious
> if
> > anyone else in the Ivy community has attempted this before and what sorts
> > of guidance they could offer before we dig in.
>
> This has been discussed, but AFAIR nothing has been done other than
> discussing the potential issues to tackle.
>
> Things which would require some attention would be to find a way to
> properly report the progress of the parallel download: in the console when
> it's done via Ant or via the listeners for IvyDE.
> A thing that is worrying me, is that Ivy is quite sensitive to which
> thread is actually working on the resolve, see IvyContext's thread local.
> So a parallelization may not be a very upper level than the artifact
> downloading.
>
>
Totally understand.  I haven't had time to check if my resolution on
different threads has mucked up the ivy context, but I know I can propagate
context as needed, as long as downloading artifacts is idempotent (i.e. no
changes to artifacts).   The first implementation just defers reporting all
progress until AFTER the entire download, then notifies, which probably
breaks IvyDE, except I'm only overriding this in sbt itself, so there is no
IvyDE there :).

So, if this was something to contribute back to ivy (which I'm not sure it
is, given the lack of real speed-up we're seeing), I'd have to figure out
how to maintain the progress reporting notifications with the IvyContext,
possibly adapting my worker pool such that we register a thread-safe
IvyContext locally for the reporting events to get fired out of?

Does the event model, as it exist, prevent parallelism?



In terms of resolution improvements, mostly I was hoping that some
memoization/cache of immutable graph pieces could help avoid re-resolving
the same bits of the graph repeatedly.  Again, I haven't dug fully into the
details of IvyNode, ResolveData + VisitNode, but it seems likely to me that
we should be able to avoid calling "resolve" for an artifact (the same
ModuleRevisionId) more than once during full graph resolution, which (and I
may be wrong) appears to be the case when we profile Ivy.  I'm mostly
concerned with this aspect, and was wondering if anyone else has seen this
or if we're doing something very wrong in sbt.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message