ant-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xavier Hanin" <>
Subject Re: Ivy: resolve latest.status (IVY-854)
Date Fri, 04 Jul 2008 15:00:05 GMT
On Fri, Jul 4, 2008 at 1:22 PM, Hans Lund <> wrote:

> I'm moving this from ivy-users:
> ...
> ´╗┐running:
> > >
> > > <ivy:resolve conf="runtime" />
> > > But: After a short while running on a CI (hudson) server, where the
> ivy
> > > repository also is placed, resolving stops working due to connection
> > > problems in httpClient.
> > > ---
> > >
> > > [ivy:resolve] 01-07-2008 13:16:24
> > > org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
> > > [ivy:resolve] INFO: I/O exception ( caught
> when
> ........
> > >
> > > which in the end results in unresolved dependencies.
> It looks like ivy:resolve is over aggressive towards the http reposiory.
> Basically I think that Ivy url resolver should operate with the same
> practice as a web crawler, that is limit the load on the http-server to
> a configurable load (max concurrent connections).

resolve in Ivy is not multi threaded, so if connections are cleanly closed
we shouldn't have more than one connection at once. But maybe we have a
problem of connection closing in some cases.

> Also, A reasonable error handler for the executeWithRetry, should be in
> place, now tries 5 times in a row. The exception here (address in use)
> on http always comes when the tcp stack on the server is exhausted, so
> trying again right away, might not be the best error handler :-).
> Improving error handling though, will have no effect if the ivy behavior
> is not changed.
> (This I think, should be considered httpClient features???)

Indeed, I think we use default behavior of httpclient, but it can certainly
be adjusted. Moreover, did you try to run Ivy without httpclient to see if
the behavior is different? It would help narrow down the problem.

> Regarding latest.status resolving strategy:
> As far as I can see the latest.status resolving fetches all ivy files
> and checksum files, which for n revisions is 2(n-1) too many http calls
> (and maybe also other calculations). Downloading checksum files for any
> other revision than latest.status, should be an error handle if the
> checksums don't match the ivy-file that resolves to latest.status.
> This should also speed up the resolving a tiny bit :-)

To be precise, Ivy doesn't download ivy files for all revisions: it
downloads ivy files for the latest revision, check if it has the expected
status, and if not, go to the revision before, and so on until it finds the
latest revision which has the expected status. The problem is that Ivy can't
know the revision of the dependency without downloading its module
descriptor. That's why I think we warn users that latest.status version
constraint is not very efficient.

What could be improved is to avoid downloading checksums in this particular
case, since checking ivy files consistency is not the highest priority when
looking for a latest.status revision.

> Now someone with inside knowledge of ivy, pleace comment. Right now I
> have no idea about the size of such changes, or even if this is the
> right strategy (eq. an alternative for ivy repositories could be to have
> an additional archive format, where all meta-data is stored in one file,
> and let ivy:publish update the archive along with normal publishing
> tasks)

You can implement this if you want with a custom resolver. If you really
need to use a lot of latest.status, it can be a good way to go. Or maybe
store one metadata file per module (like maven-metadata.xml) storing the
latest version for each status, updated only when you publish a new version.
What I don't like with that is that it's more subject to concurrency issues,
and since Ivy is a client side only tool, it's difficult to avoid
concurrency issues when you update the repository. So maybe the best option
would be to implement a small server knowing for each module what is the
latest revision by status, handling publications, and answering a custom
resolver. In other words a repository manager...


> Regards
> Hans Lund
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

Xavier Hanin - Independent Java Consultant
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message