incubator-ivy-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xavier Hanin (JIRA)" <>
Subject [jira] Commented: (IVY-399) Flexible Cache Management
Date Sun, 06 May 2007 08:56:15 GMT


Xavier Hanin commented on IVY-399:

To start working on this issue with a clean view of how the cache is currently implemented
in Ivy, I've listed all files which are cached, with their location, content and use:
All default locations are relative to the cache root.

default location: 
    the representation as an ivy file of the module descriptor used for dependency resolution
created by:
    ResolveEngine#resolve(ModuleDescriptor md, ResolveOptions options)
used by:
    PublishEngine#publish(ModuleRevisionId mrid, Collection srcArtifactPattern, String resolverName,
PublishOptions options)
        => when no src ivy pattern is provided, the cached file is used
    DeliverEngine#deliver(ModuleRevisionId mrid, String revision, String destIvyPattern, DeliverOptions
        => used as basis for the ivy file to deliver
    RetrieveEngine#getConfs(ModuleRevisionId mrid, RetrieveOptions options)
        => to list the configurations of a module to retrieve, when they are not provided
in the RetrieveOptions, or provided as '*' wildcard

default location:
    a map of resolved revision and status by dependency, stored as properties
created by:
    ResolveEngine#resolve(ModuleDescriptor md, ResolveOptions options)
used by:
    DeliverEngine#deliver(ModuleRevisionId mrid, String revision, String destIvyPattern, DeliverOptions
        => used to replace dynamic versions by static ones, and do recursive delivery according
to the dependencies status

default location (not configurable):
    resolveId + "-" + conf + ".xml" (usually resolvedId is [organisation]-[module])
    xml resolution report for one configuration
created by:
    XmlReportOutputter#output(ConfigurationResolveReport report, String resolveId, String[]
confs, File destDir)
used by:
        => to load previous dependencies to know if they have changed
    RetrieveEngine#determineArtifactsToCopy(ModuleRevisionId mrid, String destFilePattern,
RetrieveOptions options)
    IvyCacheTask (basis of cachepath and cachefileset)
        => to know the details of the last resolve operation

default location:
    an ivy file representation of a downloaded module descriptor, converted into the system
created by:
    BasicResolver#getDependency(DependencyDescriptor dd, ResolveData data)
used by:
    CacheManager#findModuleInCache(ModuleRevisionId mrid, boolean validate)
        => common method used to reload a module information from cache. This method is
in turn used by:
            * BasicResolver#parse => to avoid downloading a module descriptor before parsing
it (used when the version matcher requires metadata parsing)
            * BasicResolver#getDependency => to avoid downloading a module descriptor when
resolving a dependency
            * CacheResolver#getDependency => it's the purpose of the CacheResolver to reuse
module descriptors from cache
        => to check that the resolver does not point directly to the cache, which is forbidden
    RetrieveEngine#retrieve(ModuleRevisionId mrid, String destFilePattern, RetrieveOptions
        => when the retrieve operation is asked to retrieve module descriptors, files from
the cache are used

default location:
    a module artifact
created by:
    BasicResolver#download(Artifact[] artifacts, DownloadOptions options)
used by:
        => to know which file should be put in the path or fileset
    RetrieveEngine#retrieve(ModuleRevisionId mrid, String destFilePattern, RetrieveOptions
        => to know which files should be copied
        => to build the classloader to use for launching
        => to report the paths of artifacts
        => to add the artifact cache location in the artifact report
        => to check the file exists in cache
        => to delete old artifacts when downloading a module revision which has changed

default location:
    Artifact Origin (is local + location)
    Resolver Name
    Artifacts Resolver Name (different from Resolver in case of dual resolver)
created by:
    the cache manager to store the information listed in content
used by:
    the cache manager to load information listed in content.
    This information is used by:
        => used by the same classes as ArchiveFileInCache
    Resolve and Artifact Resolver
        => used by findModuleInCache, to reassociate a cached module to the resolvers which
originally resolved it
        Side Note: this causes problems when trying to reuse a same cache with different settings,
causing warning messages 'unknown resolver xxx'

> Flexible Cache Management
> -------------------------
>                 Key: IVY-399
>                 URL:
>             Project: Ivy
>          Issue Type: Improvement
>         Environment: ALL
>            Reporter: Eric Crahen
> Creating an issue at Xaviers request for improving the approach to cache management
> On 1/29/07, Xavier Hanin <> wrote:
>     Supporting this kind of graph
>     could be interesting, and what makes it difficult for Ivy is that Ivy
>     heavily relies on its cache mechanism, which makes it impossible to do
>     what you want (i.e. never put anything from your local repository to
>     the cache).
> This would be a very powerful feature to add. In 2.0, is there any reason for the cache
to have to be so baked into everything? In otherwords, why not implement every resolver and
all of the internal management w/ no caching what so ever baked in anywhere? Instead all caching
is done in a decorator fashion by wrapping a caching resolver around any other resolver? In
otherwords, the core of Ivy only knows about resolvers, no concept of cache exists in the
heart of Ivy.
> It seems to me this would be much more flexible, and it would still be very possible
to provide the syntactic sugar to make it very simple and even seemless to configure these
wrappers by default. At the same time, people who will use the flexibility have the power
to set up chains that might go something like.
> (logical chain)
>   localresolver
>   cacheresolver
>     httpresolver url="..."
>   cacheresolver
>     httpresolver url="..."
> There is no longer any need to have things like useLocal flags. Its already expressed
that the local resolver is not cached because its just not wrapped in a caching resolver.
> I think this idiom should be applied to both artifact and metadata resolution.
> One cool thing about this, is that in this way, since all caching is simply a type of
resolver we'd provide people who don't like the particular method we use to perform caching
in the resolver we provide are free to provide their own. This would address lots of the issues
that have been raised about caching, consistency, doing anything remotely fancy with local
resolvers - right now its very hard to address any of that because caching is not very plugable
as it stands.
> I think the only drawback is that it seems like its harder to configure out of the box
because most people by default would want to wrap every resolver with a cacheresolver - but
like I said, this is easily solvable by providing some simple syntactic sugar. For instance
the simplehttpresolver might be the name of an undecorated resolver for power users, and the
things named httpresolver would simple be an alias for the cacheresolver wrapped around the
simplehttpresolver (or subclass, whatever is the most sensible choice)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message