From dev-return-88438-apmail-ant-dev-archive=ant.apache.org@ant.apache.org Wed Mar 25 19:02:29 2015 Return-Path: X-Original-To: apmail-ant-dev-archive@www.apache.org Delivered-To: apmail-ant-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 71BC7177E6 for ; Wed, 25 Mar 2015 19:02:29 +0000 (UTC) Received: (qmail 96538 invoked by uid 500); 25 Mar 2015 19:02:24 -0000 Delivered-To: apmail-ant-dev-archive@ant.apache.org Received: (qmail 96475 invoked by uid 500); 25 Mar 2015 19:02:24 -0000 Mailing-List: contact dev-help@ant.apache.org; run by ezmlm Precedence: bulk List-Unsubscribe: List-Help: List-Post: List-Id: "Ant Developers List" Reply-To: "Ant Developers List" Delivered-To: mailing list dev@ant.apache.org Received: (qmail 96464 invoked by uid 99); 25 Mar 2015 19:02:23 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Mar 2015 19:02:23 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mgainty@hotmail.com designates 65.55.111.89 as permitted sender) Received: from [65.55.111.89] (HELO BLU004-OMC2S14.hotmail.com) (65.55.111.89) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Mar 2015 19:01:58 +0000 Received: from BLU172-W14 ([65.55.111.71]) by BLU004-OMC2S14.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.22751); Wed, 25 Mar 2015 12:00:52 -0700 X-TMN: [Onk4Y1KYbkYVo7R0kbBfxtDqZfUWgfFs] X-Originating-Email: [mgainty@hotmail.com] Message-ID: Content-Type: multipart/alternative; boundary="_f8fcbd3a-8974-4ff8-9088-1048836af148_" From: Martin Gainty To: Ant Developers List Subject: RE: Possible Ivy bug (and suggested fix) in ChainResolver Date: Wed, 25 Mar 2015 15:00:51 -0400 Importance: Normal In-Reply-To: <03c7c638579a4571a1c45326d088fc5d@MAIL02.blueorigin.com> References: <03c7c638579a4571a1c45326d088fc5d@MAIL02.blueorigin.com> MIME-Version: 1.0 X-OriginalArrivalTime: 25 Mar 2015 19:00:52.0445 (UTC) FILETIME=[03D8F0D0:01D0672E] X-Virus-Checked: Checked by ClamAV on apache.org --_f8fcbd3a-8974-4ff8-9088-1048836af148_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable > From: LKratzke@blueorigin.com > To: dev@ant.apache.org > Subject: RE: Possible Ivy bug (and suggested fix) in ChainResolver > Date: Wed=2C 25 Mar 2015 18:16:08 +0000 >=20 > I found the root cause of the fishiness. It was not a bug (and was not ac= tually in ChainResolver) but is in my opinion a rather unfortunate and poor= ly chosen default setting that is to blame (the latest-strategy). >=20 > During resolution in the chain resolver=2C the current sub-resolver tries= to determine whether it should be skipped and the previous artifact be sel= ected over the current artifact. Assuming that the "force" and "dynamic" te= sts do not result in an early rejection of the current artifact=2C the "lat= est" of the two artifacts is selected by sorting the artifacts in a List us= ing a Comparator suited to the current latest-strategy (latest-revision=2C = latest-time=2C etc). >=20 > The problem here is quite simple. The default latest-strategy is "latest-= revision". So when a second artifact has been resolved=2C and it is of the = same revision as the first=2C nothing gets sorted. The result is that you w= ill get the first artifact found based upon the ordering of your repositori= es in the chain instead of the newer of the two artifacts. >=20 > I think that either latest-time needs to be the default strategy or the l= atest revision comparator needs to do a secondary sort to sort by lastModif= ied time. >=20 > Possibly allow configuration of this behavior (should there be a secondar= y sort by time to avoid stale artifacts=2C or not so that repository order = breaks the tie). This is important (critical) behavior and should be config= urable. >=20 > Defaulting to latest-revision will not only deliver undesired stale artif= acts=2C but it is unclear to the user why they are getting stale artifacts = or how to make it stop happening. The latest-time strategy will give you th= e latest revision 99% of the time and the latest artifact 100% of the time.= But the latest-revision strategy will give you the latest artifact 100%=2C= 50%=2C 33%=2C or 25% of the time when the revision numbers are the same=2C= depending upon how many resolvers you have (1/n)=2C and assuming that any = repository may contain the latest artifact. >=20 > Furthermore=2C the docs do not give a great description of "force". I lea= rned much about the actual behavior of this attribute while debugging. Firs= t thing I learned was that it is not a good name. >=20 > What force actually does is it allows a resolver to be considered when a = previous resolver has found an artifact. After the first artifact has been = found=2C only repositories with force=3Dtrue will have a chance of competin= g (for instance=2C they might have a newer version of 1.0.0-SNAPSHOT). Othe= rwise=2C they are discarded immediately and no date comparison is attempted= . >=20 > Force should actually be named "considerAlways" or "considerAnyway". That= seems to be a more suitable name. No action requested here=2C but pointing= out that this attribute has a misleading name. >=20 > Summary: > 1 - Please reconsider changing the default latest-strategy to be latest-t= ime. >=20 > 2 - Please consider adding a secondary "lastModified" sort to LatestRevis= ionStrategy.ArtifactInfoComparator whether or not you change the default la= test-strategy to latest-time or not. >=20 > 3 - Please document=2C illustrate=2C and demonstrate in one place the beh= aviors of chain resolver in combination with force=2C returnFirst=2C defaul= tLatestStrategy=2C ivy.resolver.default.check.modified=2C useOrigin=2C and = other settings and attributes that affect resolution behavior. (I am workin= g on this document now.) >=20 > 4 - Please consider creating independent caches by default for each repos= itory. I have not drilled down on this issue yet=2C but I suspect that it f= ixes serious cache collision issues that I think I saw while debugging (fou= nd and selected local repo artifact=2C checked cache before delivery=2C end= ed up delivering cached stale artifact that came from a totally different r= epo :( ). >=20 > Thanks=2C >=20 > L.K. MG>i think we can take hints from maven brothers on a tested strategy MG>to referencing dev jars during development..their solution is to employ = SNAPSHOT version during CI cycles MG>SNAPSHOTs are available until the jar is promoted to RELEASE at which po= int a tag is assigned to version MG>SNAPSHOT delivers ${project.id}-YYYYMMDD.hhmmss.jar so unless you have m= ultiple machines able to gen MG>deployables within a second the last second is the arbiter which clearly= identifies the latest jar MG>Snapshot versions are ephemeral until the next snapshot build so remote = lookup would not be implemented http://books.sonatype.com/mvnref-book/reference/pom-relationships-sect-pom-= syntax.html MG>repository caches when stored within a regular Nexus Repository are type= d as Proxy/Hosted/Virtual MG>ProxyApache=2CProxyCentral or ProxyCodehaus MG>Hosted3rdParty=2CHostedRelease=2CHostedSnapshot MG>VirtualRepo (Virtual repos are generally for OSGI bundles) MG>Once you know the general type Proxy or Hosted or Virtual you can then s= elect sub-type (such as Hosted3rdParty=2CHostedRelease=2CHostedSnapshot) https://books.sonatype.com/nexus-book/reference/confignx-sect-manage-repo.h= tml MG>thank you for taking the necessary time to think this through >=20 > From: Loren Kratzke > Sent: Tuesday=2C March 24=2C 2015 1:09 PM > To: 'dev@ant.apache.org' > Subject: Possible Ivy bug (and suggested fix) in ChainResolver >=20 > I have a some observations about how the chain resolver selects a depende= ncy. I think this may be a bug but I am not sure because the intent of the = source code is not entirely clear. It reads one way=2C but behaves in a dif= ferent way. I have pinpointed the exact spots in code where this happens. >=20 > Here is my simple test setup used to debug this issue. I have two resolve= rs (Filesystem and URL) configured in a ChainResolver in that order. I publ= ish to one resolver and then the other repeatedly and consume the result in= another project. I use checkModified=3Dtrue and changingPattern=3D".*" on = both resolvers. >=20 > My artifact is simply a text file with the current date and time so it is= easy to see whether you get fresh or stale artifacts from the repos. >=20 > When I consume the published artifact from the other project=2C I will ge= t the artifact from the first configured resolver in the chain (Filesystem = in this case). But I know from debugging that the second resolver is also e= valuated. So as an experiment=2C I added force=3D"true" on the second resol= ver to see if I could force Ivy to ignore the first result and favor an art= ifact returned by the second resolver. Instead=2C Ivy returned the artifact= from the first resolver even though the second artifact was newer AND the = second resolver had force=3D"true". >=20 > When I debugged this to see why the first artifact was chosen over the se= cond artifact=2C I found something very fishy. >=20 > ChainResolver.getDependency() iterates over each resolver in the chain. F= irst it found the Filesystem resolver and the artifact and next it found th= e URL resolver and artifact. Next it calls BasicResolver.getDependency() wh= ich will compare the previously resolved artifact with the current artifact= . >=20 > This is where it gets very fishy. At the end of the getDependency() metho= d it calls AbstractResolver.checkLatest() which I assume is intended to ret= urn the latest of the two artifacts. But that comparison never happens. Abs= tractResolver.isAfter is invoked with two artifacts to be compared and a nu= ll Date. Since the date is null=2C the two artifacts are never compared and= no matter what=2C the first artifact will be returned and the second one d= iscarded and a verbose message will be emitted stating that the second arti= fact is older than the first artifact=2C every time. The message is on line= 533 of AbstractResolver. (I am looking at Ivy-2.3.0 so if that line does n= ot make sense on trunk then let me know.) >=20 > Message.debug("\tmodule revision kept as younger: " + newModuleDesc)= =3B > saveModuleRevisionIfNeeded(dd=2C newModuleFound)=3B > return newModuleFound=3B >=20 > The message is not true. The artifact that was kept was the older of the = two and a comparison of lastModified never happened (and never can happen i= n the current code as far as I can tell). >=20 > So the actual logic in AbstractResolver.checkLatest() simply returns the = first artifact found. While this is not a bad behavior=2C it does not seem = like it is the intended behavior. I mean=2C why go through all the trouble = of pretending to compare two artifacts using date methods when the logic ne= ver executes because the passed in Date object is null. And why emit a mess= age stating that one was determined to be older than the other. That is sup= er fishy. >=20 > Furthermore=2C the next line in ChainResolver.getDependency() after resol= ver.getDependency() is called (ChainResolver line105) references isReturnFi= rst(). That is fishy because none of that matters any more. The current art= ifact was rejected on the previous line of code and the previous (aka first= ) artifact is now the current artifact and is the one that will be returned= (without a date comparison=2C and for the arbitrary reason that is was fou= nd before the other one). >=20 > I think that the intent of the null Date object is to compare each artifa= ct to a static Date configured elsewhere (I have no idea where)=2C but if t= he code were to actually compare the lastModified dates of the two artifact= s=2C a useful result would happen - Ivy would return the latest artifact fr= om across multiple repositories. >=20 > That is huge because I have never been able to get Ivy to do this. I have= never seen anybody get Ivy to search multiple repositories and return the = latest artifact. This is useful for local development when you publish loca= lly to consume locally modified artifacts. It would be nice to have the opt= ion of picking up newer artifacts from a central repo when those occur with= out having to blow away a local repository and its cache. >=20 > (By the way=2C giving my local repo its own cache seems to have solved so= me other strange issues I was having. I recommend this to everybody and I t= hink it should be a default in Ivy=2C but that is debatable and would need = some more research and concensus.) >=20 > I think that this is a good feature and should be configurable. I think p= ossibly it was intended to be configured via ChainResolver.returnFirst=3D"t= rue|false" but that code executed when it was too late and the decision had= already been made. If I were to make this a feature=2C and make it configu= rable=2C I would configure this using an attribute named returnFirst becaus= e that is the exact facet of functionality that we are talking about here. >=20 > Thanks for your attention. Hope I am helping here. I am considering codin= g this to see if it works as expected. I would be happy to report my result= s and provide a patch if anybody is interested in evaluating this. >=20 > L.K. = --_f8fcbd3a-8974-4ff8-9088-1048836af148_--