Validate all poms are resolvable

[akurtakov]

Discussion started in #50 .
Goal is to have poms verified/validated to be resolvable by Maven independent of the way these poms are generated

[sravanlakkimsetti]

https://ci.eclipse.org/releng/view/Publish%20to%20Maven%20Central/job/CBIaggregator/ creates jars and poms in "ready for publication" state. you can see them in the built artifacts of the job run for example https://ci.eclipse.org/releng/view/Publish%20to%20Maven%20Central/job/CBIaggregator/125/artifact/repo-125/

[HannesWell]

Copying the relevant part of #50 (comment):

"I suggest to implement a sanity-check if the generated dependency really exists at https://repo1.maven.org/maven2/. At least for those dependencies that are not part of the build it should be relatively easy to check if https://repo1.maven.org/maven2/<groudId-elements>/<artifactId>/<version> exists (i.e. not returns a 404) and then fail the build if it is missing. At least we then don't publish broken poms anymore."

However checking if the entire poms resolves would of course be better, just in case there are other erroneous parts.

[mickaelistria]

Best place to add such sanity check would Maven itself or repository manager (Nexus).

[akurtakov]

@mickaelistria What is the status of this one?

[merks]

As described here, I've added such verification support to complement the cbi p2 aggregator:

• Import-Package and other requirements not translated as Maven #71
• Improve the aggregator's maven publishing to support resolving package requirements eclipse-cbi/p2repo-aggregator#11

[mickaelistria]

Would the maven pom generator fail if some incinsistency is detected (so build would fail), or do we need some extra steps in the consuming scripts to check some logs?

[merks]

At this point, only the *.aggran (in the Aggregator Analysis Editor) does deeper analysis of the generated poms to check that all dependencies resolve.

I'm not sure I understand fully how the platform does the publishing, but effectively the aggregator builds a Maven repository with everything in the Platform's p2 repository, but then some other process only publishes a subset of those things to Maven. Which subset is actually published is not know to the CBI aggregator. In the *.aggran I tried to capture that information to the best of my knowledge looking also at what I see published previously by the Platform in Maven. This way I avoid checking dependencies of things not actually published to Maven by the Platform.

In any case, there is room for further for improvement over the manual checking of the *.aggran in the editor,though that interactive approach is useful for ensuring that the *.aggr itself has specified appropriate/correct mapping rules.

[merks]

Here's what I started, expanding on @mickaelistria approach, to do better testing of resolution:

set -x

cd /c/Users/merks/build/final
rm -rf  a com* eclipse-* jak* javax* juni* net* tool* org/ap* org/api* org/ass* org/bouncycastle/ org/junit org/mockito/ org/objenesis/ org/osgi org/ow2/ org/tukaani/ org/w3c/ org/opentest4j org/slf4j/ org/eclipse/birt org/eclipse/ecf org/eclipse/jetty/ org/eclipse/orbit/ org/eclipse/emf/
cd -

repo="file:/C:/Users/merks/build/final" 
repo="https://repo.eclipse.org/content/repositories/eclipse-snapshots/"
snapshot="-SNAPSHOT"
#snapshot=""

rm -rf .m2/repository
mvn dependency:get -X -Duser.home="D:/Users/merks/test-mvn" -Dosgi.platform=win32.win32.x86_64 -U -DoverWriteSnapshots=true -Dartifact=org.eclipse.jdt:org.eclipse.jdt.ui:3.27.100$snapshot -DremoteRepositories=$repo 2>&1 | tee platform.log

rm -rf .m2/repository
mvn dependency:get -X -Duser.home="D:/Users/merks/test-mvn" -Dosgi.platform=win32.win32.x86_64 -U -DoverWriteSnapshots=true -Dartifact=org.eclipse.pde:org.eclipse.pde.ui:3.13.700$snapshot -DremoteRepositories=$repo 2>&1 | tee jdt.log

rm -rf .m2/repository
mvn dependency:get -X -Duser.home="D:/Users/merks/test-mvn" -Dosgi.platform=win32.win32.x86_64 -U -DoverWriteSnapshots=true -Dartifact=org.eclipse.platform:org.eclipse.ui.workbench:3.127.0$snapshot -DremoteRepositories=$repo 2>&1 | tee pde.log

As described in #136 (comment) it is possible to do this testing on the local repository before promoting it.

So after EclipseCon I plan to expand on this idea to add a build job/stage to run these kinds of tests before the build jobs that do the promotion are kicked off.

[merks]

I created this new job:

https://ci.eclipse.org/releng/job/RepositoryValidator/

It's triggered by the aggregator job:

https://ci.eclipse.org/releng/job/CBIaggregator/

The RepositoryValidator job does a dependency:get on each to-be-published artifact and fails if that fails. Only if all artifacts can resolve all their transitive dependencies are the following three downstream jobs triggered:

https://ci.eclipse.org/releng/job/PublishJDTtoMaven/
https://ci.eclipse.org/releng/job/PublishPDEToMaven/
https://ci.eclipse.org/releng/job/PublishPlatformToMaven/

This validation uncovered one additional failure fixed by the above pull request.

One outstanding issue is the handling of the source for ecj but that's an orthogonal issue.