Fix dependency/groupId computation when publishing to Maven #50
Comments
|
Fortunately, Tycho now reliably lists the original groupId/artifactId/version in its metadata when artifacts come from Maven, so we probably won't need further mapping rules.
Overall, it might be possible in the future to have this happening automatically while building the artifacts, as part of the build, so we'd save the need to generate the pom at that stage and could just consume something already existing; but that would be some next steps. Decoupling as suggested above will help towards that goal. |
|
I don't know much about the aggregator stuff, but should we not invest the effort directly at the projects? I think it would even be feasible to simply don't publish the aggregator stuff anymore to maven, even if we might miss a release to be available on maven then it might give the required attentions to the projects to work on that topic. I fear as long as there is "someone else to do it" there is not much attraction for the projects to do it otherwise. |
|
On Wednesday, June 15, 2022, Christoph Läubrich ***@***.***> wrote:
I don't know much about the aggregator stuff, but should we not invest
the effort directly at the projects?
The aggregator is an extra layer which brings extra requirements and
expertise over upstream (p2). If we remove this layer, we will probably
allow the pom generation to work in more contexts (eg Tycho).
IMO, we're in a case of "less is more" here.
…--
Mickael Istria
Eclipse IDE <https://www.eclipse.org/eclipseide> developer, for Red Hat
Developers <https://developers.redhat.com/>
|
|
I think it's not clear what does "aggregator" means here, it's highly overloaded term . In this case it's https://github.com/eclipse-platform/eclipse.platform.releng/blob/master/publish-to-maven-central/SDK4Mvn.aggr which is purely Eclipse TLP thing . |
|
We may also use directly Equinox instead of p2 to compute dependencies, it should usually be more reliable and code would be even more portable, even to Felix. |
|
It is a good plan to make the generation of pom's for publication on Maven-Central more reliable and more 'native' without requiring an extra aggregator-layer that needs manual care. Generating the deployed-poms (to enhance it with explicit information that are implicitly known at build-time) is also something that is planned for Maven-4 and 5 so maybe we get (better) support for that from the Maven side as well. However if the current approach is used ever again I suggest to implement a sanity-check if the generated dependency really exists at |
This is exactly what I do in my project when I compute the BOM file corresponding to an Eclipse release: I did some health check on the published artifact from the 2022.06 release (groupIds |
|
@jmini do you think you can spend some time to put your check into https://github.com/eclipse-platform/eclipse.platform.releng/tree/master/publish-to-maven-central ? Even though we are looking for better approach, having a sanity check is always welcome. I don't know the dirty details but if you're willing to give it a try @sravanlakkimsetti should be able to answer questions. |
|
Sure I can give it a try, but I am not aware how the publish process looks like. For me at some point the jar and the poms "ready for publication" are stored somewhere. And at this point in time, we could make maven verify that they are correct (point to existing artifacts). |
|
Note that I could look at doing such a check in the aggregator, but if you plan to re-implement everything it provides anyway, that would be kind of a waste of time... |
|
Please open a separate issue to track discussion and actions about checks/verification. |
|
From @mickaelistria:
As long as your solution produce jars and poms that do not need tycho to be consumed, this is fine. Because this is the real value of this aggregator approach. Being able to be consumed by "the rest" of the java world (even if they could use tycho, but this is an other story). And please also noticed that the coordinates in the projects are not the same than the published one. Example Coordinates in the project (source) <groupId>org.eclipse.core</groupId>
<artifactId>org.eclipse.core.jobs</artifactId>
<version>3.13.100-SNAPSHOT</version>Coordinates of the published artifact in maven central (created by this aggregator): <groupId>org.eclipse.platform</groupId>
<artifactId>org.eclipse.core.jobs</artifactId>
<version>3.13.0</version>(not saying that this has to stay forever like this, just telling how this currently done) |
https://ci.eclipse.org/releng/view/Publish%20to%20Maven%20Central/job/CBIaggregator/ creates jars and poms in "ready for publication" state. you can see them in the built artifacts of the job run for example https://ci.eclipse.org/releng/view/Publish%20to%20Maven%20Central/job/CBIaggregator/125/artifact/repo-125/ |
Do you mean by reliable that it also considers dependencies due to Import-Package? As a side-note the issueManagement element and connection are outdated as well. |
I really don't get why we should put that much effort here if everything is already implemented in Tycho already for a solution that more should be an intermediate than becoming a constant source of development effort. Why not improving Tycho as it is currently quite good at these (just use dependency:list and Tycho will tell you everything you need here) and even make the projects publish suitable pom data in the first place? |
Yes and finally people will realize that they rely on a best-effort service and probably would start to fix the project itself instead of using this workaround forever. There is nearly no excuse today as large effort where already put into Tycho to support generation and publishing of suitable pom data right now, it is just that no one uses it because people are either not aware or whatever of these features and just pass away for the smallest inconvenience instead of reporting good bug report and/or improving the code as long as we offer them a (for them) comfortable solution. So my "vote" her would be, announce now that we no longer will publish bundles through this script anymore after the 2022-09 release and that anyone requiring it (including Tycho of course!) has to start right now to take the required actions to make these projects capable of publishing their own maven metadata! |
Can you outline the required steps already? And aren't the projects published by this script 'only' the Eclipse-TLPs (i.e. eclipse-platform, Equinox, JDT and PDE)? At least only they occur in this repo's |
Yes, so only our issue here, no one else to blame to not start right now... I'm currently trying to get Equinox ready for this, but any help would be appreciated of course. |
|
We do need to publish Maven artifacts for the continued success of the project. The ecosystem is too big and important to suddenly break it and the ROI is good enough to justify Eclipse project publishing the jars, moreover there are allocated resources assigned to keeping it working. The question is not whether to publish or not, but instead how to do it better and better. |
|
I have to say that I am confused by some of the discussion here. Is anyone seriously suggesting that the Eclipse TLP should not publish its own artifacts to maven central? I 100% agree with @mickaelistria last comment that we must continue to publish our JARs as maven artifacts for continued success. |
Second that. We should not stop publishing to Maven-Central but we should start to work on an improved workflow. |
No one is saying that, nor that other projects should publish anything, but we should stop doing it as an extra step by trying to restore the information from an aggregated P2 repository content and instead the originating project should publish it to maven-central at the first place! Because at that point there are all information present, there is no need to guess anything, e.g. the recent issue would simply not occur, as tycho would have known the exact GAV of that dependency. In contrast, trying to deduce this information after the fact is cumbersome and error prone as just proven right now. Any more effort in this fragile solution should better be placed to enable the TLPs to do it right now, as @akurtakov explained this solution was from a time where we just have had completely different background and the goal was to just bring at least some limited set of artifacts to maven. |
Just keep in mind that publishing is more than creating mavenized artifacts. Both steps can possibly be done under different responsibility (centrally vs. decentralized).
While still some orbit bundles are consumed, does tycho know about their "original" GAV? How? Does tycho have a solution for mavenizing (platform dependent) OSGi fragments? Another observation: when the workflow for publishing to Maven Central was created, the feedback from other projects (outside Eclipse TLP) was near zero, i.e., there was not much enthusiasm about joining the party. Maybe this situation has changed in the meanwhile? |
I don't think it has original GAV for Orbit bundles. After all, they're not Maven artifacts anymore, so even having them wouldn't help as the Orbit artifact is not the Maven artifact.
It already has some good bricks to consolidate this story (
I think that the recent improvements in Tycho make it more and more accessible and appealing to get "more Maven" now. |
|
EMF publishes to Maven as well, though I can't say it's with enthusiasm: https://ci.eclipse.org/emf/job/maven-publish/ Note that direct-from-maven artifacts have this kind of information:
And even ones from Orbit have stuff like that:
|
... where a maven-groupid of |
What just came into my mind (I don't know if this is already used), but the Maven-Central search API allows to search for the artifact by its jar's checksum: https://central.sonatype.org/search/rest-api-guide/ Alternatively it can still be attempted to search by fully-qualified classname. Given that nobody else has re-published the artifacts and taking the version into account this could also lead to a unique result (maybe multiple or even all classes from that dependency have to be used) to get a singleton union. However this has to be done with care to not open the door for supply chain attack where somebody else publishes a malicious artifact that is falsely selected by such algorithm. |
As these are not maven dependencies tycho cant know the gav but no one esle could ever provide one, so they are just "hidden" to the maven world, simple solution (without guessing too much, or even query external servers), just use the now available maven-target locations and everything will work like magic :-) (ed has marked the relevant parts but not the "magic" part that is the
Not sure what exactly you mean, but also platform dependent OSGi fragments are just maven artifacts and can be deployed, so there is nothing to do here, it is just that maven has no mean to reference them in a comfortable way, there are currently two ways to solve this: |
|
A coarse grain strategy, which is an evolution of what is currently done more than a revolution. would be:
|
Funny enough the group id actually exists: https://mvnrepository.com/artifact/org.eclipse.orbit.bundles Nerveless the com.google.guava / 30.1.0.v20210127-2300 delivers the following in their META-INF/maven/.../pom.xml what is not deployed to central and the version is obviously wrong and the artifact id seems a bit too broad... so what I have learned from my investigations in Tycho, the Maven infos of Eclipse bundles in P2 and in the artifact itself are too often wrong to say one could "just use it"... |
|
Here is a script that could be as a basis to replace the aggregator some time later pushd repo
# download
# unzip
repo=$(pwd)
pushd plugins
mkdir .mvn
echo '<?xml version="1.0" encoding="UTF-8"?>
<extensions>
<extension>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-build</artifactId>
<version>2.7.2</version>
</extension>
</extensions>
' > .mvn/extensions.xml
echo "<project xmlns='http://maven.apache.org/POM/4.0.0' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd'>
<modelVersion>4.0.0</modelVersion>
<groupId>org.eclipse.platform</groupId>
<artifactId>org.eclipse.toPom</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>pom</packaging>
<build>
<plugins>
<plugin>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-maven-plugin</artifactId>
<version>2.7.2</version>
<extensions>true</extensions>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>this</id>
<layout>p2</layout>
<url>file:${repo}</url>
</repository>
</repositories>
</project>
" > pom.xml
for plugin in $(ls org.eclipse.*.jar); do
folderName="$(basename "$plugin" .jar)"
mkdir -p $folderName/META-INF
unzip -p $plugin META-INF/MANIFEST.MF > $folderName/META-INF/MANIFEST.MF
pushd $folderName
mvn dependency:list
popd
done
popdOne issue is that at the moment |
|
I'm making decent progress here and have some conclusions to share:
and some questions do remain:
|
Why not let orbit deploy them as part of the release?
Why not invest the time to add them in the "real" source at the first time, yes I know some are hard, but obviously not all ...
Why not omit them at all? Or replace by newer variants (see above) with maven? |
Indeed, that would be best. But "let" means more "require" here, and I don't think Orbit is a lively project enough to fulfill requirements coming from downstream.
Currently, I don't think anyone has enough time to invest in that to expect it can be completed for September release
For some artifacts (eg |
At least some stuff is actually deployed to maven: https://mvnrepository.com/artifact/org.eclipse.orbit.bundles?sort=newest so there seems to be a way already...
Maybe because we are too busy? |
Right, and we're going to adopt those.
Yes, We're paying back technical debt. |
|
I think this is now fixed by adoption the newer aggregator builds (which are capable of better computing Maven coordinates) and continuous work to
|
In 4.24, some artifacts are now consumed directly from Maven and already have groupId/artifactId to be used when describing them as dependencies when generating pom files to publish to Maven Central.
The current https://github.com/eclipse-platform/eclipse.platform.releng/blob/master/publish-to-maven-central/SDK4Mvn.aggr defines static mapping rules that used to be reliable in the time of Orbit, but aren't any more. See eclipse-equinox/equinox.bundles#54 / eclipse-equinox/equinox.framework#70 for an instance of this problem.
The pom generator should be fixed to use the reference groupId/artifactId when listing dependencies as much as possible instead of hardcoding rules.
The text was updated successfully, but these errors were encountered: