Switch from Ant to Maven

[adamretter]

This finally switches the build system from Ant to Maven.

Closes #1792
Closes #736
Closes #509
Closes #1576
Closes #964
Closes #1489
Closes #2070
Closes #2095
Closes #1659
Closes #2356

---

To use it...

Prerequisites

• Java 8
• Maven 3.6.0
• Docker (if you want to build the Docker images)

For building DMG (Linux users only)

• sudo yum install hfsutils hfsplus-tools (CentOS etc. only)
• sudo apt-get install hfsprogs hfsplus (Debian etc. only)

Building with Maven

1. It is as simple as: mvn clean install -DskipTests

2. If you want to create distributions mvn clean package -DskipTests

   1. or... if you are on MacOS and also wish to sign the Mac App and DMG mvn -Dmac-signing=true -Dmac.codesign.identity="The ID of your Apple Developer Certificate" clean package -DskipTests
   2. if you want to also build the Docker images and have Docker installed, you can also add the flag -Ddocker=true.

3. If you want to run tests mvn clean test

IDE Support

Most Java IDE's (IntelliJ, NetBeans, etc.) natively support Maven projects. Just open the root folder of the project with the IDE and make sure to click "Enable auto-import" if given the option.

Running eXist-db from the IDE

TBC...

[dizzzz]

Nice to see this PR ! A big step forwards.....

[dizzzz]

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:21 min
[INFO] Finished at: 2019-04-25T13:18:57Z
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.codehaus.mojo:appassembler-maven-plugin:2.0.1-SNAPSHOT or one of its dependencies could not be resolved: Failure to find org.codehaus.mojo:appassembler-maven-plugin:jar:2.0.1-SNAPSHOT in http://clojars.org/repo was cached in the local repository, resolution will not be reattempted until the update interval of clojars.org has elapsed or updates are forced -> [Help 1]

[adamretter]

@dizzzz see the temporary pre-requires bit in the PR description ;-)

[wolfgangmm]

I tried on two machines, but the build always fails with the error shown in the screenshot.

[duncdrum]

same problem here: appassembler build fails, subsequently exist build fails

[INFO] ------------< org.codehaus.mojo.appassembler:appassembler >-------------
[INFO] Building Appassembler 2.0.1-SNAPSHOT                               [1/4]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) @ appassembler ---
[INFO] 
[INFO] --- cobertura-maven-plugin:2.7:clean (default) @ appassembler ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Appassembler 2.0.1-SNAPSHOT:
[INFO] 
[INFO] Appassembler ....................................... FAILURE [  0.559 s]
[INFO] Appassembler :: Model .............................. SKIPPED
[INFO] Appassembler :: Generic Bootstrapper ............... SKIPPED
[INFO] Appassembler :: Maven Plug-In ...................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.837 s
[INFO] Finished at: 2019-04-26T18:04:30+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:cobertura-maven-plugin:2.7:clean (default) on project appassembler: Execution default of goal org.codehaus.mojo:cobertura-maven-plugin:2.7:clean failed: Plugin org.codehaus.mojo:cobertura-maven-plugin:2.7 or one of its dependencies could not be resolved: Could not find artifact com.sun:tools:jar:0 at specified path /Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk/Contents/Home/../lib/tools.jar

thats with me telling inteellij to use java8, no clue why its searching in java11

[duncdrum]

see eXist-db/documentation#333

[dizzzz]

@duncdrum I suspect java11 is on you class path..... and default jDK ? what is returned by /usr/libexec/java_home -V ?

[dizzzz]

for me the local build is OK; interestingly slow, but it finishes

[wolfgangmm]

Two questions I have been asking myself trying to use the build and which likely need to be documented properly somewhere:

1. how are we supposed to run a development instance of eXist during development or for testing? Obviously the previous approach of just calling bin/startup.sh no longer works. I see I can launch eXist from one of the generated distribution directories below exist-distribution/target/. Is this the way to go?
2. can we have run configurations for intellij to launch either eXist with jetty or the Java admin client? Both configurations are missing now, but I often used them for debugging.

[adamretter]

@dizzzz The first time you build eXist-db with Maven, there are a lot of plugins and dependencies that it needs to download which will make the build quite slow. Subsequent builds are much faster.

We also have a security analysis plugin (org.owasp:dependency-check-maven) as part of the build that checks for known vulnerabilities in our dependencies. This can be quite slow as it updates its database every 24 hours and the checks against the dependencies themselves can take a few seconds each to run.

If you wish to skip that step, which is fine when you are doing dev and lots of rebuilds etc, you can add -Ddependency-check.skip=true after your install goal when running Maven, e.g. mvn install -Ddependency-check.skip=true or whatever goal you are calling.

[adamretter]

@duncdrum I suspect you have the Project SDK in IntelliJ set to a Java 11 JDK.

In IntelliJ, if you Right-Click on the root of the project and click Module Settings in the popup menu, then click project you should have something like in the attached screenshot.

[adamretter]

@wolfgangmm I am not sure if you are using Linux, but I am guessing so... there was a very small bug (just a single character) in the Linux/Unix script for creating a DMG, this was causing your build to fail. Sorry about that. I have pushed a fix and re-tested on Fedora.

The build was passing for @dizzzz as he is on a Mac and the script on the Mac uses the native Mac tools and did not have a bug.

[adamretter]

@ALL when reporting bugs please tell me the Operating System that you are using as some parts of the build (i.e. the DMG) are Operating System specific.

[adamretter]

@wolfgangmm in answer to your question:

how are we supposed to run a development instance of eXist during development or for testing? > Obviously the previous approach of just calling bin/startup.sh no longer works. I see I can launch eXist from one of the generated distribution directories below exist-distribution/target/. Is this the way to go?

Yes. Similarly to before. You can just run exist-distribution/target/exist-distribution-5.0.0-RC8-SNAPSHOT-dir/bin/startup.sh

can we have run configurations for intellij to launch either eXist with jetty or the Java admin client
Both configurations are missing now, but I often used them for debugging.

Firstly, you should make sure that your IntelliJ is using a sufficiently new version of Maven. I have been testing with 3.6.0 and that works well, so maybe best to install that and set IntelliJ to use it via. the Menu: Settings (Preferences on Mac) -> Build, Execution, Deployment -> Build Tools -> Maven, and setting it to wherever you installed Maven 3.6.0.

IntelliJ natively supports Maven, in this way we no longer need to ship all .iml and .idea/ files. Instead we can just ship shared Run Configurations. I have now committed some Run Configurations, the one for the "Java Admin Client" is working fine, but I the one for "Jetty Server" still needs some tweaks... I will fix that soon, unless you get to it first ;-)

For those configurations the data dir will be located in exist-distribution/target/data and the config files come from exist-distribution/src/main/config

[adamretter]

@dizzzz on my very overheating and slow laptop, an mvn compile -Ddependency-check.skip=true -DskipTests takes 02:43 min of which exist-core takes 60% of that time:

Or an mvn -o compile -Ddependency-check.skip=true -DskipTests takes just 01:54 min

[duncdrum]

@adamretter
macOS 10.14.4
IntelliJ 2018.3.6 (Ultimate)
Atom: 1.36.1
Maven: 3.6.1

as i said earlier, i have configured intellij to use java8 and to use my own maven (s.a.)
the error occurs both when using intellij and when building from atom.

[wolfgangmm]

So the main problem seems to be on both of my macs, that the SetFile binary referenced from create-dmg-mac.sh is located in /usr/bin, not /Applications/Xcode.app/Contents/Developer/Tools/. After changing the path, the build succeeds! Could you guys check if you have /usr/bin/SetFile, which looks like the more generic path to me?

macOS 10.14.3
Maven 3.6.0

[adamretter]

@wolfgang Do you have XCode developer command line tools installed? I'm thinking maybe not

…

On Sat, 27 Apr 2019, 17:52 Wolfgang Meier, ***@***.***> wrote: So one of the problems seems to be on my machine, that the SetFile binary referenced from create-dmg-mac.sh is located in /usr/bin, not /Applications/Xcode.app/Contents/Developer/Tools/. macOS 10.14.3 Maven 3.6.0 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2657 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJUTOOVPBKCAO5DRQWTOG3PSQO6LANCNFSM4HIM2WCQ> .

[adamretter]

@duncdrum why is Atom involved?

…

On Sat, 27 Apr 2019, 14:54 Duncan Paterson, ***@***.***> wrote: @adamretter <https://github.com/adamretter> macOS 10.14.4 IntelliJ 2018.3.6 (Ultimate) Atom: 1.36.1 Maven: 3.6.1 as i said earlier, i have configured intellij to use java8 and to use my own maven (s.a.) the error occurs both when using intellij and when building from atom. [image: Screenshot 2019-04-27 at 08 51 16] <https://user-images.githubusercontent.com/6205362/56846015-e15a0880-68c9-11e9-87e2-af7a953fe402.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2657 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJUTOLVWOKFIJ5IVLF745LPSP2DDANCNFSM4HIM2WCQ> .

[duncdrum]

@adamretter to see if this is just an intellij bug or not, the build should work with any IDE or reasonably modern shell.

[wolfgangmm]

@adamretter I have XCode developer command line tools on all machines. They install into /usr/bin.

Two more small issues. Writing them down before I forget:

1. the counter module which was previously enabled by default is no longer included in the distribution. This is a frequently used module.
2. jar files shipping with expath packages do not seem to be picked up by the classloader. To test, try

import module namespace console="http://exist-db.org/xquery/console";
console:log("Hello world!")

[adamretter]

@adamretter I have XCode developer command line tools on all machines. They install into /usr/bin.

@wolfgangmm I have not changed the path where we look for SetFile at! It is the same as when we used Ant to build the DMG, see here: https://github.com/eXist-db/exist/blob/eXist-4.6.1/build/scripts/macosx.xml#L316

Were you able to build DMGs in the past with Ant? If so I think something must have changed with your machine.

If I look on my machine I see three things:

$ ls -la /Applications/Xcode.app/Contents/Developer/Tools/SetFile
lrwxr-xr-x  1 root  wheel  18 20 Sep  2017 /Applications/Xcode.app/Contents/Developer/Tools/SetFile -> ../usr/bin/SetFile

$ ls -la /Applications/Xcode.app/Contents/Developer/usr/bin/SetFile
-rwxr-xr-x  1 root  wheel  51536  6 Apr 08:34 /Applications/Xcode.app/Contents/Developer/usr/bin/SetFile

And then also the file you describe:

$ ls -la /usr/bin/SetFile
-rwxr-xr-x  1 root  wheel  18304 21 Mar 14:11 /usr/bin/SetFile

According to the MD5 sums, /Applications/Xcode.app/Contents/Developer/usr/bin/SetFile is not the same file as /usr/bin/SetFile.

Do you also have XCode.app installed? or just the command line developer tools?

[adamretter]

@wolfgangmm

1. the counter module which was previously enabled by default is no longer included in the distribution. This is a frequently used module.

Interesting. I actually hadn't myself seen anyone use it, a quick Google and I found one result which is for TEI-Publisher - https://gitlab.existsolutions.com/tei-publisher/tei-publisher-lib/commit/8d6089452448044811a3f97e8a7c5a8a965b8655.
I have just pushed a commit for you which re-enables it as part of the distribution :-)

2. jar files shipping with expath packages do not seem to be picked up by the classloader.

Hmm I hadn't seen that, let me take a look and get back to you... Just wondering - how are you testing this? You reported so far that you hadn't been able to build with Maven...

[wolfgangmm]

@adamretter Linking /Applications/Xcode.app/Contents/Developer/usr/bin/SetFile to /usr/bin/SetFile resulted in the build working for me. I had XCode on my old laptop, but on the new one and my imac I only have the developer tools, no XCode. For generating the dmg it does not seem to matter which SetFile is used. In this case I would prefer users to not have to install XCode, which weighs several gigabyte.

So with this little change in place, I have been able to build with maven on all three machines :-) I tested running from the generated distribution directory and debug in intellij.

[dizzzz]

IMO ... OK to go ! @adamretter @wolfgangmm @ljo @duncdrum @joewiz

[wolfgangmm]

Also OK unless someone else has objections.

[adamretter]

@dizzzz @wolfgangmm Okay by me :-)

[duncdrum]

We should open issues for the excluded tests, and get to the bottom of this. But this can happen after merge to develop.

[joewiz]

No objections!