Abort build if shading fails

[sauvala]

Hi,

We faced an issue with shading step in our project. The 'fat' JAR building step generated a JAR, but it did not work correctly when executed because the shading step seemed to work only 'partially' with our dependency. It seemed that it updated the references to the dependency but did not then 'rename' the path to match the shading path so the dependency was not found during runtime. The problematic dependency in our case was com.fasterxml.jackson.module.

We noticed that in our build logs this was mentioned as:

[info] Including: jackson-core-2.15.3.jar
[info] Including: javax.ws.rs-api-2.0.1.jar
[info] Including: netty-common-4.1.22.Final.jar
Fully-qualified classname does not match jar entry:
  jar entry: META-INF/versions/9/module-info.classl 32s
  class name: module-info.classbly / assembledMappings 35s
Omitting META-INF/versions/9/module-info.class.
Unable to read classname from bytecode in META-INF/versions/11/com/fasterxml/jackson/core/io/doubleparser/BigSignificand.class
Shading is therefore impossible, so this entry will be skipped.
java.lang.IllegalArgumentException: null
Unable to read classname from bytecode in META-INF/versions/11/com/fasterxml/jackson/core/io/doubleparser/FastDoubleSwar.class
Shading is therefore impossible, so this entry will be skipped.
java.lang.IllegalArgumentException: null
Unable to read classname from bytecode in META-INF/versions/11/com/fasterxml/jackson/core/io/doubleparser/FastIntegerMath.class
Shading is therefore impossible, so this entry will be skipped.
java.lang.IllegalArgumentException: null
Unable to read classname from bytecode in META-INF/versions/11/com/fasterxml/jackson/core/io/doubleparser/FastIntegerMath$UInt128.class
Shading is therefore impossible, so this entry will be skipped.
java.lang.IllegalArgumentException: null
Unable to read classname from bytecode in META-INF/versions/17/com/fasterxml/jackson/core/io/doubleparser/FastDoubleSwar.class
Shading is therefore impossible, so this entry will be skipped.
java.lang.IllegalArgumentException: null
Unable to read classname from bytecode in META-INF/versions/17/com/fasterxml/jackson/core/io/doubleparser/FastIntegerMath.class
Shading is therefore impossible, so this entry will be skipped.
java.lang.IllegalArgumentException: null
Unable to read classname from bytecode in META-INF/versions/17/com/fasterxml/jackson/core/io/doubleparser/FastIntegerMath$UInt128.class
Shading is therefore impossible, so this entry will be skipped.
java.lang.IllegalArgumentException: null
Unable to read classname from bytecode in META-INF/versions/19/com/fasterxml/jackson/core/io/doubleparser/FastDoubleSwar.class
Shading is therefore impossible, so this entry will be skipped.
java.lang.IllegalArgumentException: null
Unable to read classname from bytecode in META-INF/versions/19/com/fasterxml/jackson/core/io/doubleparser/FastIntegerMath.class
Shading is therefore impossible, so this entry will be skipped.
java.lang.IllegalArgumentException: null
Unable to read classname from bytecode in META-INF/versions/19/com/fasterxml/jackson/core/io/doubleparser/FastIntegerMath$UInt128.class
Shading is therefore impossible, so this entry will be skipped.
java.lang.IllegalArgumentException: null
[info] Including: javax.annotation-api-1.2.jar
[info] Including: scala-java8-compat_2.11-0.7.0.jar
[info] Including: value-2.0.16.jar
[info] Including: spark-utils_2.11-0.0.173.jar
[info] Including: netty-codec-4.1.22.Final.jar
[info] Including: jackson-module-jaxb-annotations-2.6.3.jar
[info] Including: error_prone_annotations-2.1.2.jar

More details:

• We were previously using Jackson version 2.8.4 without any issues but when we upgraded it to 2.15.3 then this issue started to appear
• At that time we were using a quite old version of sbt-assembly: addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6"). When we updated it to 2.1.5, then the issue went away.
• We were thinking that the Jackson's multi-release could have triggered this https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15#jar-changes + we were using an older version of the sbt-assembly. Weird thing was, that the problem did not show immediately when we upgraded Jackson (with together an older version of sbt-assembly) but it showed up in a subsequent build which did introduced changes that did not change this part of our project.

I had a few questions I was hoping you could shine some light on:

1. Were we on the right track of thinking that the Jackson multi-release JAR introduced this problem and has there been fixes related to this in the new releases of sbt-assembly?

2. Is the shading a non-deterministic process, which could explain why we didn't get this problem on the first change, which upgraded Jackson, but only in the following changes/builds?

3. Is there something like a build flag that would fail the build step if the shading was not possible to do? We would like to have this so we could prevent us from deploying a 'broken' JAR.

I did my best to find answers to these questions from the issues and StackOverflow, but I failed to find something that would answer to my questions.

Thank you in advance.

Janne

[eed3si9n]

1. Were we on the right track of thinking that the Jackson multi-release JAR introduced this problem and has there been fixes related to this in the new releases of sbt-assembly?

That sounds like a good hypothesis because MR JARs contain additional class files.

2. Is the shading a non-deterministic process, which could explain why we didn't get this problem on the first change, which upgraded Jackson, but only in the following changes/builds?

I don't think shading itself is non-deterministic but sbt-assembly has assemblyRepeatableBuild, which can be set to false for performance.

3. Is there something like a build flag that would fail the build step if the shading was not possible to do? We would like to have this so we could prevent us from deploying a 'broken' JAR.

If the build isn't failing today, I'd say that's a bug, but I guess jarjar (Jar Jar Links or Jar Jar Abrams) was written a bit more laxed way? We need to look into this.