Add a slim version of the Corretto JDK 15 Docker images. #38

luketn · 2020-10-24T02:48:17Z

Issue #34

Description of changes:
Add a slim version of the Corretto JDK 15 Docker images.

JLink is used (retaining all modules) to create a slimmer version of the JDK excluding man-pages, header files and debugging symbols.

The size saved is significant:

Uncompressed:
Alpine Linux: 175 MB (183 MB smaller)

Amazon Linux 2: 357 MB (113 MB smaller)

The compressed sizes when pushed to DockerHub are pretty good too.
Alpine Linux: 58 MB (140 MB smaller)
Amazon Linux 2: 123 MB (83 MB smaller)

Ref:
https://hub.docker.com/r/luketn/amazoncorretto/tags
https://hub.docker.com/_/amazoncorretto?tab=tags&page=1&name=15-al&ordering=last_updated

Note that the objcopy linux tool is required for jlink to run, without which the error will occur:
"java.io.IOException: Cannot run program “objcopy”: error=2, No such file or directory"
Ref: https://command-not-found.com/objcopy

Also note I have not attempted to update any of the Travis build files - only the proposed new Dockerfiles are added with this PR, meaning further work would be required to integrate them into the CI/CD pipeline.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

JLink is used (retaining all modules) to create a slimmer version of the JDK excluding man-pages, header files and debugging symbols - saving ~113MB. Note that the objcopy linux tool is required for jlink to run, without which the error will occur: "java.io.IOException: Cannot run program “objcopy”: error=2, No such file or directory" Ref: https://command-not-found.com/objcopy

joshenders

Some minor comments on the Debian Dockerfile.

joshenders · 2020-12-24T03:39:01Z

15/slim/debian/Dockerfile

+#
+# Slim:
+#   JLink is used (retaining all modules) to create a slimmer version of the JDK excluding man-pages, header files and debugging symbols - saving ~113MB.
+RUN set -eux \


I believe your boolean ANDs obviates -e usage. In my testing, the build fails with or without -e when any command in the chain fails.

Done. I also tested a failing command in the chain and saw the build fail.

joshenders · 2020-12-24T03:39:53Z

15/slim/debian/Dockerfile

+    && apt-get update \
+    && apt-get install -y --no-install-recommends \
+        curl ca-certificates gnupg software-properties-common fontconfig java-common \
+    && curl -fL https://apt.corretto.aws/corretto.key | apt-key add - \


I am aware that this is the official method recommended by Amazon however, this adds the key in manner which grants it validity over any software repository on the system, including those not distributed by Amazon. I understand this is quite pedantic but for illustrative purposes, the most correct and idiomatic way to add repositories on Debian is to add the following two apt config files:

/etc/apt/sources.list.d/apt.corretto.aws.list

/etc/apt/preferences.d/apt.corretto.aws.pref

And a gpg de-armored key (gpg --dearmor) here: /usr/share/keyrings/apt.corretto.aws.gpg.

apt.corretto.aws.list can be created with:

echo 'deb [signed-by=/usr/share/keyrings/apt.corretto.aws.gpg] https://apt.corretto.aws stable main' | tee /etc/apt/sources.list.d/apt.corretto.aws.list

The contents of apt.corretto.aws.pref should look like this:

Package: * Pin: origin apt.corretto.aws Pin-Priority: 1 Package: java-1.8.0-amazon-corretto-jdk Pin: origin apt.corretto.aws Pin-Priority: 500 Package: java-11-amazon-corretto-jdk Pin: origin apt.corretto.aws Pin-Priority: 500 Package: java-15-amazon-corretto-jdk Pin: origin apt.corretto.aws Pin-Priority: 500

This method also avoids pulling in software-properties-common for apt-key.

See:

https://wiki.debian.org/DebianRepository/UseThirdParty

https://unix.stackexchange.com/questions/463072/why-does-the-apt-key-man-page-advise-against-using-its-add-command

java-common can probably be removed as well. This is pulled in automatically as a dependency to corretto

Removed the unnecessary java-common apt-get install.

Regarding the apt-get install, I struggled with this one. Could I suggest that this update be made as a separate PR to both this slim Dockerfile and the regular JDK Dockerfile by someone with better expertise in the Debian OS?

I attempted to follow the instructions in the given reference documentation however saw errors with GPG validation during apt-get that I do not feel sufficiently well versed in Debian to address. I'm concerned that if I push through and implement something that I will inadvertently introduce bugs that would not be present using the standard install process.

Edited my comment for clarity and future reference but I think your comment makes perfect sense. This policy should really be reviewed for all of these Debian derived containers and updated in a separate PR along with a corresponding update to the official documentation.

joshenders · 2020-12-24T03:40:11Z

15/slim/debian/Dockerfile

+    && mkdir -p /usr/share/man/man1 || true \
+    && apt-get update \
+    && apt-get install -y java-15-amazon-corretto-jdk=1:$version \
+    && apt-get install -y binutils-i586-linux-gnu \


Why binutils-i586-linux-gnu and not binutils-i686-linux-gnu or just binutils? Also why isn't this grouped into the same invocation with the line above it? It could also be moved into the earlier apt-get install for clarity.

Great point - grouped the install into the previous apt-get install, and used the more generic 'binutils' package.
As a benefit I no longer need to symbolic link the objcopy in the line below as binutils does this.

joshenders

Some more comments on the Debian Dockerfile after playing with it a bit more locally

joshenders · 2020-12-24T04:56:06Z

15/slim/debian/Dockerfile

+        curl ca-certificates gnupg software-properties-common fontconfig java-common \
+    && curl -fL https://apt.corretto.aws/corretto.key | apt-key add - \
+    && add-apt-repository 'deb https://apt.corretto.aws stable main' \
+    && mkdir -p /usr/share/man/man1 || true \


|| true is unnecessary here. -p returns a zero exits status on failure to create the directory or intermediate path if it exists. (You wouldn't really want to hide this error even if it were necessary though...)

$ mkdir -p /tmp/test/it && mkdir -p /tmp/test/it $ echo $? 0

joshenders · 2020-12-24T04:57:15Z

15/slim/debian/Dockerfile

+    && apt-get update \
+    && apt-get install -y java-15-amazon-corretto-jdk=1:$version \
+    && apt-get install -y binutils-i586-linux-gnu \
+    && ln -s /usr/bin/i686-linux-gnu-objcopy /usr/bin/objcopy \


In my testing this is already symlink'd to the correct objcopy. This may be unnecessary.

:) saw that too after switching to binutils.

joshenders · 2020-12-24T04:58:03Z

15/slim/debian/Dockerfile

+    && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
+            curl gnupg software-properties-common \


Would it be better to move this to the end and include binutils as well? Additionally, does it make sense to also: apt-get clean --assume-yes && rm -rf /var/lib/apt/lists/*

Great idea. I moved to the end and added binutils to the removed packages.
Removing binutils reduced the final image size by 15MB (from 268MB to 253MB).

joshenders · 2020-12-24T05:04:00Z

15/slim/debian/Dockerfile

+    && for line in $(java --list-modules); do echo -n "${line%@15}," >> modules-comma-separated.txt; done \
+    && jlink --add-modules "$(cat modules-comma-separated.txt)" --no-man-pages --no-header-files --strip-debug --output /opt/corretto-slim \


This could be rewritten as the following single line without any temporary files:

jlink --add-modules "$(java --list-modules | sed -e 's/@[0-9].*$/,/' | tr -d \\n)" --no-man-pages --no-header-files --strip-debug --output /opt/corretto-slim

Nice! Much cleaner. Also handles the newer minor version suffixes :). Updated on the other Dockerfiles as well.

joshenders · 2020-12-24T05:34:43Z

Forgot to mention, thanks for your effort here @luketn 😃. Since it will be used by many as a base image, I just want to make sure it's the highest possible quality before it gets merged.

@15

…ion suffix on the modules can include a minor version. Since the last build the module suffix changed from "@15" to "@15.0.1". The sed command will handle any minor version following "@15"

… jlink to build slim JDK)

…bstitution using sed and tr.

luketn · 2020-12-29T04:06:48Z

Thanks for the awesome review @joshenders!

I've updated based on all of the comments, addressing all the issues with the exception of the GPG apt-get change which is a bit beyond my Debian skills.

If you do merge this PR, would you also add to the automation code to build and publish these new Dockerfiles to DockerHub? I'm happy to help make changes there too if needed, but felt that might be outside of the scope of a new contributor.

luketn · 2020-12-29T06:43:57Z

@joshenders - the review also managed to reduce the size of the image further as well (due to uninstalling binutils at the end - good thinking!).
e.g. the Alpine Linux compressed image is now down by 2.31MB (reduced from 58.19 MB to 55.88 MB):
https://hub.docker.com/layers/luketn/amazoncorretto/15-alpine-jdk-slim/images/sha256-88621b6a1b08cb4b2d50a7127ec081abb927f076d5e2232ccfe8105c3128b44a?context=explore

I also pushed all three Linux flavours to my personal DockerHub in case you'd like to see what the finished images would look like:
https://hub.docker.com/r/luketn/amazoncorretto/tags?page=1&ordering=last_updated

LS80 · 2021-01-04T10:43:39Z

@luketn and @joshenders Thanks for working on this! Do you intend to add a slim version of the JDK 11 images also?

luketn · 2021-01-05T01:07:05Z

@luketn and @joshenders Thanks for working on this! Do you intend to add a slim version of the JDK 11 images also?

There actually already was a JDK 11 'JRE' Dockerfile in this repo (only for Alpine Linux). It appears not to have been published to DockerHub though.

I think it would be great if we could agree on a standard approach for a slim/jre build of all Corretto 11+ distributions on all Linux flavours.

cliveverghese · 2021-01-05T20:48:08Z

Thank you for your feedback @luketn and @joshenders, We appreciate the interest in Corretto Slim images for Corretto-11 and above. However, The approach of using JLink was something that was previously discouraged by Dockerhub team..

The recommendation by joshenders is also valuable and we would like to incorporate this into the debian based docker images. If you have a Pull Request for that Please feel free to submit the same.

luketn · 2021-01-14T04:53:55Z

Thank you for your feedback @luketn and @joshenders, We appreciate the interest in Corretto Slim images for Corretto-11 and above. However, The approach of using JLink was something that was previously discouraged by Dockerhub team..

@cliveverghese I read through the comments by Dockerhub team. I think you're referring to this comment?:

I would think that if you want to publish a JRE image, you would probably want to also publish an "official" JRE package, like the AdoptOpenJDK community does, right?

The comment from the Dockerhub team doesn't seem to be counselling against creating a slim/JRE build - only proposing to expand it beyond just being a docker image to being an official separate package. I can't comment whether that would be a possibility, however I think that the size of the Corretto JDK package and the size of the Corretto docker image have different implications.

Since Docker uses layered image caching, you couldn't use the full Corretto JDK as a base image to construct a slim/JRE image. You would essentially have to start over and build from a base Linux image. The Dockerhub team acknowledge this in the next comment they make:

(Although my preference is still pretty strongly in favor of creating proper separate JRE packages, especially given how poorly multi-stage builds like this will interact with the Docker build cache.)

I think a compact version of the Corretto docker image would have a major impact on the Corretto JDK community. It would save a huge volume of unnecessary bytes being copied from disk into memory and over the network millions of times over.

My personal use-case as an example:
I am building Java services based on the Corretto 15 Amazon Linux 2 base image to deploy as Fargate tasks/services. I like the support implications of using Amazon Linux 2 within the AWS environment, and similarly the support implications of using the Corretto JDK too. I would like the base image to be as small as possible so that my services startup fast and use the least resources whilst doing so. Most of the size would come from the OS and JDK layers. Only a small percentage of the size comes from my service code. e.g. If my service was a 10MB jar:

amazonlinux:2 - 163MB (34%)
amazoncorretto:15-al2-jdk - 307MB (64%)
someservice:1 - 10MB (2%)

With the slim docker image we could save 45% of the Corretto layer, reducing it from 307MB to 170MB.

I could imagine a huge number of use-cases like mine which could benefit from a smaller distribution size.

If there is a possibility of reducing the size, either through this PR's approach or through another approach, I'd urge you to consider it.

cliveverghese · 2021-01-16T00:33:30Z

We understand the use case and are would like to merge the changes, However, would you consider the following changes to the pull request before proceeding

Could you remove the Amazon Corretto using the package manager command instead of rm so that the package manager is aware that the package is no longer installed.
Could you also update the test case at verify-images.yml to include the new images?

luketn · 2021-01-17T02:05:48Z

Thanks @cliveverghese - that's awesome!

Re: the two points:

Could you remove the Amazon Corretto using the package manager command instead of rm so that the package manager is aware that the package is no longer installed.

This is a little complex. I tried doing this (just swapping rm for yum remove), but it breaks the image. At the moment, we install the JDK with yum and then swap out the JDK directory with the jlink'd version. All the other configuration done by yum stays in place. However, when I tried removing the package with yum, the system loses the configured java context so java is no longer on the path and configured with alternatives etc... I could go through the yum installer and replicate whatever it was doing by hand if needed - otherwise would you consider leaving it as-is? To be honest I'm not sure what else yum / apt does when the JDK installer runs - it might not be much - I'll look into it.

Could you also update the test case at verify-images.yml to include the new images?

Done. I've run the GitHub actions workflows on my fork and they are all passing. See results here:
https://github.com/luketn/corretto-docker/actions/runs/490921998

luketn · 2021-01-18T12:51:02Z

@cliveverghese - further on point 1:

Could you remove the Amazon Corretto using the package manager command instead of rm so that the package manager is aware that the package is no longer installed.

I took a look at the RPM and DEB package builds in the corretto-jdk repository. It looks as though the post install scripts use the alternatives command to register Java symbolic links in /usr/bin:
RPM
https://github.com/corretto/corretto-jdk/blob/develop-jdk15/installers/linux/universal/rpm/scripts/postin_java.sh.template
and
https://github.com/corretto/corretto-jdk/blob/develop-jdk15/installers/linux/universal/rpm/scripts/postin_javac.sh.template
DEB
https://github.com/corretto/corretto-jdk/blob/develop-jdk15/installers/linux/universal/deb/scripts/postin_jdk.sh.template

I think we could register the new jlink'd slim JDK directory with alternatives / update-alternatives again after removing the original installed Corretto JDK.

This would also have the advantage of not setting man pages which don't exist any more following the jlink too - these would otherwise be dangling symbolic links.

I'll have a go at replicating these alternatives scripts in the Dockerfile for each OS.

cliveverghese · 2021-01-19T23:32:38Z

.github/workflows/verify-images.yml

@@ -8,13 +8,20 @@ jobs:
    runs-on: ubuntu-latest
    strategy:
      matrix:
-        version: [8, 11, 15]
+        version-package: [8-jdk, 11-jdk, 15-jdk, 15-slim]


https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#example-including-additional-values-into-combinations

version: [8,11,15] package: [jdk] include: - version: 15 package: [slim]

Nice! I've included the JRE 8 verification using the same include syntax, which cleans up the build a bit more too.

mattnelson · 2021-01-21T20:51:37Z

15/slim/alpine/Dockerfile

+    echo "${SHA_SUM}  /etc/apk/keys/amazoncorretto.rsa.pub" | sha256sum -c - && \
+    echo "https://apk.corretto.aws" >> /etc/apk/repositories && \
+    apk add --no-cache amazon-corretto-15=$version-r0 binutils && \
+    /usr/lib/jvm/default-jvm/bin/jlink --add-modules "$(java --list-modules | sed -e 's/@[0-9].*$/,/' | tr -d \\n)" --no-man-pages --no-header-files --strip-debug --output /opt/corretto-slim && \


WARNING: Using incubator modules: jdk.incubator.foreign, jdk.incubator.jpackage

Should these be filtered out? Incubating features must be opted into with the compiler, so it seems like they should also be opted into for your runtime image.

Suggested change

/usr/lib/jvm/default-jvm/bin/jlink --add-modules "$(java --list-modules | sed -e 's/@[0-9].*$/,/' | tr -d \\n)" --no-man-pages --no-header-files --strip-debug --output /opt/corretto-slim && \

/usr/lib/jvm/default-jvm/bin/jlink --add-modules "$(java --list-modules | grep -v incubator | sed -e 's/@[0-9].*$/,/' | tr -d \\n)" --no-man-pages --no-header-files --strip-debug --output /opt/corretto-slim && \

I would recommend keeping these incubator modules in the slim JDK image. They are in the full JDK image too and I don't think they do any harm to the runtime of applications which do not use them. I could imagine some use-cases for them, particularly the foreign memory mapping one. I think one of the benefits of keeping all modules is that we're not imposing any opinion on what should or shouldn't be included in the JDK - only reducing the size by omitting files required only at development time.

cliveverghese · 2021-01-22T21:03:30Z

Hi,
I have merged this PR as part of #50.
You can view the commit at bdbc4c3

Thank you for your effort in adding Slim images for Corretto-15.

joshenders reviewed Dec 24, 2020

View reviewed changes

luketn added 9 commits December 29, 2020 12:23

Merge remote-tracking branch 'upstream/master'

5b1f9ae

Update the arg versions to the latest values matching the jdk versions

8584188

Change from a straight bash replace to sed with regex as the JVM vers…

7dfe527

…ion suffix on the modules can include a minor version. Since the last build the module suffix changed from "@15" to "@15.0.1". The sed command will handle any minor version following "@15"

Remove the unnecessary -e from the RUN statement.

e1aa25b

Remove the unnecessary java-common install.

0ec4d4c

Use the more generic binutils package to install objcopy (required by…

293b8df

… jlink to build slim JDK)

cleanup unnecessary || true

1b108fa

Add binutils to the apt-get purge and move the purge to the final step.

3c0989c

Switch to a single-line jlink statement embedding the list-modules su…

1caa60f

…bstitution using sed and tr.

luketn added 3 commits January 17, 2021 12:49

Add capability to verify new docker images in GitHub actions.

1bf06e0

Fix order and missing }

a90f8d8

Fix step name

3e1183a

cliveverghese reviewed Jan 19, 2021

View reviewed changes

mattnelson reviewed Jan 21, 2021

View reviewed changes

luketn added 3 commits January 22, 2021 22:52

Use a better matrix approach

a9699cf

Fix the YAML syntax

1d9231c

Fix the YAML syntaxπ

c87bbfa

luketn added 2 commits January 22, 2021 23:17

Be specific about the platforms supported by slim and jre builds

113e437

Use a non-array syntax for variables included in the matrix.

0f61e8f

cliveverghese mentioned this pull request Jan 22, 2021

Update slim images to delete package using repo manager #50

Merged

cliveverghese closed this Jan 22, 2021

mattnelson mentioned this pull request Jan 22, 2021

Add JRE 15 alpine tag #34

Closed

luketn mentioned this pull request Feb 25, 2021

Publishing the slim version of the Corretto JDK 17 & 18 Docker images #51

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a slim version of the Corretto JDK 15 Docker images. #38

Add a slim version of the Corretto JDK 15 Docker images. #38

luketn commented Oct 24, 2020 •

edited

Loading

joshenders left a comment

joshenders Dec 24, 2020

luketn Dec 29, 2020

joshenders Dec 24, 2020 •

edited

Loading

joshenders Dec 24, 2020

luketn Dec 29, 2020

luketn Dec 29, 2020

joshenders Dec 29, 2020 •

edited

Loading

joshenders Dec 24, 2020 •

edited

Loading

luketn Dec 29, 2020

joshenders left a comment

joshenders Dec 24, 2020 •

edited

Loading

luketn Dec 29, 2020

joshenders Dec 24, 2020

luketn Dec 29, 2020

joshenders Dec 24, 2020 •

edited

Loading

luketn Dec 29, 2020

joshenders Dec 24, 2020 •

edited

Loading

luketn Dec 29, 2020

joshenders commented Dec 24, 2020 •

edited

Loading

luketn commented Dec 29, 2020

luketn commented Dec 29, 2020 •

edited

Loading

LS80 commented Jan 4, 2021

luketn commented Jan 5, 2021

cliveverghese commented Jan 5, 2021

luketn commented Jan 14, 2021

cliveverghese commented Jan 16, 2021

luketn commented Jan 17, 2021

luketn commented Jan 18, 2021

cliveverghese Jan 19, 2021

luketn Jan 22, 2021

mattnelson Jan 21, 2021

luketn Jan 22, 2021

cliveverghese commented Jan 22, 2021

		&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
		curl gnupg software-properties-common \

		&& for line in $(java --list-modules); do echo -n "${line%@15}," >> modules-comma-separated.txt; done \
		&& jlink --add-modules "$(cat modules-comma-separated.txt)" --no-man-pages --no-header-files --strip-debug --output /opt/corretto-slim \

	/usr/lib/jvm/default-jvm/bin/jlink --add-modules "$(java --list-modules \| sed -e 's/@[0-9].*$/,/' \| tr -d \\n)" --no-man-pages --no-header-files --strip-debug --output /opt/corretto-slim && \
	/usr/lib/jvm/default-jvm/bin/jlink --add-modules "$(java --list-modules \| grep -v incubator \| sed -e 's/@[0-9].*$/,/' \| tr -d \\n)" --no-man-pages --no-header-files --strip-debug --output /opt/corretto-slim && \

Add a slim version of the Corretto JDK 15 Docker images. #38

Add a slim version of the Corretto JDK 15 Docker images. #38

Conversation

luketn commented Oct 24, 2020 • edited Loading

joshenders left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshenders Dec 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshenders Dec 29, 2020 • edited Loading

Choose a reason for hiding this comment

joshenders Dec 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshenders left a comment

Choose a reason for hiding this comment

joshenders Dec 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshenders Dec 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshenders Dec 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshenders commented Dec 24, 2020 • edited Loading

luketn commented Dec 29, 2020

luketn commented Dec 29, 2020 • edited Loading

LS80 commented Jan 4, 2021

luketn commented Jan 5, 2021

cliveverghese commented Jan 5, 2021

luketn commented Jan 14, 2021

cliveverghese commented Jan 16, 2021

luketn commented Jan 17, 2021

luketn commented Jan 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cliveverghese commented Jan 22, 2021

luketn commented Oct 24, 2020 •

edited

Loading

joshenders Dec 24, 2020 •

edited

Loading

joshenders Dec 29, 2020 •

edited

Loading

joshenders Dec 24, 2020 •

edited

Loading

joshenders Dec 24, 2020 •

edited

Loading

joshenders Dec 24, 2020 •

edited

Loading

joshenders Dec 24, 2020 •

edited

Loading

joshenders commented Dec 24, 2020 •

edited

Loading

luketn commented Dec 29, 2020 •

edited

Loading