Auto-generate README with tag information (#192) (#199)

* Auto-generate README with tag information (#192)

* Remove VERSION_FORWARDER file.
Update version.yml to latest 4.0 release.

* Update docker/forwarder/hooks/build

Co-authored-by: Othello Maurer <othello@graylog.com>

* Update version.

* Update README.

* Fix forwarder version.

* Fix forwarder version in README.

Co-authored-by: Othello Maurer <othello@graylog.com>

README.j2

@@ -0,0 +1,97 @@
+# Graylog Docker Image
+
+[![Docker Stars](https://img.shields.io/docker/stars/graylog/graylog.svg)][hub] [![Docker Pulls](https://img.shields.io/docker/pulls/graylog/graylog.svg)][hub]
+
+[hub]: https://hub.docker.com/r/graylog/graylog/
+
+The latest stable version of Graylog is **`{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}`**.
+
+## What is Graylog?
+
+Graylog is a centralized logging solution that enables aggregating and searching through logs. It provides a powerful query language, a processing pipeline for data transformation, alerting abilities, and much more. It is fully extensible through a REST API. Add-ons can be downloaded from the [Graylog Marketplace](https://marketplace.graylog.org/).
+
+
+## Image Details
+
+There are several different image variants available, with variants for Java 8 and 11 on platforms `linux/amd64` and `linux/arm64`. All images run on Debian 11.
+
+#### `graylog/graylog`
+
+This is the open source [Graylog ](https://hub.docker.com/r/graylog/graylog/) image. It contains [Graylog](https://github.com/Graylog2/graylog2-server) as well as the [Integrations](https://docs.graylog.org/docs/integrations) plugin.
+
+| Java Version  | Platform  | Tags  |
+|---|---|---|
+| OpenJDK 11  | `linux/amd64`, `linux/arm64`  | `{{ graylog.major_version }}.{{ graylog.minor_version }}-jre11`, `{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}-jre11`, `{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}-{{ graylog.release }}-jre11`  |
+| OpenJDK 8  | `linux/amd64` | `{{ graylog.major_version }}.{{ graylog.minor_version }}`, `{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}`, `{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}-{{ graylog.release }}` |
+| OpenJDK 8  | `linux/arm64` | `{{ graylog.major_version }}.{{ graylog.minor_version }}-arm64`, `{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}-arm64`, `{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}-{{ graylog.release }}-arm64` |
+
+
+> Note: There is no 'latest' tag. You'll need to specify which version you want.
+
+#### `graylog/graylog-enterprise`
+
+This is the [Graylog Enterprise](https://hub.docker.com/r/graylog/graylog-enterprise/) image. It contains [Graylog](https://github.com/Graylog2/graylog2-server), the [Graylog Enterprise](https://docs.graylog.org/docs/intro) plugin, the [Integrations](https://docs.graylog.org/docs/integrations) plugin, and the Enterprise Integrations plugin.
+
+| Java Version  | Platform  | Tags  |
+|---|---|---|
+| OpenJDK 11  | `linux/amd64`, `linux/arm64`  | `{{ graylog.major_version }}.{{ graylog.minor_version }}-jre11`, `{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}-jre11`, `{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}-{{ graylog.release }}-jre11`  |
+| OpenJDK 8  | `linux/amd64` | `{{ graylog.major_version }}.{{ graylog.minor_version }}`, `{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}`, `{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}-{{ graylog.release }}` |
+| OpenJDK 8  | `linux/arm64` | `{{ graylog.major_version }}.{{ graylog.minor_version }}-arm64`, `{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}-arm64`, `{{ graylog.major_version }}.{{ graylog.minor_version }}.{{ graylog.patch_version }}-{{ graylog.release }}-arm64` |
+
+
+#### `graylog/graylog-forwarder`
+
+This image runs the [Graylog Forwarder](https://hub.docker.com/r/graylog/graylog-forwarder/). Documentation on the Forwarder can be found [here](https://docs.graylog.org/docs/forwarder).
+
+The latest stable version is **`{{ forwarder.version }}`**, with support for Java 8 on platform `linux/amd64`.
+
+| Java Version  | Platform  | Tags  |
+|---|---|---|
+| OpenJDK 8 | `linux/amd64` | `{{ forwarder.version }}`, `forwarder-{{ forwarder.version }}-{{ forwarder.release }}` |
+
+
+## Architecture
+
+Take a look at the minimal [Graylog architecture](https://docs.graylog.org/docs/architecture) to get the big picture of a Graylog setup. In essence, Graylog needs to talk to MongoDB to store configuration data as well as Elasticsearch to store the actual log data.
+
+
+## Configuration
+
+Please refer to the [Graylog Docker documentation](https://docs.graylog.org/docs/docker) for a comprehensive overview and detailed description of the Graylog Docker image.
+
+If you want to quickly spin up an instance for testing, you can use our [Docker Compose template](https://github.com/Graylog2/docker-compose).
+
+Notably, this image **requires** that two important configuration options be set (although in practice you will likely need to set more):
+1. `password_secret` (environment variable `GRAYLOG_PASSWORD_SECRET`)
+    * A secret that is used for password encryption and salting.
+    * Must be at least 16 characters, however using at least 64 characters is strongly recommended.
+    * Must be the same on all Graylog nodes in the cluster.
+    * May be generated with something like: `pwgen -N 1 -s 96`
+2. `root_password_sha2` (environment variable `GRAYLOG_ROOT_PASSWORD_SHA2`)
+    * A SHA2 hash of a password you will use for your initial login as Graylog's root user.
+      * The default username is `admin`.  This value is customizable via configuration option `root_username` (environment variable `GRAYLOG_ROOT_USERNAME`).
+    * In general, these credentials will only be needed to initially set up the system or reconfigure the system in the event of an authentication backend failure.
+    * This password cannot be changed using the API or via the Web interface.
+    * May be generated with something like: `echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1`
+
+
+Every [Graylog configuration option](https://docs.graylog.org/docs/server-conf) can be set via environment variable. To get the environment variable name for a given configuration option, simply prefix the option name with `GRAYLOG_` and put it all in upper case. Another option is to store the configuration file outside of the container and edit it directly.
+
+This image includes the [wait-for-it](https://github.com/vishnubob/wait-for-it) script, which allows you to have Docker wait for Elasticsearch to start up before starting Graylog. For example, if you are using Docker Compose you could override the entrypoint for Graylog like this:
+
+`entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh`
+
+
+
+
+
+
+## Documentation
+
+Documentation for Graylog is hosted [here](https://docs.graylog.org/). Please read through the docs and familiarize yourself with the functionality before opening an [issue on GitHub](https://github.com/Graylog2/graylog2-server/issues).
+
+## License
+
+Graylog itself is licensed under the Server Side Public License (SSPL), see [license information](https://www.mongodb.com/licensing/server-side-public-license).
+
+This Docker image is licensed under the Apache 2.0 license, see [LICENSE](LICENSE).

README.md

@@ -1,41 +1,97 @@
 # Graylog Docker Image
 
-Latest stable version of Graylog is *4.0.15* this Version is available with the tags `4.0` or `4.0.15-1`.
-
 [![Docker Stars](https://img.shields.io/docker/stars/graylog/graylog.svg)][hub] [![Docker Pulls](https://img.shields.io/docker/pulls/graylog/graylog.svg)][hub]
 
 [hub]: https://hub.docker.com/r/graylog/graylog/
 
-Use the stable `4.0` release for your production environments. Please check the [latest stable documentation](http://docs.graylog.org/en/4.0/pages/installation/docker.html) for complete installation and configuration instruction.
-
+The latest stable version of Graylog is **`4.0.15`**.
 
 ## What is Graylog?
 
-Graylog is a centralized logging solution that allows the user to aggregate and search through logs. It provides a powerful query language, a processing pipeline for data transformation, alerting abilities and much more. It is fully extensible through a REST API. Add-Ons can be downloaded from the [Graylog Marketplace](https://marketplace.graylog.org/).
+Graylog is a centralized logging solution that enables aggregating and searching through logs. It provides a powerful query language, a processing pipeline for data transformation, alerting abilities, and much more. It is fully extensible through a REST API. Add-ons can be downloaded from the [Graylog Marketplace](https://marketplace.graylog.org/).
+
+
+## Image Details
+
+There are several different image variants available, with variants for Java 8 and 11 on platforms `linux/amd64` and `linux/arm64`. All images run on Debian 11.
+
+#### `graylog/graylog`
+
+This is the open source [Graylog ](https://hub.docker.com/r/graylog/graylog/) image. It contains [Graylog](https://github.com/Graylog2/graylog2-server) as well as the [Integrations](https://docs.graylog.org/docs/integrations) plugin.
+
+| Java Version  | Platform  | Tags  |
+|---|---|---|
+| OpenJDK 11  | `linux/amd64`, `linux/arm64`  | `4.0-jre11`, `4.0.15-jre11`, `4.0.15-1-jre11`  |
+| OpenJDK 8  | `linux/amd64` | `4.0`, `4.0.15`, `4.0.15-1` |
+| OpenJDK 8  | `linux/arm64` | `4.0-arm64`, `4.0.15-arm64`, `4.0.15-1-arm64` |
+
+
+> Note: There is no 'latest' tag. You'll need to specify which version you want.
+
+#### `graylog/graylog-enterprise`
+
+This is the [Graylog Enterprise](https://hub.docker.com/r/graylog/graylog-enterprise/) image. It contains [Graylog](https://github.com/Graylog2/graylog2-server), the [Graylog Enterprise](https://docs.graylog.org/docs/intro) plugin, the [Integrations](https://docs.graylog.org/docs/integrations) plugin, and the Enterprise Integrations plugin.
+
+| Java Version  | Platform  | Tags  |
+|---|---|---|
+| OpenJDK 11  | `linux/amd64`, `linux/arm64`  | `4.0-jre11`, `4.0.15-jre11`, `4.0.15-1-jre11`  |
+| OpenJDK 8  | `linux/amd64` | `4.0`, `4.0.15`, `4.0.15-1` |
+| OpenJDK 8  | `linux/arm64` | `4.0-arm64`, `4.0.15-arm64`, `4.0.15-1-arm64` |
 
-## Architecture
 
-Take a look at the minimal [Graylog architecture](http://docs.graylog.org/en/4.0/pages/architecture.html) to get the big picture of a Graylog setup. In essence, Graylog needs to talk to MongoDB to store configuration data as well as Elasticsearch to store the actual log data.
+#### `graylog/graylog-forwarder`
 
-## How to use this image
+This image runs the [Graylog Forwarder](https://hub.docker.com/r/graylog/graylog-forwarder/). Documentation on the Forwarder can be found [here](https://docs.graylog.org/docs/forwarder).
+
+The latest stable version is **`4.0.6`**, with support for Java 8 on platform `linux/amd64`.
+
+| Java Version  | Platform  | Tags  |
+|---|---|---|
+| OpenJDK 8 | `linux/amd64` | `4.0.6`, `forwarder-4.0.6-20210326184330` |
+
+
+## Architecture
+
+Take a look at the minimal [Graylog architecture](https://docs.graylog.org/docs/architecture) to get the big picture of a Graylog setup. In essence, Graylog needs to talk to MongoDB to store configuration data as well as Elasticsearch to store the actual log data.
 
-Please refer to the [Graylog Docker documentation](http://docs.graylog.org/en/4.0/pages/installation/docker.html) for a comprehensive overview and a detailed description of the Graylog Docker image.
 
 ## Configuration
 
-Every configuration option can be set via environment variables, take a look [here](http://docs.graylog.org/en/4.0/pages/configuration/server.conf.html) for an overview. Simply prefix the parameter name with `GRAYLOG_` and put it all in upper case. Another option would be to store the configuration file outside of the container and edit it directly.
+Please refer to the [Graylog Docker documentation](https://docs.graylog.org/docs/docker) for a comprehensive overview and detailed description of the Graylog Docker image.
+
+If you want to quickly spin up an instance for testing, you can use our [Docker Compose template](https://github.com/Graylog2/docker-compose).
 
-We've also added the [wait-for-it](https://github.com/vishnubob/wait-for-it) script to the graylog image. This allows you to have Docker wait for Elasticsearch to start up before starting Graylog. For example, if you are using Docker Compose, you could override the entrypoint for Graylog like this:
+Notably, this image **requires** that two important configuration options be set (although in practice you will likely need to set more):
+1. `password_secret` (environment variable `GRAYLOG_PASSWORD_SECRET`)
+    * A secret that is used for password encryption and salting.
+    * Must be at least 16 characters, however using at least 64 characters is strongly recommended.
+    * Must be the same on all Graylog nodes in the cluster.
+    * May be generated with something like: `pwgen -N 1 -s 96`
+2. `root_password_sha2` (environment variable `GRAYLOG_ROOT_PASSWORD_SHA2`)
+    * A SHA2 hash of a password you will use for your initial login as Graylog's root user.
+      * The default username is `admin`.  This value is customizable via configuration option `root_username` (environment variable `GRAYLOG_ROOT_USERNAME`).
+    * In general, these credentials will only be needed to initially set up the system or reconfigure the system in the event of an authentication backend failure.
+    * This password cannot be changed using the API or via the Web interface.
+    * May be generated with something like: `echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1`
+
+
+Every [Graylog configuration option](https://docs.graylog.org/docs/server-conf) can be set via environment variable. To get the environment variable name for a given configuration option, simply prefix the option name with `GRAYLOG_` and put it all in upper case. Another option is to store the configuration file outside of the container and edit it directly.
+
+This image includes the [wait-for-it](https://github.com/vishnubob/wait-for-it) script, which allows you to have Docker wait for Elasticsearch to start up before starting Graylog. For example, if you are using Docker Compose you could override the entrypoint for Graylog like this:
 
 `entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh`
 
 
+
+
+
+
 ## Documentation
 
-Documentation for Graylog is hosted [here](http://docs.graylog.org/). Please read through the docs and familiarize yourself with the functionality before opening an [issue on GitHub](https://github.com/Graylog2/graylog2-server/issues).
+Documentation for Graylog is hosted [here](https://docs.graylog.org/). Please read through the docs and familiarize yourself with the functionality before opening an [issue on GitHub](https://github.com/Graylog2/graylog2-server/issues).
 
 ## License
 
 Graylog itself is licensed under the Server Side Public License (SSPL), see [license information](https://www.mongodb.com/licensing/server-side-public-license).
 
-This Docker image is licensed under the Apache 2.0 license, see [LICENSE](LICENSE).
+This Docker image is licensed under the Apache 2.0 license, see [LICENSE](LICENSE).

docker/enterprise/hooks/build

@@ -7,9 +7,12 @@ cd ../..
 
 patch config/graylog.conf patches/graylog-server.conf.patch
 
+apt-get install -y python3-pip
+pip3 install pyyaml
+
 # Build Graylog Enterprise & Integration Included
 docker build --build-arg VCS_REF=$GIT_SHA1 \
-             --build-arg GRAYLOG_VERSION=`cat VERSION` \
+             --build-arg GRAYLOG_VERSION=`./release.py --get-graylog-version` \
              --build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` \
              --file docker/enterprise/Dockerfile \
              --tag $IMAGE_NAME .

docker/forwarder/hooks/build

@@ -5,9 +5,12 @@
 
 cd ../..
 
+apt-get install -y python3-pip
+pip3 install pyyaml
+
 # Build Graylog Forwarder
 docker build --build-arg VCS_REF=$GIT_SHA1 \
-             --build-arg GRAYLOG_FORWARDER_VERSION=`cat VERSION_FORWARDER` \
+             --build-arg GRAYLOG_FORWARDER_VERSION=`./release.py --get-forwarder-version` \
              --build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` \
              --file docker/forwarder/Dockerfile \
              --tag $IMAGE_NAME .

docker/oss/hooks/build

@@ -5,9 +5,12 @@
 
 cd ../..
 
+apt-get install -y python3-pip
+pip3 install pyyaml
+
 # Build Graylog
 docker build --build-arg VCS_REF=$GIT_SHA1 \
-             --build-arg GRAYLOG_VERSION=`cat VERSION` \
+             --build-arg GRAYLOG_VERSION=`./release.py --get-graylog-version` \
              --build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` \
              --file docker/oss/Dockerfile \
              --tag $IMAGE_NAME .

release.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+
+import yaml
+import argparse
+import sys
+
+parser = argparse.ArgumentParser(description='Release utility for the Graylog Docker image.')
+parser.add_argument('--get-graylog-version', help="Get Graylog image version.", action='store_true')
+parser.add_argument('--get-forwarder-version', help="Get Forwarder image version.", action='store_true')
+parser.add_argument('--generate-readme', help="Generate a new README.md with the latest tags", action='store_true')
+
+if len(sys.argv)==1:
+    parser.print_help(sys.stderr)
+    sys.exit(1)
+
+args = parser.parse_args()
+
+with open('version.yml', 'r') as version_file:
+  version_parsed = yaml.safe_load(version_file)
+
+  if args.get_graylog_version:
+    print(str(version_parsed['graylog']['major_version']) + '.' + str(version_parsed['graylog']['minor_version']) + '.' + str(version_parsed['graylog']['patch_version']), end='')
+
+  if args.get_forwarder_version:
+    print(str(version_parsed['forwarder']['version']) + '-' + str(version_parsed['forwarder']['release']), end='')
+
+  if args.generate_readme:
+    from jinja2 import Template
+    with open('README.j2', 'r') as template_file:
+      j2_template = Template(template_file.read())
+
+    with open("README.md", "w") as readme_file:
+      readme_file.write(j2_template.render(version_parsed))

test/integration_test.sh

@@ -51,7 +51,7 @@ compose_up() {
 
   cat << EOF > .env
 VCS_REF=$(git rev-parse --short HEAD)
-GRAYLOG_VERSION=$(cat ${PWD}/../VERSION)
+GRAYLOG_VERSION=$(cd .. && ./release.py --get-graylog-version)
 EOF
 
   docker-compose --file docker-compose.tpl config  > ./docker-compose.yml

version.yml

@@ -0,0 +1,8 @@
+graylog:
+  major_version: 4
+  minor_version: 0
+  patch_version: 15
+  release: 1
+forwarder:
+  version: 4.0.6
+  release: 20210326184330