Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[ES-8436] feat: Change Dockerfile base images to Wolfi #1871

Merged
merged 13 commits into from
Sep 27, 2024
+145 −61
Merged

[ES-8436] feat: Change Dockerfile base images to Wolfi #1871

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
87adff7
feat: Change Dockerfile base images to Wolfi
favilo Aug 31, 2024
88fc6cd
fix: Update Dockerfiles to use elastic specific wolfi python images
favilo Sep 3, 2024
3d6670b
fix: Fix broken PATH environment in docker imaes.
favilo Sep 3, 2024
b6482e4
fix: Add zstd to docker image
favilo Sep 6, 2024
c8f8d41
feat: Move docker build to two stage build.
favilo Sep 6, 2024
50646bf
fix: Change back to user 1000 so we don't have to update documentation
favilo Sep 20, 2024
bbdb597
docs: add documentation for the new configuration file variables
favilo Sep 23, 2024
8eb3748
docs: use rst syntax, not markdown
favilo Sep 23, 2024
1503825
fix(docs): fix underlining typo
favilo Sep 23, 2024
a741f06
feat: update to 3.12.3 as the latest version of python we support.
favilo Sep 25, 2024
7da5ea5
fix: try to fix docker doc formatting
favilo Sep 26, 2024
5f11ce8
fix: docker docs cleanup formatting 2nd attempt
favilo Sep 26, 2024
0e2ce5b
I added double colon in the wrong place
favilo Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 31 additions & 39 deletions docker/Dockerfiles/Dockerfile-dev
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,51 +1,31 @@
################################################################################
# Build stage 0 `builder`:
# Install Rally from source inside a virtualenv
################################################################################
FROM docker.elastic.co/wolfi/python:3.11.7-dev AS builder

Copy link
Contributor

@gbanasiak gbanasiak Sep 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably a good opportunity to define the rule going forward. Will we keep this version aligned with the version of the nightlies (3.11.x), or perhaps we should bump it up all the way to the newest supported version (3.12.x)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I tried 3.12 and it wasn't playing nice, but I could also be remembering something else. I don't mind pinning it to the newest version we support.

FROM python:3.8.13-slim-bullseye as builder
USER root

RUN apt-get -y update && \
apt-get install -y curl git gcc && \
apt-get -y upgrade && \
rm -rf /var/lib/apt/lists/*
RUN apk update
RUN apk add curl git gcc pigz bash zstd bzip2 gzip

RUN mkdir -p /rally/esrally
COPY pyproject.toml /rally/
COPY README.md /rally/
COPY esrally/ /rally/esrally/

RUN python3 -m venv /rally/venv
ENV PATH="/rally/venv/bin:$PATH"

WORKDIR /rally
# Wipe away any lingering caches, copied over from the local machine
RUN find /rally -name "__pycache__" -exec rm -rf -- \{\} \; 2>/dev/null || true
RUN find /rally -name ".pyc" -exec rm -rf -- \{\} \; 2>/dev/null || true
RUN pip3 install --upgrade hatch hatchling pip wheel
RUN pip3 install /rally
# pbzip2 doesn't have a package for wolfi, so we build it from source
RUN apk add bzip2-dev make wget

################################################################################
# Build stage 1 (the actual Rally image):
# Copy Rally from stage 0 and fix permissions to support randomized UIDs
# Define VOLUME for ~/.rally
# Add entrypoint
################################################################################
RUN cd /tmp && \
wget -q https://launchpad.net/pbzip2/1.1/1.1.13/+download/pbzip2-1.1.13.tar.gz && \
tar -xzf pbzip2-1.1.13.tar.gz && \
cd pbzip2-1.1.13/ && \
make install && \
rm -r /tmp/pbzip2-1.1.13/

FROM python:3.8.12-slim-bullseye
FROM docker.elastic.co/wolfi/python:3.11.7-dev
ARG RALLY_VERSION
ARG RALLY_LICENSE
ENV RALLY_RUNNING_IN_DOCKER True

RUN apt-get -y update && \
apt-get install -y curl git pbzip2 pigz && \
apt-get -y upgrade && \
rm -rf /var/lib/apt/lists/*
ENV RALLY_RUNNING_IN_DOCKER=True

RUN groupadd --gid 1000 rally && \
useradd -d /rally -m -k /dev/null -g 1000 -N -u 1000 -l -s /bin/bash rally
USER root
COPY --from=builder /usr/bin/ /usr/bin/

COPY --chown=1000:0 --from=builder /rally/venv /rally/venv
RUN addgroup --gid 1000 rally && \
adduser --system --home /rally --ingroup rally --no-create-home --uid 1000 --shell /bin/bash rally

WORKDIR /rally
COPY --chown=1000:0 docker/bin/entrypoint.sh /entrypoint.sh
Expand All @@ -57,12 +37,24 @@ RUN chgrp 0 /entrypoint.sh && \
chmod 0775 /entrypoint.sh

RUN mkdir -p /rally/.rally && \
chown -R 1000:0 /rally/.rally
chown -R 1000:0 /rally/

USER 1000

RUN mkdir -p /rally/esrally
COPY pyproject.toml /rally/
COPY README.md /rally/
COPY esrally/ /rally/esrally/

RUN python3 -m venv /rally/venv
ENV PATH=/rally/venv/bin:$PATH

# Wipe away any lingering caches, copied over from the local machine
RUN find /rally -name "__pycache__" -exec rm -rf -- \{\} \; 2>/dev/null || true
RUN find /rally -name ".pyc" -exec rm -rf -- \{\} \; 2>/dev/null || true
RUN python3 -m pip install --upgrade hatch hatchling pip wheel
RUN python3 -m pip install /rally

LABEL org.label-schema.schema-version="1.0" \
org.label-schema.vendor="Elastic" \
org.label-schema.name="rally" \
Expand Down
42 changes: 31 additions & 11 deletions docker/Dockerfiles/Dockerfile-release
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,34 @@
FROM python:3.8.13-slim-bullseye
FROM docker.elastic.co/wolfi/python:3.11.7-dev AS builder

USER root

RUN apk update
RUN apk add curl git gcc pigz bash zstd bzip2 gzip

# pbzip2 doesn't have a package for wolfi, so we build it from source
RUN apk add bzip2-dev make wget


RUN cd /tmp && \
wget -q https://launchpad.net/pbzip2/1.1/1.1.13/+download/pbzip2-1.1.13.tar.gz && \
tar -xzf pbzip2-1.1.13.tar.gz && \
cd pbzip2-1.1.13/ && \
make install && \
rm -r /tmp/pbzip2-1.1.13/

FROM docker.elastic.co/wolfi/python:3.11.7-dev
ARG RALLY_VERSION
ARG RALLY_LICENSE

ENV RALLY_RUNNING_IN_DOCKER True
ENV RALLY_RUNNING_IN_DOCKER=True

RUN apt-get -y update && \
apt-get install -y curl git gcc pbzip2 pigz && \
apt-get -y upgrade && \
rm -rf /var/lib/apt/lists/*
USER root
COPY --from=builder /usr/bin/ /usr/bin/

RUN groupadd --gid 1000 rally && \
useradd -d /rally -m -k /dev/null -g 1000 -N -u 1000 -l -s /bin/bash rally
RUN addgroup --gid 1000 rally && \
adduser --system --home /rally --ingroup rally --no-create-home --uid 1000 --shell /bin/bash rally

RUN pip3 install --upgrade hatch hatchling pip wheel
RUN pip3 install esrally==$RALLY_VERSION
RUN mkdir /rally/

WORKDIR /rally
COPY --chown=1000:0 docker/bin/entrypoint.sh /entrypoint.sh
Expand All @@ -25,12 +40,17 @@ RUN chgrp 0 /entrypoint.sh && \
chmod 0775 /entrypoint.sh

RUN mkdir -p /rally/.rally && \
chown -R 1000:0 /rally/.rally
chown -R 1000:0 /rally/

USER 1000

RUN python3 -m venv /rally/venv
ENV PATH=/rally/venv/bin:$PATH

RUN python3 -m pip install --upgrade hatch hatchling pip wheel
RUN python3 -m pip install esrally==$RALLY_VERSION


LABEL org.label-schema.schema-version="1.0" \
org.label-schema.vendor="Elastic" \
org.label-schema.name="rally" \
Expand Down
7 changes: 5 additions & 2 deletions esrally/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,10 @@ def present(self):

def load(self) -> configparser.ConfigParser:
config = configparser.ConfigParser()
config.read(self.location, encoding="utf-8")
with open(self.location, encoding="utf-8") as src:
contents = src.read()
contents = Template(contents).substitute(CONFIG_DIR=self.config_dir)
config.read_string(contents, source=self.location)
return config

def store_default_config(self, template_path=None):
Expand All @@ -64,7 +67,7 @@ def store_default_config(self, template_path=None):
with open(self.location, "w", encoding="utf-8") as target:
with open(source_path, encoding="utf-8") as src:
contents = src.read()
target.write(Template(contents).substitute(CONFIG_DIR=self.config_dir))
target.write(contents)

def store(self, config: configparser.ConfigParser):
io.ensure_dir(self.config_dir)
Expand Down
23 changes: 18 additions & 5 deletions esrally/log.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,16 +95,29 @@ def install_default_log_config():
source_path = io.normalize_path(os.path.join(os.path.dirname(__file__), "resources", "logging.json"))
with open(log_config, "w", encoding="UTF-8") as target:
with open(source_path, encoding="UTF-8") as src:
# Ensure we have a trailing path separator as after LOG_PATH there will only be the file name
log_path = os.path.join(paths.logs(), "")
# the logging path might contain backslashes that we need to escape
log_path = io.escape_path(log_path)
Comment on lines -100 to -101
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're dropping backlash escaping which was meant to help with Windows judging by #829. But we never officially supported Windows so I'm fine with it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, how did I not see this? I am happy to add this logic back to the code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, on second thought, this would only have been valuable when we were writing the file out to disk for deserialization later. Since we are doing the replacement on the fly, it should still work, even on Windows

contents = src.read().replace("${LOG_PATH}", log_path)
contents = src.read()
target.write(contents)
add_missing_loggers_to_config()
io.ensure_dir(paths.logs())


# pylint: disable=unused-argument
def configure_file_handler(*args, **kwargs) -> logging.Handler:
"""
Configures the WatchedFileHandler supporting expansion of `~` and `${LOG_PATH}` to the user's home and the log path respectively.
"""
filename = kwargs.pop("filename").replace("${LOG_PATH}", paths.logs())
return logging.handlers.WatchedFileHandler(filename=filename, encoding=kwargs["encoding"], delay=kwargs.get("delay", False))


def configure_profile_file_handler(*args, **kwargs) -> logging.Handler:
"""
Configures the FileHandler supporting expansion of `~` and `${LOG_PATH}` to the user's home and the log path respectively.
"""
filename = kwargs.pop("filename").replace("${LOG_PATH}", paths.logs())
return logging.FileHandler(filename=filename, encoding=kwargs["encoding"], delay=kwargs.get("delay", False))


def load_configuration():
"""
Loads the logging configuration. This is a low-level method and usually
Expand Down
8 changes: 4 additions & 4 deletions esrally/resources/logging.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,15 @@
},
"handlers": {
"rally_log_handler": {
"class": "logging.handlers.WatchedFileHandler",
"filename": "${LOG_PATH}rally.log",
"()": "esrally.log.configure_file_handler",
"filename": "${LOG_PATH}/rally.log",
"encoding": "UTF-8",
"formatter": "normal",
"filters": ["isActorLog"]
},
"rally_profile_handler": {
"class": "logging.FileHandler",
"filename": "${LOG_PATH}profile.log",
"()": "esrally.log.configure_profile_file_handler",
"filename": "${LOG_PATH}/profile.log",
"delay": true,
"encoding": "UTF-8",
"formatter": "profile"
Expand Down
Loading