Skip to content

Updating rosetta dockerfile #563

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merged
merged 2 commits into from
May 10, 2021
+122 −133
Merged

Updating rosetta dockerfile #563

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0189ee0
Updating rosetta dockerfile
wileyj Apr 21, 2021
41f0ffe
Hardcoding versions
wileyj May 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
255 changes: 122 additions & 133 deletions stx-rosetta.Dockerfile
View file
Original file line number Diff line number Diff line change
@@ -1,147 +1,136 @@
### Build blockstack-core-sidecar API
FROM node:lts-buster as build

ARG API_TAG=v0.29.4

RUN apt-get -y update && apt-get -y install openjdk-11-jre-headless

ARG STACKS_API_VERSION=v0.56.0
ARG STACKS_NODE_VERSION=2.0.11.0.0
ARG STACKS_API_REPO=blockstack/stacks-blockchain-api
ARG STACKS_NODE_REPO=blockstack/stacks-blockchain
ARG PG_VERSION=12
ARG STACKS_NETWORK=testnet
ARG STACKS_LOG_DIR=/var/log/stacks-node
ARG STACKS_SVC_DIR=/etc/service
ARG STACKS_BLOCKCHAIN_DIR=/stacks-blockchain
ARG STACKS_BLOCKCHAIN_API_DIR=/stacks-blockchain-api
ARG PG_DATA=/data/postgres
ARG V2_POX_MIN_AMOUNT_USTX=90000000260

#######################################################################
## Build the stacks-blockchain-api
FROM node:lts-buster as stacks-blockchain-api-build
ARG STACKS_API_REPO
ARG STACKS_API_VERSION
ENV STACKS_API_REPO=${STACKS_API_REPO}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this ENV var can be removed. ${STACKS_API_REPO} will just use the ARG like it's an ENV var. The ENV var directive should be used when the env variable needs to be available in the resulting image.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unless the repo name changes, in which case this is a single change or a build-arg vs multiple changes in the file

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm saying you can delete just this ENV var and the image will work the same as it did before. There's no benefit to declaring an ARG then setting an ENV var to it if the env var won't be used in the final image because it's redundant. If the repo name changes, you'd update the ARG default value.

ENV STACKS_API_VERSION=${STACKS_API_VERSION}
WORKDIR /app

RUN git clone -b $API_TAG --depth 1 https://github.com/blockstack/stacks-blockchain-api.git .
RUN echo "GIT_TAG=$(git tag --points-at HEAD)" >> .env
RUN npm install && npm run build && npm prune --production

### Build stacks-node binary

FROM rust:stretch as stacks-node-build

ARG STACKS_TAG=2.0.5

RUN mkdir -p /src /stacks
RUN apt-get update -y \
&& apt-get install -y \
curl \
jq \
openjdk-11-jre-headless \
&& git clone -b ${STACKS_API_VERSION} --depth 1 https://github.com/${STACKS_API_REPO} . \
&& echo "GIT_TAG=$(git tag --points-at HEAD)" >> .env \
&& npm config set unsafe-perm true \
&& npm install \
&& npm run build \
&& npm prune --production

#######################################################################
## Build the stacks-blockchain
FROM rust:buster as stacks-blockchain-build
ARG STACKS_NODE_REPO
ARG STACKS_NODE_VERSION
ENV STACKS_NODE_REPO=${STACKS_NODE_REPO}
ENV STACKS_NODE_VERSION=${STACKS_NODE_VERSION}
WORKDIR /src
RUN git clone -b $STACKS_TAG --depth 1 https://github.com/blockstack/stacks-blockchain.git .
RUN rustup target add x86_64-unknown-linux-gnu
RUN cargo build --release --workspace=./ --target x86_64-unknown-linux-gnu
RUN cp -R /src/target/x86_64-unknown-linux-gnu/release/. /stacks

### Fetch stacks-node binary

### Begin building base image
FROM ubuntu:focal

ARG STACKS_NETWORK=testnet

SHELL ["/bin/bash", "-c"]

### Install utils
RUN apt-get update
RUN apt-get install -y sudo curl pslist

### Set noninteractive apt-get
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections

### Storage goes in /data; see https://www.rosetta-api.org/docs/standard_storage_location.html
RUN mkdir -p /data

### Install nodejs
RUN curl -sL https://deb.nodesource.com/setup_14.x | bash -
RUN apt-get install -y nodejs

### stacky user ###
# see https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#user
RUN useradd -l -u 33333 -G sudo -md /data/stacky -s /bin/bash -p stacky stacky \
# passwordless sudo for users in the 'sudo' group
&& sed -i.bkp -e 's/%sudo\s\+ALL=(ALL\(:ALL\)\?)\s\+ALL/%sudo ALL=NOPASSWD:ALL/g' /etc/sudoers
ENV HOME=/data/stacky
WORKDIR $HOME
USER stacky
RUN sudo chown -R stacky:stacky $HOME
RUN mkdir /data/stacky/.bashrc.d

### Node.js
RUN node -e 'console.log("Node.js runs")'

### Setup stacks-node
COPY --from=stacks-node-build /stacks/stacks-node stacks-node/
ENV PATH="$PATH:$HOME/stacks-node"

### Setup stacks-blockchain-api
COPY --from=build /app stacks-blockchain-api

#### Copy stacks-node mocknet config
RUN cp stacks-blockchain-api/stacks-blockchain/*.toml .

RUN sudo chown -Rh stacky:stacky stacks-blockchain-api
RUN printf '#!/bin/bash\ncd $(dirname $0)\nnpm run start\n' > stacks-blockchain-api/stacks_api \
&& chmod +x stacks-blockchain-api/stacks_api
ENV PATH="$PATH:$HOME/stacks-blockchain-api"
EXPOSE 3999

### Install Postgres
RUN sudo apt-get install -y postgresql-12 postgresql-contrib-12

### Setup Postgres
# Borrowed from https://github.com/gitpod-io/workspace-images/blob/master/postgres/Dockerfile
ENV PATH="$PATH:/usr/lib/postgresql/12/bin"
ENV PGDATA="/data/stacky/.pgsql/data"
RUN mkdir -p ~/.pg_ctl/bin ~/.pg_ctl/sockets \
&& printf '#!/bin/bash\n[ ! -d $PGDATA ] && mkdir -p $PGDATA && initdb -D $PGDATA\npg_ctl -D $PGDATA -l ~/.pg_ctl/log -o "-k ~/.pg_ctl/sockets" start\n' > ~/.pg_ctl/bin/pg_start \
&& printf '#!/bin/bash\npg_ctl -D $PGDATA -l ~/.pg_ctl/log -o "-k ~/.pg_ctl/sockets" stop\n' > ~/.pg_ctl/bin/pg_stop \
&& chmod +x ~/.pg_ctl/bin/*
ENV PATH="$PATH:$HOME/.pg_ctl/bin"

### Clear caches
RUN sudo apt-get clean && sudo rm -rf /var/cache/apt/* /var/lib/apt/lists/* /tmp/*

### Setup service env vars
RUN apt-get update -y \
&& apt-get install -y \
curl \
jq \
&& mkdir -p /out \
&& git clone -b ${STACKS_NODE_VERSION} --depth 1 https://github.com/${STACKS_NODE_REPO} . \
&& cd testnet/stacks-node \
&& cargo build --features monitoring_prom,slog_json --release \
&& cp /src/target/release/stacks-node /out

#######################################################################
## Build the final image with all components from build stages
FROM debian:buster
ARG STACKS_NETWORK
ARG STACKS_LOG_DIR
ARG STACKS_SVC_DIR
ARG STACKS_BLOCKCHAIN_DIR
ARG STACKS_BLOCKCHAIN_API_DIR
ARG PG_DATA
ARG PG_VERSION
ARG V2_POX_MIN_AMOUNT_USTX
ENV PG_HOST=127.0.0.1
ENV PG_PORT=5432
ENV PG_USER=stacky
ENV PG_USER=postgres
ENV PG_PASSWORD=postgres
ENV PG_DATABASE=postgres

ENV PG_DATA=${PG_DATA}
ENV PG_VERSION=${PG_VERSION}
ENV STACKS_SVC_DIR=${STACKS_SVC_DIR}
ENV STACKS_BLOCKCHAIN_DIR=${STACKS_BLOCKCHAIN_DIR}
ENV STACKS_BLOCKCHAIN_API_DIR=${STACKS_BLOCKCHAIN_API_DIR}
ENV STACKS_NETWORK=${STACKS_NETWORK}
ENV STACKS_LOG_DIR=${STACKS_LOG_DIR}
ENV STACKS_CORE_EVENT_PORT=3700
ENV STACKS_CORE_EVENT_HOST=127.0.0.1
ENV STACKS_NETWORK=$STACKS_NETWORK

ENV STACKS_EVENT_OBSERVER=127.0.0.1:3700

ENV STACKS_BLOCKCHAIN_API_PORT=3999
ENV STACKS_BLOCKCHAIN_API_HOST=0.0.0.0

ENV STACKS_CORE_RPC_HOST=127.0.0.1
ENV STACKS_CORE_RPC_PORT=20443

### Startup script & coordinator
RUN printf '#!/bin/bash\n\
trap "exit" INT TERM\n\
trap "kill 0" EXIT\n\
echo Your container args are: "$@"\n\
tail --retry -F stacks-api.log stacks-node.log 2>&1 &\n\
while true\n\
do\n\
pg_start\n\
stacks_api &> stacks-api.log &\n\
stacks_api_pid=$!\n\
if [ $STACKS_NETWORK = "mocknet" -o $STACKS_NETWORK = "dev" ]; then\n\
stacks-node start --config=/data/stacky/Stacks-${STACKS_NETWORK}.toml &> stacks-node.log &\n\
elif [ $STACKS_NETWORK = "testnet"]; then \n\
stacks-node start --config=/data/stacky/Stacks-mocknet.toml &> stacks-node.log &\n\
else\n\
stacks-node mainnet &> stacks-node.log &\n\
fi\n\
stacks_node_pid=$!\n\
wait $stacks_node_pid\n\
echo "node exit, restarting..."\n\
rkill -9 $stacks_api_pid\n\
pg_stop\n\
rm -rf $PGDATA\n\
sleep 5\n\
done\n\
' >> run.sh && chmod +x run.sh

ENV MAINNET_STACKS_CHAIN_ID=0x00000001
ENV TESTNET_STACKS_CHAIN_ID=0x80000000
ENV V2_POX_MIN_AMOUNT_USTX=${V2_POX_MIN_AMOUNT_USTX}
RUN apt-get update \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Combining this RUN command with the following three will further decrease the number of layers and size of the resulting image.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above - any change in the process/dockerfile would invalidate the cache layers and cause a longer rebuild.
Can they all be combined? yes - iniitially that's what i was doing. but any small change required a full rebuild vs updating a single layer or two.

Copy link
Member

@CharlieC3 CharlieC3 Apr 23, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any change in the process/dockerfile would invalidate the cache layers and cause a longer rebuild.

Only true if you COPY the dockerfile into the image. For RUN directives the docker engine only compares the text of the old RUN vs the text of the new RUN to determine if it can be reused from cache. So if there's a change in the command, it won't use the cache either way you configure it. These are only minor suggestions, so feel free to do what you like.

&& apt install -y \
gnupg2 \
lsb-release \
curl procps \
netcat \
gosu \
runit-init \
rsyslog
RUN curl -sL https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
&& echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" > /etc/apt/sources.list.d/pgsql.list \
&& curl -sL https://deb.nodesource.com/setup_14.x | bash -
RUN apt-get update \
&& apt-get install -y \
postgresql-${PG_VERSION} \
postgresql-client-${PG_VERSION} \
nodejs \
&& echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
RUN mkdir -p \
${STACKS_SVC_DIR}/postgresql/log \
${STACKS_SVC_DIR}/stacks-blockchain-api/log \
${STACKS_SVC_DIR}/stacks-blockchain \
${STACKS_LOG_DIR}/postgresql \
${STACKS_LOG_DIR}/stacks-blockchain-api/log \
&& apt-get clean \
&& rm -rf /var/cache/apt/* /var/lib/apt/lists/* /tmp/* ${STACKS_SVC_DIR}/getty*
COPY --from=stacks-blockchain-build /out ${STACKS_BLOCKCHAIN_DIR}
COPY --from=stacks-blockchain-api-build /app ${STACKS_BLOCKCHAIN_API_DIR}
RUN cp ${STACKS_BLOCKCHAIN_API_DIR}/stacks-blockchain/Stacks-mocknet.toml ${STACKS_BLOCKCHAIN_DIR}/Stacks-testnet.toml \
&& cp ${STACKS_BLOCKCHAIN_API_DIR}/stacks-blockchain/Stacks-mocknet.toml ${STACKS_BLOCKCHAIN_DIR}/Stacks-mocknet.toml

###################################
## runit service files
RUN printf '#!/bin/sh\nexec 2>&1\n[ ! -d %s ] && mkdir -p %s && chown -R postgres:postgres %s && gosu postgres /usr/lib/postgresql/%s/bin/pg_ctl init -D %s\nexec gosu postgres /usr/lib/postgresql/%s/bin/postmaster -D %s' ${PG_DATA} ${PG_DATA} ${PG_DATA} ${PG_VERSION} ${PG_DATA} ${PG_VERSION} ${PG_DATA} > ${STACKS_SVC_DIR}/postgresql/run \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you considered committing the ruinit service files to this repo, and COPYing them into the image? Might be easier to maintain and read that way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possible, sure - but changes to the repo would require a full rebuild since the cache layers would no longer be used.
Since there are many parts of this process, the goal was to create an image that can be built using cache layers as quickly as is feasible.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but changes to the repo would require a full rebuild since the cache layers would no longer be used.

If you COPY only those ruinit files, changes to the repo wouldn't require a full rebuild. That only happens when you write COPY . . for example. Either way will work the same, it's just difficult to read or maintain the scripts when they're embedded.

&& printf '#!/bin/sh\nrm -rf %s' ${PG_DATA} > ${STACKS_SVC_DIR}/postgresql/finish \
&& printf '#!/bin/sh\nexec svlogd -tt %s/postgresql' ${STACKS_LOG_DIR} > ${STACKS_SVC_DIR}/postgresql/log/run \
&& printf '#!/bin/sh\nexec 2>&1\nif [ $STACKS_NETWORK != "mainnet" ]; then\n exec %s/stacks-node start --config=%s/Stacks-testnet.toml 2>&1\nelse\n exec %s/stacks-node mainnet 2>&1\nfi' ${STACKS_BLOCKCHAIN_DIR} ${STACKS_BLOCKCHAIN_DIR} ${STACKS_BLOCKCHAIN_DIR} > ${STACKS_SVC_DIR}/stacks-blockchain/run \
&& printf '#!/bin/bash\nexec 2>&1\nsv start postgresql stacks-blockchain || exit 1\nif [ $STACKS_NETWORK != "mainnet" ]; then\n export STACKS_CHAIN_ID=%s\nelse\n export STACKS_CHAIN_ID=%s\n export V2_POX_MIN_AMOUNT_USTX=%s\nfi\ncd %s && exec node ./lib/index.js 2>&1' ${TESTNET_STACKS_CHAIN_ID} ${MAINNET_STACKS_CHAIN_ID} ${V2_POX_MIN_AMOUNT_USTX} ${STACKS_BLOCKCHAIN_API_DIR} > ${STACKS_SVC_DIR}/stacks-blockchain-api/run \
&& printf '#!/bin/sh\nexec svlogd -tt %s/stacks-blockchain-api' ${STACKS_LOG_DIR} > ${STACKS_SVC_DIR}/stacks-blockchain-api/log/run \
&& printf '#!/bin/sh\n/usr/bin/runsvdir %s' ${STACKS_SVC_DIR} > /entrypoint.sh \
&& chmod 755 \
${STACKS_SVC_DIR}/postgresql/run \
${STACKS_SVC_DIR}/postgresql/finish \
${STACKS_SVC_DIR}/postgresql/log/run \
${STACKS_SVC_DIR}/stacks-blockchain/run \
${STACKS_SVC_DIR}/stacks-blockchain-api/run \
${STACKS_SVC_DIR}/stacks-blockchain-api/log/run \
/entrypoint.sh

EXPOSE ${STACKS_BLOCKCHAIN_API_PORT} ${STACKS_CORE_RPC_PORT}
VOLUME /data

ENTRYPOINT ["/data/stacky/run.sh"]

CMD ["/data/stacky/run.sh"]
ENTRYPOINT ["/entrypoint.sh"]