Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Adding the Docker workflow for v1 #66

Merged
merged 26 commits into from
Jul 13, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
da9c3e9
added docker workflow
sarthakpati Jul 12, 2023
e48e2ef
added paths and updating $PATH AND $LD_LIBRARY_PATH
sarthakpati Jul 12, 2023
01663f8
added cython installation
sarthakpati Jul 12, 2023
1c781d9
Merge branch 'master' into docker_for_v1
sarthakpati Jul 12, 2023
5e0b9a9
trying to save space
sarthakpati Jul 12, 2023
924cfca
Update docker-image.yml
sarthakpati Jul 12, 2023
ebf492e
trying codeql next
sarthakpati Jul 12, 2023
5f34b40
Update docker-image.yml
sarthakpati Jul 12, 2023
c9ab2d5
updated reserve
sarthakpati Jul 12, 2023
9223019
trying another action
sarthakpati Jul 12, 2023
2e982e8
manual deletion of android and .net
sarthakpati Jul 12, 2023
b4f315a
Update docker-image.yml
sarthakpati Jul 12, 2023
824c753
gdcm was not working
sarthakpati Jul 12, 2023
da32f7b
updating setuptools
sarthakpati Jul 13, 2023
3a81e73
Update Dockerfile
sarthakpati Jul 13, 2023
f5f6e99
ensure same base env for both
sarthakpati Jul 13, 2023
d61c082
trying install for `scikit-learn`
sarthakpati Jul 13, 2023
82495a7
installing specific version of grpcio and updated entry path
sarthakpati Jul 13, 2023
84627ec
updated hash
sarthakpati Jul 13, 2023
44977bb
Merge branch 'docker_for_v1' of https://github.com/FETS-AI/Front-End …
sarthakpati Jul 13, 2023
7a90712
syntax fix
sarthakpati Jul 13, 2023
2992e4c
checking path
sarthakpati Jul 13, 2023
5d4df9e
checking path
sarthakpati Jul 13, 2023
73deb1b
fixed path
sarthakpati Jul 13, 2023
b8d49a2
ensure that libGL is found
sarthakpati Jul 13, 2023
7948858
trying sklearn deprecated flag
sarthakpati Jul 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 151 additions & 0 deletions .github/workflows/docker-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
name: Docker Image CI

on:
push:
branches: [ master ]
tags: [ '*.*.*' ]
pull_request:
branches: [ master ]
workflow_dispatch:
inputs:
versionTag:
description: 'Version tag to push to Docker Hub (lowercase, alphanumeric)'
required: true
type: string

jobs:

build_test_push:
runs-on: ubuntu-latest
strategy:
fail-fast: false # So that remaining jobs don't instantly quit if one fails (e.g, CPU/ROCm don't upload if CUDA just fails to push to ghcr...)
# matrix:
# include: # Platform locates Dockerfile ("Dockerfile-{PLATFORM}"), docker tag has to be all lowercase alphanumeric for mysterious docker reasons
# - platform: CPU
# dockertag: cpu
# - platform: ROCm
# dockertag: rocm

steps:
- name: Check out the repository
uses: actions/checkout@v3
# with:
# lfs: false
# submodules: 'recursive'

# - name: Check if the repository has changed
# run: ls -l

- name: Docker prune to save space # If you switch to self-hosted runners, this should be removed.
run: echo y | docker system prune -a

- name: Check free space before
run: |
echo "Free space:"
df -h

- name: Free Disk Space (Ubuntu)
run: |
sudo rm -rf /usr/local/lib/android
sudo rm -rf /usr/share/dotnet

- name: Check free space after
run: |
echo "Free space:"
df -h

# Install the cosign tool except on PR
# https://github.com/sigstore/cosign-installer
- name: Install cosign
if: github.event_name != 'pull_request'
uses: sigstore/cosign-installer@1e95c1de343b5b0c23352d6417ee3e48d5bcd422
with:
cosign-release: 'v1.4.0'

- name: Log into GitHub Packages registry (ghcr.io)
if: github.event_name != 'pull_request'
uses: docker/#-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

# Extract metadata (tags, labels) for Docker
# https://github.com/docker/metadata-action
- name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v4
with:
images: ghcr.io/fets-ai/front-end
flavor: | # Handle prefixing and "latest" generation -- use "tags" property to specify events/tags further
latest=true
tags: |
type=semver,pattern={{version}}
type=ref,event=branch
type=ref,event=pr
type=ref,event=tag

# Build Docker Image (but don't push yet -- wait for the test step first).
# https://github.com/docker/build-push-action
- name: Build Docker images
id: build
uses: docker/build-push-action@v4
with:
context: .
# cache-from: type=local,src=${{ github.workspace }}
# build-args: --mount 'type=bind,src=${{ github.workspace }},target=/Front-End,readwrite' #-v ${{ github.workspace }}:/Front-End
file: ./Dockerfile
push: false
load: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

# - uses: addnab/docker-run-action@v3
# with:
# username: ${{ github.actor }}
# password: ${{ secrets.GITHUB_TOKEN }}
# registry: ghcr.io
# image: ghcr.io/fets-ai/front-end:latest
# options: -v ${{ github.workspace }}:/Front-End
# run: |
# echo "Running Script"
# /work/run-script
# - name: Build the Docker image
# run: docker build . -v /Front-End:/home/runner/work/Front-End/Front-End/ --file Dockerfile --tag ${{ steps.meta.outputs.tags }}


# Run the image from the base entrypoint as a test
- name: Test container with entrypoint
# Run a tag we generated from the metadata extraction above -- they're all the same image, but echo it regardless just so we know.
run: docker run --rm ghcr.io/fets-ai/front-end:latest -h

# Push Docker image with Buildx (but don't push on PR)
# https://github.com/docker/build-push-action
# This won't re-build the images fully or anything, they should already exist from the build step and use the cache.
- name: Upload to Docker Hub (docker.io) and GitHub Packages (ghcr.io)
id: upload
if: github.event_name != 'pull_request'
uses: docker/build-push-action@v4
with:
context: .
file: ./Dockerfile
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

# Below is for signing images (keyless) with cosign. But this adds confusing sha256-digest.sig tags to the container registry.
# Leave this commented if container signing is not required.
# # Sign the resulting Docker image digest except on PRs.
# # Uses cosign keyless signing: https://github.com/sigstore/cosign/blob/main/KEYLESS.md
# # This will only write to the public Rekor transparency log when the Docker
# # repository is public to avoid leaking data. If you would like to publish
# # transparency data even for private images, pass --force to cosign below.
# # https://github.com/sigstore/cosign
#- name: Sign published Docker image (ghcr.io)
# if: ${{ github.event_name != 'pull_request' }}
# env:
# COSIGN_EXPERIMENTAL: "true"
# # This step uses the identity token to provision an ephemeral certificate
# # against the sigstore community Fulcio instance.
# run: cosign sign ghcr.io/fets-ai/front-end@${{ steps.upload.outputs.digest }}

23 changes: 14 additions & 9 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ RUN apt-get update -y

RUN apt-get install wget zip unzip software-properties-common gcc g++ make -y

RUN apt-get update -y && add-apt-repository ppa:deadsnakes/ppa && apt update -y && apt install python3.7 python3.7-venv python3.7-dev python3-setuptools -y
RUN apt-get update -y && add-apt-repository ppa:deadsnakes/ppa && apt update -y && apt install python3.7 python3.7-venv python3.7-dev python3-setuptools ffmpeg libsm6 libxext6 -y

# We will do git pull on the FeTS_Front-End master, because that is the repo using which the base image is made
# We will not do compiles on the PR because the idea is that the Xenial build will check the build status of
Expand All @@ -20,21 +20,23 @@ RUN wget https://fets.projects.nitrc.org/FeTS_${VERSION}_Installer.bin && chmod
# install FeTS and remove installer
RUN yes yes | ./FeTS_${VERSION}_Installer.bin --target ./FeTS_${VERSION} -- --cudaVersion 11 && rm -rf ./FeTS_${VERSION}_Installer.bin

ENV PATH=./FeTS_${VERSION}/squashfs-root/usr/bin/:$PATH
ENV LD_LIBRARY_PATH=./FeTS_${VERSION}/squashfs-root/usr/lib/:$LD_LIBRARY_PATH
ENV PATH=/workspace/FeTS_${VERSION}/squashfs-root/usr/bin/:$PATH
ENV LD_LIBRARY_PATH=/workspace/FeTS_${VERSION}/squashfs-root/usr/lib/:$LD_LIBRARY_PATH
ENV SKLEARN_ALLOW_DEPRECATED_SKLEARN_PACKAGE_INSTALL="True"

# set up environment and install correct version of pytorch
RUN cd ./FeTS_${VERSION}/squashfs-root/usr/bin/OpenFederatedLearning && \
rm -rf ./venv && python3.7 -m venv ./venv && \
rm -rf ./venv && python3.7 -m venv ./venv && ./venv/bin/pip install Cython && \
./venv/bin/pip install --upgrade pip setuptools wheel setuptools-rust && \
./venv/bin/pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

RUN cd ./FeTS_${VERSION}/squashfs-root/usr/bin/OpenFederatedLearning && \
./venv/bin/pip install wheel && \
./venv/bin/pip install scikit-build && \
./venv/bin/pip install scikit-build scikit-learn && \
./venv/bin/pip install SimpleITK==1.2.4 && \
./venv/bin/pip install protobuf==3.17.3 && \
./venv/bin/pip install opencv-python==4.2.0.34 && \
./venv/bin/pip install python-gdcm
./venv/bin/pip install protobuf==3.17.3 grpcio==1.30.0 && \
./venv/bin/pip install opencv-python==4.2.0.34
# ./venv/bin/pip install python-gdcm

RUN cd ./FeTS_${VERSION}/squashfs-root/usr/bin/OpenFederatedLearning && \
./venv/bin/pip install setuptools --upgrade && \
Expand All @@ -43,11 +45,14 @@ RUN cd ./FeTS_${VERSION}/squashfs-root/usr/bin/OpenFederatedLearning && \
./venv/bin/pip install -e ./submodules/fets_ai/Algorithms/GANDLF && \
cd ../LabelFusion && \
rm -rf venv && python3.7 -m venv ./venv && \
./venv/bin/pip install --upgrade pip setuptools wheel setuptools-rust && \
./venv/bin/pip install -e .

# set up the docker for GUI
ENV QT_X11_NO_MITSHM=1
ENV QT_GRAPHICSSYSTEM="native"

RUN echo "Env paths\n" && echo $PATH && echo $LD_LIBRARY_PATH

# define entry point
ENTRYPOINT ["/FeTS_\${VERSION}/bin/install/appdir/usr/bin/FeTS_CLI_Inference"]
ENTRYPOINT ["/workspace/FeTS_0.0.9/squashfs-root/usr/bin/FeTS_CLI_Inference"]
2 changes: 1 addition & 1 deletion OpenFederatedLearning