Jobs on GPU #230

hhsecond · 2019-09-27T03:20:02Z

CircleCI config for running GPU jobs
Test cases running on GPU Device

I am not using any of the existing build utilities such as system-setup.py since relying on a file inside the repo trigger rebuilding of that layer on each push which is time-consuming. We need to iterate over this and other build processes to unify the build processes and to do the packaging and release of GPU docker image etc

Also, because of the allow growth feature in TF, the test cases on GPU takes a lot of time now

test/basic_tests.py

opt/Makefile

Dockerfile.gpu-test

.circleci/config.yml

lantiga

Looks good.
Now that we know this works, we should indeed consider moving the installation of deps to system_setup.py to get non-dockerized setups for free and for consistency with the rest of the configurations.

hhsecond · 2019-10-03T05:40:23Z

@lantiga @rafie I agree with you on moving the dependency installations to system-setup.py but the fact that docker layer caching doesn't work on ADD for some reason causes all of the dependency installation to reruns on all the builds. This seems inefficient for me especially since GPU resources are costly. Let's see if CircleCI folks have some idea about this and I'll move to system-setup anyways

hhsecond · 2019-10-03T17:17:07Z

@lantiga @rafie
I have removed all the dependency installations from the docker file and relying on system-setup.py for most of the part. But dependencies required for the Redis installations are still in the docker file. It is because docker layer caching we had enabled wasn't working as expected and putting those dependencies also to system-setup will cause the job to take quite a lot of time to finish. I have raised a complaint with them already. Let see what they have got.
Also, the GPU backend is disabled in circle CI from today and they are going to send us a contract soon.
LET'S NOT MERGE THIS TILL THE GPU SUPPORT IS BACK

lantiga · 2019-10-04T07:12:51Z

@hhsecond Thank you Sherin, great job. Given the costs associated with building on GPU, it's really important we don't waste precious compute minutes building.

opt/build/libtorch/repack.sh

test/test_requirements.txt

get_deps.sh

lantiga

Looks great!

lantiga · 2019-12-31T15:29:46Z

(@hhsecond please address Rafi’s comments before merging)

lantiga · 2020-01-02T08:00:09Z

Hey @hhsecond, are we good to go?

Jobs on GPU (#230) - GPU build on CI - ORT CUDA compatibility issue fix

hhsecond added the workinprogress label Sep 27, 2019

hhsecond force-pushed the gputests branch 3 times, most recently from 46366ff to 1073b8c Compare October 2, 2019 08:13

hhsecond removed the workinprogress label Oct 2, 2019

hhsecond changed the title ~~[WIP] Jobs on GPU~~ Jobs on GPU Oct 2, 2019

hhsecond requested review from lantiga and rafie October 2, 2019 08:16