Reducing wait time for cold caches with large apps

[toolmantim]

Currently the best way to handle a large app that takes ages to build is to have two separate instances of the stack: builders and runners, and https://github.com/buildkite-plugins/docker-compose-buildkite-plugin to have a build step that runs before the steps.

The builder doesn't autoscale, sits there with a warm docker cache, and pushes to a registry. The test steps run on the runners queue, and pulls from docker hub without doing any git clone at all.

The pipeline looks like this:

steps:

  - name: ":docker: :package:"
    plugins:
      docker-compose:
        build: app
        image-repository: index.docker.io/myorg/myapp
    agents:
      queue: elastic-builders

  - wait

  - name: ":spec: %n"
    command: ".buildkite/steps/knapsack"
    plugins:
      docker-compose:
        run: app
    agents:
      queue: elastic-runners
    parallelism: 75

A simpler setup would be to use something like http://blog.runnable.com/post/145362675491/distributing-docker-cache-across-hosts to make the cache mostly invisible, and require no separate queue or pipeline config changes.

[lox]

Distributing cache across hosts would be great. For that to work, you'd need a list of images to save and then some code in the environment and post-command hooks. Thoughts on how you'd specify which images to cache?

[toolmantim]

So @tonistiigi just released https://github.com/tonistiigi/buildcache (moby/moby#20316 (comment)) which we could potentially use

[mtibben]

Codeship has a blog post explaining how they tackled this problem https://blog.codeship.com/building-a-remote-caching-system/

[lox]

This will be solved by #234.

It could also be solved by using the new docker 1.13 --cache-from stuff, which needs some investigation.

[jdotjdot]

I'm seeing some recursion here between this Issue and #234. Is the shared layer cache support being handled?

[lox]

@jdotjdot you should be able to use the new --cache-from stuff in docker-compose 1.14, which is on the latest stack version. I'm going to close this for now, do you have a specific issue that you are currently seeing?

[jdotjdot]

We don't use docker-compose is the issue. Is there another way?

…

On Tue, Jul 11, 2017 at 23:02 Lachlan Donald ***@***.***> wrote: Closed #81 <#81>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#81 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABRHQSHsFmFUD6LgxxZtSUCfpK8bMNrIks5sNDdIgaJpZM4JGnPX> .

[lox]

Ah, right, you don't use docker at all? Or just not docker-compose?

[jdotjdot]

Docker compose. We use docker for everything.

…

On Tue, Jul 11, 2017 at 23:07 Lachlan Donald ***@***.***> wrote: Ah, right, you don't use docker at all? Or just not docker-compose? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#81 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABRHQe4UPTkSz6lRyLXhdHBQJAmmqwb4ks5sNDhYgaJpZM4JGnPX> .

[lox]

You should be able to make use of the --cache-from stuff in Docker.