Discussion: Performance improvements

[craig-willis]

(Replaces #140)

Problems:

• Building and pushing images can take a long time
• Spawning instances can take a long time when images need to be pulled to the nodes

Ideas:

• Try to build image on the same node it was last built on
• Run the instance on the same node it was last built on
• Make push async on launch -- no need to wait for push to run
• Make sure nightly cache rebuilds are also pushed to registry has up-to-date copy of base images

Many of these hinge on storing the last-known-node-id and submitting to the queue directly, which has apparent problems for distributing workloads. (Can we "suggest" a node to celery?)

[craig-willis]

Largely solved by new caching strategy.