401 error when pushing image to local docker registry with TLS (Self Signed) and Basic Auth enabled.

[ashimusmani]

Describe the bug
401 error is coming when makisu builds image and tries to push to local docker registry which has TLS enabled using self signed certificates with Basic Auth enabled.

Note: Tested without TLS and disabled tls client in registry config, in this case too same error is coming.

To Reproduce
Build the docker image and specify the destination to push image to a local docker registry which has TLS enabled using self signed certs and has basic auth enabled.

Command:
/Users/neo/go/bin/makisu build -t registry.ketchup.com:5000/makisutest . --registry-config='{"registry.ketchup.com:5000": {".*": {"security": {"tls": {"client": {"disabled":false} , "ca": {"cert": {"path":"/Users/neo/Documents/dev/java/ketchup-demo-basicspringboot/registry_certs/domain.cert"}}}, "basic": {"username": "local-registry-username", "password": "local-registry-pwd"}}}}, "index.docker.io": {".*": {"security": {"tls": {"client": {"disabled":true}}, "basic": {"username": "docker-hub-username", "password": "docker-hub-pwd"}}}}}' --push=registry.ketchup.com:5000

Note: Tried running the build with tag having local registry username:
-t registry.ketchup.com:5000/tessuser/makisutest
In this case too the error is same.

Expected behavior
Final image should get pushed to local docker registry.

Screenshots

Environments
Machine: MBPr2015
OS: MacOS Catalina 10.15.3 (19D76)
Makisu Version: master-unreleased
Docker Version: 19.03.5
Kubernetes Version: 1.15.5
Docker Desktop Version: 2.2.0.0

Additional context

Dockerfile:

FROM openjdk:8-jre-alpine
ADD target/ketchup-demo-basicspringboot-0.0.1-SNAPSHOT.jar ketchup-demo-basicspringboot.jar
EXPOSE 8080
ENTRYPOINT exec java -Djava.security.egd=file:/dev/./urandom -jar ketchup-demo-basicspringboot.jar

Logs:

{"level":"info","ts":1580807871.639841,"msg":"Starting Makisu build (version=master-unreleased)"}
{"level":"info","ts":1580807871.645922,"msg":"Using build context: /Users/neo/Documents/dev/java/ketchup-demo-basicspringboot"}
{"level":"info","ts":1580807871.6465971,"msg":"Using local file at /tmp/makisu-storage/cache_key_value.json for cacheID storage"}
{"level":"info","ts":1580807871.6779258,"msg":"* Stage 1/1 : (alias=0,latestfetched=-1)"}
{"level":"error","ts":1580807871.678006,"msg":"Failed to fetch intermediate layer with cache ID b53a0e79: find layer b53a0e79: layer not found in cache"}
{"level":"info","ts":1580807871.67803,"msg":"* Step 1/4 (commit) : FROM openjdk:8-jre-alpine  (66f7fb65)"}
{"level":"info","ts":1580807871.678211,"msg":"* Started pulling image index.docker.io/library/openjdk:8-jre-alpine"}
{"level":"info","ts":1580807871.678245,"msg":"Client TLS is disabled"}
{"level":"info","ts":1580807876.013236,"msg":"Client TLS is disabled"}
{"level":"info","ts":1580807876.013236,"msg":"Client TLS is disabled"}
{"level":"info","ts":1580807876.013238,"msg":"Client TLS is disabled"}
{"level":"info","ts":1580807876.2796311,"msg":"* Started pulling layer index.docker.io/library/openjdk:sha256:f910a506b6cb1dbec766725d70356f695ae2bf2bea6224dbe8c7c6ad4f3664a2"}
{"level":"info","ts":1580807877.0499158,"msg":"* Started pulling layer index.docker.io/library/openjdk:sha256:b6abafe80f63b02535fc111df2ed6b3c728469679ab654e03e482b6f347c9639"}
{"level":"info","ts":1580807877.107436,"msg":"* Started pulling layer index.docker.io/library/openjdk:sha256:e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10"}
{"level":"info","ts":1580807877.864074,"msg":"* Finished pulling layer library/openjdk:f910a506b6cb1dbec766725d70356f695ae2bf2bea6224dbe8c7c6ad4f3664a2"}
{"level":"info","ts":1580807877.8642151,"msg":"Client TLS is disabled"}
{"level":"info","ts":1580807878.615558,"msg":"* Started pulling layer index.docker.io/library/openjdk:sha256:f7a292bbb70c4ce57f7704cc03eb09e299de9da19013b084f138154421918cb4"}
{"level":"info","ts":1580807880.6084402,"msg":"* Finished pulling layer library/openjdk:f7a292bbb70c4ce57f7704cc03eb09e299de9da19013b084f138154421918cb4"}
{"level":"info","ts":1580807886.9492102,"msg":"* Finished pulling layer library/openjdk:e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10"}
{"level":"info","ts":1580807969.6935968,"msg":"* Finished pulling layer library/openjdk:b6abafe80f63b02535fc111df2ed6b3c728469679ab654e03e482b6f347c9639"}
{"level":"info","ts":1580807969.700617,"msg":"* Pulled image index.docker.io/library/openjdk:8-jre-alpine","duration":98.005981778}
{"level":"info","ts":1580807969.709734,"msg":"* Processing FROM layer e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10"}
{"level":"info","ts":1580807969.7924008,"msg":"* Merged 476 headers from tar to memfs"}
{"level":"info","ts":1580807969.796198,"msg":"* Processing FROM layer f910a506b6cb1dbec766725d70356f695ae2bf2bea6224dbe8c7c6ad4f3664a2"}
{"level":"info","ts":1580807969.7964559,"msg":"* Merged 4 headers from tar to memfs"}
{"level":"info","ts":1580807969.797358,"msg":"* Processing FROM layer b6abafe80f63b02535fc111df2ed6b3c728469679ab654e03e482b6f347c9639"}
{"level":"info","ts":1580807970.942784,"msg":"* Merged 1380 headers from tar to memfs"}
{"level":"info","ts":1580807970.94283,"msg":"* Executed FROM openjdk:8-jre-alpine  (66f7fb65)","duration":99.248352021}
{"level":"info","ts":1580807970.9434261,"msg":"* Step 2/4 (commit) : ADD target/ketchup-demo-basicspringboot-0.0.1-SNAPSHOT.jar ketchup-demo-basicspringboot.jar  (b53a0e79)"}
{"level":"info","ts":1580807970.9441931,"msg":"* Executed ADD target/ketchup-demo-basicspringboot-0.0.1-SNAPSHOT.jar ketchup-demo-basicspringboot.jar  (b53a0e79)","duration":0.000493089}
{"level":"info","ts":1580807972.167839,"msg":"* Created copy layer with 1 files"}
{"level":"info","ts":1580807972.201034,"msg":"* Committed gzipped layer sha256:c075a5fe0a3cc72770b4c7d4f0b668abdeaeac798ed7ca419203a92180bad4b8 (15532155 bytes)"}
{"level":"info","ts":1580807972.201067,"msg":"* Pushing with cache ID b53a0e79"}
{"level":"info","ts":1580807972.203242,"msg":"* Step 3/4 (commit) : EXPOSE 8080  (41fcf1fd)"}
{"level":"info","ts":1580807972.2033339,"msg":"* Executed EXPOSE 8080  (41fcf1fd)","duration":0.000003883}
{"level":"info","ts":1580807972.203365,"msg":"* Pushing with cache ID 41fcf1fd"}
{"level":"info","ts":1580807972.20385,"msg":"* Step 4/4 (commit) : ENTRYPOINT exec java -Djava.security.egd=file:/dev/./urandom -jar ketchup-demo-basicspringboot.jar  (e8422021)"}
{"level":"info","ts":1580807972.20409,"msg":"* Executed ENTRYPOINT exec java -Djava.security.egd=file:/dev/./urandom -jar ketchup-demo-basicspringboot.jar  (e8422021)","duration":0.000007183}
{"level":"info","ts":1580807972.2041159,"msg":"* Pushing with cache ID e8422021"}
{"level":"info","ts":1580807972.206353,"msg":"Stored cacheID mapping to KVStore: 41fcf1fd => MAKISU_CACHE_EMPTY"}
{"level":"info","ts":1580807972.207135,"msg":"Stored cacheID mapping to KVStore: e8422021 => MAKISU_CACHE_EMPTY"}
{"level":"error","ts":1580807972.650284,"msg":"Failed to push cache: push layer sha256:c075a5fe0a3cc72770b4c7d4f0b668abdeaeac798ed7ca419203a92180bad4b8: check layer exists: registry.ketchup.com:5000/makisutest (sha256:c075a5fe0a3cc72770b4c7d4f0b668abdeaeac798ed7ca419203a92180bad4b8): check manifest exists: HEAD https://registry.ketchup.com:5000/v2/makisutest/blobs/sha256:c075a5fe0a3cc72770b4c7d4f0b668abdeaeac798ed7ca419203a92180bad4b8 401"}
{"level":"info","ts":1580807972.6523352,"msg":"Computed total image size 73225061","total_image_size":73225061}
{"level":"info","ts":1580807972.652364,"msg":"Successfully built image makisutest:latest"}
{"level":"info","ts":1580807972.652719,"msg":"* Started pushing image registry.ketchup.com:5000/makisutest:latest"}
{"level":"error","ts":1580807972.707519,"msg":"failed to push image: failed to push image: check manifest exists for image registry.ketchup.com:5000/makisutest:latest: check manifest exists: HEAD https://registry.ketchup.com:5000/v2/makisutest/manifests/latest 401"}