Problems using a custom boot2docker ISO hosted on an internal HTTPS site with self-signed cert

[hairyhenderson]

So I'm getting this error:

$ VIRTUALBOX_BOOT2DOCKER_URL=https://my-internal-server.mydomain.com/boot2docker.iso
$ docker-machine create -d virtualbox --virtualbox-boot2docker-url=$VIRTUALBOX_BOOT2DOCKER_URL dev         
INFO[0000] Downloading boot2docker.iso from https://my-internal-server.mydomain.com/boot2docker.iso... 
ERRO[0000] Error creating machine: Get https://my-internal-server.mydomain.com/boot2docker.iso: x509: certificate signed by unknown authority 
WARN[0000] You will want to check the provider to make sure the machine and associated resources were properly removed. 
FATA[0000] Error creating machine        

I need a custom boot2docker.iso because our internal HTTPS all uses an internal corporate Root CA for signing (so, basically self-signed). I had this working fine with boot2docker-cli (see boot2docker/boot2docker#347), but I want to try out docker-machine with the same ISO.

I'm on OS X, and I've installed the Root CA certs in such a way that most apps recognize it (browsers, etc...). Is there some special other way to install my certs such that docker-machine will be OK with them?

Thanks!

[hairyhenderson]

Some clarification - this actually doesn't work with boot2docker-cli either. I had built the ISO locally at first, so it never had to actually download it. So I guess this is a more general issue. Anyone had any experience with this?

[sthulb]

I'm not quite sure how this would work. I guess it all depends on how "net/http" reads CA certs from the system, I'd assume it'll use the keychain on OS X and the CA store on Linux...

[hairyhenderson]

Yeah, I did some dtrussing and found that it's opening the various keychains in the right order (local, followed by System, followed by System Roots). The certs are in the System keychain. Other (non-golang) commands work fine, such as curl. I dtrussed curl and it looked like it was opening the same keychains... Obviously I have no idea what net/http is doing with those keychains, but... ;)

[databus23]

We are experiencing this problem as well. It seems to me this is caused by the cross-compilation on linux. If I go run a simple go file that creates a http client this same way as getClient it works but not when building the docker-machine binary with script/build

[ehazlett]

Ok thanks. I'll try the same for me and see if we can get a fix.

On Tuesday, February 10, 2015, Fabian Ruff notifications@github.com wrote:

We are experiencing this problem as well. It seems to me this is caused by
the cross-compilation on linux. If I go run a simple go file that creates
a http client this same way as getClient
https://github.com/docker/machine/blob/master/utils/b2d.go#L23 it works
but not when building the docker-machine binary with script/build

—
Reply to this email directly or view it on GitHub
#491 (comment).

[databus23]

Apparently this problem is pretty old news. See boot2docker/boot2docker-cli#13 or moby/moby#3683 for example.

Looking at go's source there are two different ways for reading the root certificates for darwin depending on whether cgo is enabled or not:

1. https://github.com/golang/go/blob/master/src/crypto/x509/root_cgo_darwin.go
2. https://github.com/golang/go/blob/master/src/crypto/x509/root_nocgo_darwin.go (which calls this function)

Looking at the /System/Library/Keychains/SystemRootCertificates.keychain on my machine it does only contain the Apple compiled list of root CAs shipped with OS X and is read only.

Commit golang/go@4f23481 mentions that there are some differences:

The set of certs fetched via exec'ing security is not quite identical to the certs fetched via the cgo call. The cgo fetch includes any trusted root certs that the user may have added; exec does not. The exec fetch includes an Apple-specific root cert; the cgo fetch does not. Other than that, they appear to be the same.

When building natively on OS X this is a non issue but when cross-compiling for darwin it seems to be the CA handling is partly broken. There just seems to be no easy way of making go pick up user supplied trusted root CAs from the Keychain.

[ehazlett]

Should we vendor a set of root CA certs like Swarm does? These could be updated periodically.

[hairyhenderson]

@ehazlett: how would that solve this issue? I need to add CA certs that should never be out in the wild...

[ehazlett]

@hairyhenderson sorry i was thinking of a different CA issue. perhaps allowing the ability to pass a root CA cert for http or something. Not sure, just trying to throw out some ideas :)

[sthulb]

That's probably the only way

[hairyhenderson]

From @databus23 's comments, it seems like this has to do with golang's cross-compilation on Linux... Maybe a fix would be to compile the OS X binary on OS X instead? I'll admit to having close-to-zero clue about golang, or how docker-machine gets built, so maybe it's not feasible...

[ehazlett]

@hairyhenderson Would be worth a test. Would you mind trying this build: https://public.evanhazlett.com/docker-machine/test/docker-machine-f9278b99-darwin-amd64

Built using godep go build from OS X.

[hairyhenderson]

@ehazlett d'oh. Same error as before.

[ehazlett]

hmm ok

[databus23]

Building natively on OS X definitely makes the problem go away. I tested it before and I just tried the linked binary above. It works for me.
I have our corporate CA certificate in the System Keychain and it does not work with cross-compiled binaries but does work whenever I compile natively.
@hairyhenderson Can you re-check where you have installed the self-signed certificate?

@ehazlett I don't like the idea that the user has to specifically pass in the certificate. Ideally it should work with all trusted certificates from the keychain as it does when building natively.

I see three ways to solve this issue:

1. Build natively
2. Provide a homebrew formula and let the user build natively by doing brew install docker-machine
3. There is a way to link against native cgo-enabled versions of the go stdlib when cross compiling. See inconshreveable/gonative and this blog post for more details.

I think 1. is the cleanest and most straight forward but its hard to automate and probably requires manual building for the foreseeable future. 2. adds some requirements for running machine: homebrew has to be installed and when installing machine brew also needs to pull down the go package to build the thing. 3. is the most fiddly but could provide a working binary with the current build workflow via docker.

I added gonative to the docker-machine Dockerfile briefly and was able to cross compile a darwin binary that worked with custom certificates so 3. really does work btw.

[hairyhenderson]

@databus23 - I have my CAs in the System Keychain as well - i.e. /Libraries/Keychains/System.keychain

When I have a bit more time (probably not until Sunday or Monday evening, unfortunately), I can try to compile natively myself and see if that works differently...

[sthulb]

This is an odd bug, we should trace it through and report it.

[ehazlett]

@databus23 completely agree with using system certs.

[databus23]

@sthulb I believe this is a bug that can't be fixed easily. There just seems to be no way of getting a list of all trusted certificates without interfacing with the Security Framework c library provided by Apple. The command line utility /usr/bin/security provides no way of getting certificates and there trust settings from the System Keychain. I spend half day with it and gave up on it.

[sthulb]

I guess we'll just have to make OS X builds on OS X from now on :)

[nathanleclaire]

Wow, very sorry to hear about this @hairyhenderson and @databus23. Thank you for being so diligent with assessment of the issue. I definitely want to help you get a solution to this.

I'm very against changing the default build process to native. gox is fast and easy to use and is set up well now in our build toolchain as it is. It would slow down contribution and releases significantly to not be able to cross-compile so fast and easily (and in a container) - it's not uncommon for me to chuck some of those binaries onto other platforms I want to test machine on where don't have Go tools installed at all. It also might introduce cross-platform differences which could cause bugs. Most of the issues that we would usually need cgo for e.g. home directory we are already working around in various ways.

So this is what I propose to mitigate the issue (short-term):

1. Provide a script/native_build script so that users which need to can compile their own natively (this should be relatively fast to put together) if they need it.
2. (optional) Like @databus23 mentioned, provide a brew recipe which will compile docker-machine from source natively. The official install would still be to curl a binary into $PATH, but this would be the "easy way out" for users having this issue. Any volunteers for this? Also, this won't mitigate the problem on Windows, I'm curious if @jeffmendoza has any ideas.
3. (optional) Provide a way to add a cert to machine's trusted cert list. There's security implications to this that have to be weighed carefully.

How does that sound to everyone? @hairyhenderson @databus23 which sounds best to you?

[databus23]

I think a brew recipe is the best short term solution for OSX from the user perspective. The only hassle is that this also needs to be updated/published for every new release otherwise users who need custom certificates will be stuck on older versions.

@nathanleclaire I'm not sure this problem exists for cross compiled windows binaries.

[databus23]

@nathanleclaire I agree that changing the build progress is not a good idea. Its really slick the way it is. That being said I still think that the solution proposed in the linked blog post would be better solution in the long run. It still allows for cross compiling with gox in a container but without the limitations cross-compiling usually brings regarding the go stdlib.

In general you guys should be aware that shipping cross-compiled darwin binaries that do anything with https might blow up for users which use certificates Apple didn't put in the SystemRoots Keychain.

[hairyhenderson]

@nathanleclaire - if there's a way to add a cert to machine's trusted cert list, would it be possible to get machine to add /Libraries/Keychains/System.keychain by default? Or is that not a good idea (or impossible) for some reason?

[databus23]

@hairyhenderson To my knowledge this is not possible without negative security implications. Using /usr/bin/security -a -p find-certificate /Libraries/Keychains/System.keychain to dump all certificates from the System.keychain is a security risk because not all certificates in the keychain are actually trusted. In fact you can have certificates in the system keychain which you explicitly distrusted. As far as I can tell there is no way of sorting out certificates by their trust settings using /usr/bin/security. The only viable way seems to be the Security Framework c-binding which is not available when cross-compiling.

[hairyhenderson]

@databus23 ah - I see... sounds like we need some other tool that's aware of trust settings.

[databus23]

@hairyhenderson I think the options outlined in my above post are better solutions then trying to reimplemented the certificate trust setting evaluation as done by the Security Framework of OSX. Specifically option 3. allows to still cross-compile using a docker container while having the exact same code path for x509 certificate handling as when compiling natively.

[md5]

@databus23 I was looking into this recently and I don't believe the security command provides a way to list certificates anyways. I just tried your command under Yosemite and it complained about -a being unsupported.

I think the only way to programmatically get a list of certificates is to go to the low-level SecItemCopyMatching function.

[databus23]

@md5 It sure does. This is how the go stdlib itself fetches the certificates for the SystemRootCertificates.keychain in a nocgo environment since v1.3.

[md5]

@databus23 Thanks. Looks like the command is actually /usr/bin/security find-certificate -a -p /Libraries/Keychains/System.keychain

I realize now that I was looking into enumerating passwords, not certificates.

[ehazlett]

@databus23 I like the idea of using gonative as it stays in the docker container build flow. The only thing that concerns me is the "Open Issue" of that it hasn't been tested on Windows but I think we could give it a shot.

[databus23]

@ehazlett I might be misunderstanding this, but I think this statement means cross-compiling with gonative directly on windows is not tested. In docker-machine's case the container build flow ensures we always compile on linux. The author of gonative writes in the accompanying blog post:

I’ve tested targeting Windows/Linux/Darwin and they all work correctly!

[ehazlett]

@databus23 correct. Ah ok -- perhaps I misunderstood -- I read it as "the cross compiled binary hasn't been tested on Windows". I did see the comment from the blog so I think it's worth it to try.

[FraBle]

@hairyhenderson, is there any solution now to retrieve a custom (corporate) boot2docker.iso from a server with https (and own company certificate)?

[databus23]

@FraBle Yes, just compile docker-machine natively. brew install docker-machine will do.

[FraBle]

@databus23, thanks for your super fast reply.
I tried the Homebrew version:

frank >>> brew install docker-machine
==> Downloading https://homebrew.bintray.com/bottles/docker-machine-0.6.0.el_capitan.bottle.tar.gz
Already downloaded: /Library/Caches/Homebrew/docker-machine-0.6.0.el_capitan.bottle.tar.gz
==> Pouring docker-machine-0.6.0.el_capitan.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d
==> Summary
🍺  /usr/local/Cellar/docker-machine/0.6.0: 5 files, 36.8M
frank >>> rm -rf /Library/Caches/Homebrew/docker-machine-0.6.0.el_capitan.bottle.tar.gz
frank >>> brew uninstall docker-machine
Uninstalling /usr/local/Cellar/docker-machine/0.6.0... (5 files, 36.8M)
frank >>> brew install docker-machine
==> Downloading https://homebrew.bintray.com/bottles/docker-machine-0.6.0.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring docker-machine-0.6.0.el_capitan.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d
==> Summary
🍺  /usr/local/Cellar/docker-machine/0.6.0: 5 files, 36.8M
frank >>> docker-machine create dev -d virtualbox --virtualbox-boot2docker-url https://<mycompany_github>/api/v3/repos/monsoon-docker/boot2docker-sap-iso/releases/latest --engine-registry-mirror https://mycompany_docker-mirror>
Running pre-create checks...
Creating machine...
Error creating machine: Error in driver during machine creation: Get https://<mycompany_github>/api/v3/repos/monsoon-docker/boot2docker-sap-iso/releases/latest: x509: certificate signed by unknown authority

The certificate for out Github is imported in the System Keychain.

ps: in terms of dependencies:

==> Dependencies
Build: go ✔, automake ✔

[databus23]

@FraBle Thats odd. It should work if you have the SAP Global Root CA in your keychain. Are you sure you are invoking the right docker-machine binary? What does which docker-machine say? We can continue this in-house just ping me via Lync. :)

[nathanleclaire]

@FraBle What is output of which docker-machine? Of ls -lah $(which docker-machine)?

[FraBle]

@nathanleclaire , @databus23

frank >>> which docker-machine
/usr/local/bin/docker-machine

frank >>> ls -lah $(which docker-machine)
lrwxr-xr-x 1 <myuser> 49 Feb 29 14:42 /usr/local/bin/docker-machine -> ../Cellar/docker-machine/0.6.0/bin/docker-machine

[nathanleclaire]

Yup, looks symlinked to the Homebrew version correctly.

[databus23]

So this definitely should work. I'm using the same homebrew version against the same internal github enterprise installation. Has to be something with your local setup.

[FraBle]

@databus23, you have mail in your corporate mailbox :)

[databus23]

@FraBle I'm on CET so i'll look into tomorrow. Cheers

[FraBle]

@databus23 👍

[hairyhenderson]

Wow, this was a blast from the past 😉 -- makes me thankful I'm no longer working for the company I was at when I logged this 😁

Seriously though, it seems we probably need to update some docs somewhere to cover this use-case...

[arunanc]

This issue has nothing to do with homebrew. It's how 'go' reads certificates.

Refer this and this.

Solution is to import your corporate root certificate into SystemRootCertificates.keychain.

@databus23 @FraBle Ping me via Lync if you need some inputs. ;)