skopeo copy fails with segmentation violation (static container build)

[fhemberger]

I built skopeo on a fresh Ubuntu 18.04 using Docker:

Steps to reproduce:

sudo apt install make git
git clone https://github.com/containers/skopeo
make binary-static DISABLE_CGO=1
sudo make install-binary

I'm trying to copy a simple Docker image from the official registry to a private registry:

skopeo --insecure-policy copy \
  --src-no-creds --dest-no-creds --dest-tls-verify=false \
  docker://busybox:latest \
  docker://my.private.registry.net/docker/busybox:latest

Unfortunately, I'm running into this error:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x63 pc=0x7f5308089448]

runtime stack:
runtime.throw(0xc50c1c, 0x2a)
	/usr/lib/go-1.10/src/runtime/panic.go:616 +0x81
runtime.sigpanic()
	/usr/lib/go-1.10/src/runtime/signal_unix.go:372 +0x28e

goroutine 30 [syscall]:
runtime.cgocall(0x9f5d80, 0xc4200455f8, 0x29)
	/usr/lib/go-1.10/src/runtime/cgocall.go:128 +0x64 fp=0xc4200455b8 sp=0xc420045580 pc=0x4033d4
net._C2func_getaddrinfo(0xc42039c150, 0x0, 0xc420214630, 0xc42009a080, 0x0, 0x0, 0x0)
	_cgo_gotypes.go:92 +0x55 fp=0xc4200455f8 sp=0xc4200455b8 pc=0x5cca15
net.cgoLookupIPCNAME.func1(0xc42039c150, 0x0, 0xc420214630, 0xc42009a080, 0xf, 0xf, 0xc42009e180)
	/usr/lib/go-1.10/src/net/cgo_unix.go:149 +0x13b fp=0xc420045640 sp=0xc4200455f8 pc=0x5d370b
net.cgoLookupIPCNAME(0xc42053a4c0, 0xe, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/lib/go-1.10/src/net/cgo_unix.go:149 +0x174 fp=0xc420045738 sp=0xc420045640 pc=0x5ce084
net.cgoIPLookup(0xc42039ac00, 0xc42053a4c0, 0xe)
	/usr/lib/go-1.10/src/net/cgo_unix.go:201 +0x4d fp=0xc4200457c8 sp=0xc420045738 pc=0x5ce74d
runtime.goexit()
	/usr/lib/go-1.10/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4200457d0 sp=0xc4200457c8 pc=0x457d41
created by net.cgoLookupIP
	/usr/lib/go-1.10/src/net/cgo_unix.go:211 +0xaf

goroutine 1 [select]:
net/http.(*Transport).getConn(0xc4200b22d0, 0xc4202144b0, 0x0, 0xc4204cc328, 0x5, 0xc42053a4c0, 0x12, 0x0, 0x0, 0x0)
	/usr/lib/go-1.10/src/net/http/transport.go:962 +0x558
net/http.(*Transport).RoundTrip(0xc4200b22d0, 0xc4204d0700, 0xc4200b22d0, 0x0, 0x0)
	/usr/lib/go-1.10/src/net/http/transport.go:409 +0x632
net/http.send(0xc4204d0700, 0xcc1de0, 0xc4200b22d0, 0x0, 0x0, 0x0, 0xc42009a050, 0x42aad4, 0xc420212908, 0x1)
	/usr/lib/go-1.10/src/net/http/client.go:252 +0x185
net/http.(*Client).send(0xc4202143f0, 0xc4204d0700, 0x0, 0x0, 0x0, 0xc42009a050, 0x0, 0x1, 0xc420212990)
	/usr/lib/go-1.10/src/net/http/client.go:176 +0xfa
net/http.(*Client).Do(0xc4202143f0, 0xc4204d0700, 0xc420212ba0, 0x2, 0x2)
	/usr/lib/go-1.10/src/net/http/client.go:615 +0x28d
github.com/containers/skopeo/vendor/github.com/containers/image/docker.(*dockerClient).getBearerToken(0xc4200b20f0, 0xcc89a0, 0xc420098020, 0xc42039c110, 0x6, 0xc420214330, 0xc420212d68, 0x1, 0x1, 0x0, ...)
	/src/github.com/containers/skopeo/vendor/github.com/containers/image/docker/docker_client.go:530 +0x629
github.com/containers/skopeo/vendor/github.com/containers/image/docker.(*dockerClient).setupRequestAuth(0xc4200b20f0, 0xc4204d0400, 0x0, 0xc58836, 0x39)
	/src/github.com/containers/skopeo/vendor/github.com/containers/image/docker/docker_client.go:481 +0x48e
github.com/containers/skopeo/vendor/github.com/containers/image/docker.(*dockerClient).makeRequestToResolvedURL(0xc4200b20f0, 0xcc89a0, 0xc420098020, 0xc3542f, 0x3, 0xc420024740, 0x40, 0xc4202131f0, 0x0, 0x0, ...)
	/src/github.com/containers/skopeo/vendor/github.com/containers/image/docker/docker_client.go:437 +0x3d1
github.com/containers/skopeo/vendor/github.com/containers/image/docker.(*dockerClient).makeRequest(0xc4200b20f0, 0xcc89a0, 0xc420098020, 0xc3542f, 0x3, 0xc42035f8f0, 0x24, 0xc4202131f0, 0x0, 0x0, ...)
	/src/github.com/containers/skopeo/vendor/github.com/containers/image/docker/docker_client.go:411 +0x25a
github.com/containers/skopeo/vendor/github.com/containers/image/docker.(*dockerImageSource).fetchManifest(0xc420460b80, 0xcc89a0, 0xc420098020, 0xc4203342fa, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/src/github.com/containers/skopeo/vendor/github.com/containers/image/docker/docker_image_src.go:170 +0x2b3
github.com/containers/skopeo/vendor/github.com/containers/image/docker.(*dockerImageSource).ensureManifestIsLoaded(0xc420460b80, 0xcc89a0, 0xc420098020, 0x0, 0xc3596c)
	/src/github.com/containers/skopeo/vendor/github.com/containers/image/docker/docker_image_src.go:202 +0xa0
github.com/containers/skopeo/vendor/github.com/containers/image/docker.newImageSource(0xcc89a0, 0xc420098020, 0xc42008e420, 0xcc5d60, 0xc42038aa60, 0xc42009c140, 0x0, 0x453740)
	/src/github.com/containers/skopeo/vendor/github.com/containers/image/docker/docker_image_src.go:114 +0x3ea
github.com/containers/skopeo/vendor/github.com/containers/image/docker.dockerReference.NewImageSource(0xcc5d60, 0xc42038aa60, 0xcc89a0, 0xc420098020, 0xc42008e420, 0xc42009c170, 0xc4204bf600, 0x18, 0xc4204bf5f8)
	/src/github.com/containers/skopeo/vendor/github.com/containers/image/docker/docker_transport.go:144 +0x57
github.com/containers/skopeo/vendor/github.com/containers/image/copy.Image(0xcc89a0, 0xc420098020, 0xc42038aa00, 0xcce320, 0xc420464a80, 0xcce320, 0xc420464a70, 0xc4202139b8, 0x0, 0x0, ...)
	/src/github.com/containers/skopeo/vendor/github.com/containers/image/copy/copy.go:151 +0x19e
main.(*copyOptions).run(0xc4204c04d0, 0xc4200a0060, 0x2, 0x2, 0xcc1fe0, 0xc42009a008, 0x0, 0x0)
	/src/github.com/containers/skopeo/cmd/skopeo/copy.go:150 +0xa51
main.(*copyOptions).(main.run)-fm(0xc4200a0060, 0x2, 0x2, 0xcc1fe0, 0xc42009a008, 0xc42039a060, 0x0)
	/src/github.com/containers/skopeo/cmd/skopeo/copy.go:52 +0x5c
main.commandAction.func1(0xc42008e2c0, 0x10100, 0xc42008e2c0)
	/src/github.com/containers/skopeo/cmd/skopeo/utils.go:26 +0x6b
github.com/containers/skopeo/vendor/github.com/urfave/cli.HandleAction(0xb5c4a0, 0xc42002e340, 0xc42008e2c0, 0xc42039a000, 0x0)
	/src/github.com/containers/skopeo/vendor/github.com/urfave/cli/app.go:488 +0x101
github.com/containers/skopeo/vendor/github.com/urfave/cli.Command.Run(0xc35868, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc5370d, 0x2f, 0x0, ...)
	/src/github.com/containers/skopeo/vendor/github.com/urfave/cli/command.go:210 +0xa36
github.com/containers/skopeo/vendor/github.com/urfave/cli.(*App).Run(0xc4204d61a0, 0xc4200a0000, 0x8, 0x8, 0x0, 0x0)
	/src/github.com/containers/skopeo/vendor/github.com/urfave/cli/app.go:255 +0x6a0
main.main()
	/src/github.com/containers/skopeo/cmd/skopeo/main.go:124 +0x5c

goroutine 19 [syscall]:
os/signal.signal_recv(0x0)
	/usr/lib/go-1.10/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
	/usr/lib/go-1.10/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/lib/go-1.10/src/os/signal/signal_unix.go:28 +0x41

goroutine 28 [select]:
net.(*Resolver).LookupIPAddr(0x1078fd0, 0xcc89e0, 0xc42039ab40, 0xc42053a4c0, 0xe, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/lib/go-1.10/src/net/lookup.go:212 +0x50d
net.(*Resolver).internetAddrList(0x1078fd0, 0xcc89e0, 0xc42039ab40, 0xc3559a, 0x3, 0xc42053a4c0, 0x12, 0x0, 0x0, 0x0, ...)
	/usr/lib/go-1.10/src/net/ipsock.go:293 +0x5c4
net.(*Resolver).resolveAddrList(0x1078fd0, 0xcc89e0, 0xc42039ab40, 0xc3588c, 0x4, 0xc3559a, 0x3, 0xc42053a4c0, 0x12, 0x0, ...)
	/usr/lib/go-1.10/src/net/dial.go:193 +0x50c
net.(*Dialer).DialContext(0xc42039a960, 0xcc89a0, 0xc420098020, 0xc3559a, 0x3, 0xc42053a4c0, 0x12, 0x0, 0x0, 0x0, ...)
	/usr/lib/go-1.10/src/net/dial.go:375 +0x22b
net.(*Dialer).Dial(0xc42039a960, 0xc3559a, 0x3, 0xc42053a4c0, 0x12, 0x60, 0x110, 0x110, 0xc4202a9440)
	/usr/lib/go-1.10/src/net/dial.go:320 +0x75
github.com/containers/skopeo/vendor/golang.org/x/net/proxy.(Dialer).Dial-fm(0xc3559a, 0x3, 0xc42053a4c0, 0x12, 0xc42039aae0, 0xc420561ae8, 0x405246, 0x73c0a9)
	/src/github.com/containers/skopeo/vendor/github.com/containers/image/pkg/tlsclientconfig/tlsclientconfig.go:109 +0x5b
net/http.(*Transport).dial(0xc4200b22d0, 0xcc89a0, 0xc420098020, 0xc3559a, 0x3, 0xc42053a4c0, 0x12, 0x40bbfb, 0x44f37d, 0xb5ec00, ...)
	/usr/lib/go-1.10/src/net/http/transport.go:901 +0x78
net/http.(*Transport).dialConn(0xc4200b22d0, 0xcc89a0, 0xc420098020, 0x0, 0xc4204cc328, 0x5, 0xc42053a4c0, 0x12, 0x1, 0x0, ...)
	/usr/lib/go-1.10/src/net/http/transport.go:1143 +0x317
net/http.(*Transport).getConn.func4(0xc4200b22d0, 0xcc89a0, 0xc420098020, 0xc4202144e0, 0xc4204e2240)
	/usr/lib/go-1.10/src/net/http/transport.go:957 +0x78
created by net/http.(*Transport).getConn
	/usr/lib/go-1.10/src/net/http/transport.go:956 +0x363

goroutine 29 [select]:
net.cgoLookupIP(0xcc8960, 0xc420460b40, 0xc42053a4c0, 0xe, 0x0, 0x0, 0x0, 0x0, 0xc4203349a8, 0x2)
	/usr/lib/go-1.10/src/net/cgo_unix.go:212 +0x19f
net.(*Resolver).lookupIP(0x1078fd0, 0xcc8960, 0xc420460b40, 0xc42053a4c0, 0xe, 0x42d49b, 0xc420001980, 0xc42009e180, 0xc420081080, 0xc420045658)
	/usr/lib/go-1.10/src/net/lookup_unix.go:95 +0x12d
net.(*Resolver).(net.lookupIP)-fm(0xcc8960, 0xc420460b40, 0xc42053a4c0, 0xe, 0x42a1d9, 0xc400000008, 0xc42009e180, 0xc420460a40, 0xc4200456a0)
	/usr/lib/go-1.10/src/net/lookup.go:192 +0x56
net.glob..func10(0xcc8960, 0xc420460b40, 0xc420456200, 0xc42053a4c0, 0xe, 0x0, 0xc42009a098, 0xc42046b5c0, 0xc42039a6b8, 0xc42046b140)
	/usr/lib/go-1.10/src/net/hook.go:19 +0x52
net.(*Resolver).LookupIPAddr.func1(0x0, 0x0, 0x0, 0x0)
	/usr/lib/go-1.10/src/net/lookup.go:206 +0xd8
internal/singleflight.(*Group).doCall(0x1078fc0, 0xc420564190, 0xc42053a4c0, 0xe, 0xc4202145a0)
	/usr/lib/go-1.10/src/internal/singleflight/singleflight.go:95 +0x2e
created by internal/singleflight.(*Group).DoChan
	/usr/lib/go-1.10/src/internal/singleflight/singleflight.go:88 +0x2d0

I'm already getting warnings during make binary-static, I don't know if this is related somehow:

docker run --rm --security-opt label=disable -v $(pwd):/src/github.com/containers/skopeo \
	skopeobuildimage make binary-local-static  BUILDTAGS='containers_image_ostree_stub exclude_graphdriver_devicemapper exclude_graphdriver_btrfs containers_image_openpgp'
go build -ldflags "-extldflags \"-static\" -X main.gitCommit=6307635b5fc719e6e74588c7d04df449eb9557f7" -gcflags "" -tags "containers_image_ostree_stub exclude_graphdriver_devicemapper exclude_graphdriver_btrfs containers_image_openpgp" -o skopeo ./cmd/skopeo
# github.com/containers/skopeo/cmd/skopeo
/usr/bin/ld: /tmp/go-link-979354278/000005.o: in function `mygetgrouplist':
/build/golang-1.10-0Gb4kq/golang-1.10-1.10.4/src/os/user/getgrouplist_unix.go:15: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-979354278/000004.o: in function `mygetgrgid_r':
/build/golang-1.10-0Gb4kq/golang-1.10-1.10.4/src/os/user/cgo_lookup_unix.go:38: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-979354278/000004.o: in function `mygetgrnam_r':
/build/golang-1.10-0Gb4kq/golang-1.10-1.10.4/src/os/user/cgo_lookup_unix.go:43: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-979354278/000004.o: in function `mygetpwnam_r':
/build/golang-1.10-0Gb4kq/golang-1.10-1.10.4/src/os/user/cgo_lookup_unix.go:33: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-979354278/000004.o: in function `mygetpwuid_r':
/build/golang-1.10-0Gb4kq/golang-1.10-1.10.4/src/os/user/cgo_lookup_unix.go:28: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-979354278/000023.o: in function `_cgo_f7895c2c5a3a_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:46: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

[mtrmac]

Thanks for your report.

Yes, the warnings are correct, and basically the answer is not to use static or DISABLE_CGO builds if at all possible. Possibly see #670 .

[fhemberger]

Installing required packages on Ubuntu (see #672) and just using make build worked for me, thanks.

Still hoping you can fix the issue and/or provide binary builds. 😉

[vrothberg]

@rhatdan @mtrmac, since static builds have some serious drawbacks, I wonder if it would make sense to create images with a pre-installed Skopeo as @TomSweeneyRedHat did for Buildah and Podman.

[rhatdan]

Yes I think we it makes more sense to have a container image for Skopeo, then for Podman. Product Management actually wants these for RHEL8.