Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

cmd/go: deadlock occurs if GOCACHEPROG program exits after receiving "close" command without sending response back #70848

Open
xakep666 opened this issue Dec 14, 2024 · 4 comments
Labels
GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Comments

@xakep666
Copy link

Go version

go version devel go1.24-e39e965 Fri Dec 13 15:07:27 2024 -0800 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE='on'
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/xakep666/.cache/go-build'
GODEBUG=''
GOENV='/home/xakep666/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build381408197=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/home/xakep666/sources/buildcacher/go.mod'
GOMODCACHE='/home/xakep666/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/xakep666/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/xakep666/sdk/gotip'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/xakep666/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/xakep666/sdk/gotip/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.24-e39e965 Fri Dec 13 15:07:27 2024 -0800'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I'm trying to play with GOCACHEPROG. I've compiled following code

package main

import (
	"encoding/json"
	"errors"
	"flag"
	"fmt"
	"io"
	"os"
)

var declareClose = flag.Bool("declare-close", false, "declare close command")

func main() {
	flag.Parse()
	enc := json.NewEncoder(os.Stdout)
	dec := json.NewDecoder(os.Stdin)

	// send handshake
	cmds := []string{"get"}
	if *declareClose {
		cmds = append(cmds, "close")
	}
	if err := enc.Encode(Response{ID: 0, KnownCommands: cmds}); err != nil {
		panic(err)
	}

	for {
		var req Request
		if err := dec.Decode(&req); err != nil {
			if errors.Is(err, io.EOF) {
				fmt.Fprintln(os.Stderr, "EOF bye!")
				return
			}

			panic(err)
		}

		if req.Command == "close" {
			fmt.Fprintln(os.Stderr, "close bye!")
			return
		}

		if req.Command != "get" {
			if err := enc.Encode(Response{ID: req.ID, Err: "unknown command"}); err != nil {
				panic(err)
			}
			continue
		}

		if err := enc.Encode(Response{ID: req.ID, Miss: true}); err != nil {
			panic(err)
		}
	}
}

type Request struct {
	ID      int
	Command string
}

type Response struct {
	ID            int
	Err           string   `json:",omitempty"`
	KnownCommands []string `json:",omitempty"`
	Miss          bool     `json:",omitempty"`
}

using go build -o testcacheprog main.go.

And I'm trying to use resulting program like this GOCACHEPROG="$(pwd)/testcacheprog" go build std" and like this GOCACHEPROG="$(pwd)/testcacheprog -declare-close" go build std".

What did you see happen?

>  GOCACHEPROG="$(pwd)/testcacheprog" go install std                                                                                                                                      
EOF bye!

exit code 0

>  GOCACHEPROG="$(pwd)/testcacheprog -declare-close" go install std                                                                                                                 
close bye!
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [select]:
cmd/go/internal/cache.(*ProgCache).send(0xc0002cb720, {0xc80848, 0xc0001480a0}, 0xc000279100)
        /home/xakep666/sdk/gotip/src/cmd/go/internal/cache/prog.go:269 +0xd0
cmd/go/internal/cache.(*ProgCache).Close(0xc0002cb720)
        /home/xakep666/sdk/gotip/src/cmd/go/internal/cache/prog.go:436 +0x85
cmd/go/internal/work.(*Builder).Do.func1()
        /home/xakep666/sdk/gotip/src/cmd/go/internal/work/exec.go:81 +0x1c
cmd/go/internal/work.(*Builder).Do(0xc00024b440, {0xc807d8, 0x10b9f60}, 0xc000b1ef20)
        /home/xakep666/sdk/gotip/src/cmd/go/internal/work/exec.go:238 +0x489
cmd/go/internal/work.InstallPackages({0xc807d8, 0x10b9f60}, {0xc000020230, 0x1, 0x1}, {0xc000d31308, 0x158, 0x25f})
        /home/xakep666/sdk/gotip/src/cmd/go/internal/work/build.go:836 +0x9fb
cmd/go/internal/work.runInstall({0xc807d8, 0x10b9f60}, 0xc00002c810?, {0xc000020230, 0x1, 0x1})
        /home/xakep666/sdk/gotip/src/cmd/go/internal/work/build.go:739 +0x2af
main.invoke(0x108abe0, {0xc000020220, 0x2, 0x2})
        /home/xakep666/sdk/gotip/src/cmd/go/main.go:341 +0x845
main.main()
        /home/xakep666/sdk/gotip/src/cmd/go/main.go:220 +0xe8b

goroutine 31 [select]:
os/exec.(*Cmd).watchCtx(0xc00012c300, 0xc0000322a0)
        /home/xakep666/sdk/gotip/src/os/exec/exec.go:789 +0xb2
created by os/exec.(*Cmd).Start in goroutine 19
        /home/xakep666/sdk/gotip/src/os/exec/exec.go:775 +0x8f3

exit code 2

What did you expect to see?

>  GOCACHEPROG="$(pwd)/testcacheprog" go install std                                                                                                                                      
EOF bye!

exit code 0

>  GOCACHEPROG="$(pwd)/testcacheprog -declare-close" go install std                                                                                                                 
close bye!

exit code 0

@xakep666
Copy link
Author

xakep666 commented Dec 14, 2024

Addition: sending response to "close" command actually eliminates this issue.

Program

package main

import (
	"encoding/json"
	"errors"
	"flag"
	"fmt"
	"io"
	"os"
)

var declareClose = flag.Bool("declare-close", false, "declare close command")

func main() {
	flag.Parse()
	enc := json.NewEncoder(os.Stdout)
	dec := json.NewDecoder(os.Stdin)

	// send handshake
	cmds := []string{"get"}
	if *declareClose {
		cmds = append(cmds, "close")
	}
	if err := enc.Encode(Response{ID: 0, KnownCommands: cmds}); err != nil {
		panic(err)
	}

	for {
		var req Request
		if err := dec.Decode(&req); err != nil {
			if errors.Is(err, io.EOF) {
				fmt.Fprintln(os.Stderr, "EOF bye!")
				return
			}

			panic(err)
		}

		if req.Command == "close" {
			if err := enc.Encode(Response{ID: req.ID}); err != nil {
				panic(err)
			}
			fmt.Fprintln(os.Stderr, "close bye!")
			return
		}

		if req.Command != "get" {
			if err := enc.Encode(Response{ID: req.ID, Err: "unknown command"}); err != nil {
				panic(err)
			}
			continue
		}

		if err := enc.Encode(Response{ID: req.ID, Miss: true}); err != nil {
			panic(err)
		}
	}
}

type Request struct {
	ID      int
	Command string
}

type Response struct {
	ID            int
	Err           string   `json:",omitempty"`
	KnownCommands []string `json:",omitempty"`
	Miss          bool     `json:",omitempty"`
}

Result:

> GOCACHEPROG="$(pwd)/testcacheprog -declare-close" go install std                                                                                                                           
close bye!

exit code 0

@xakep666 xakep666 changed the title cmd/go: deadlock occurs if GOCACHEPROG program exits after receiving "close" command cmd/go: deadlock occurs if GOCACHEPROG program exits after receiving "close" command without sending response back Dec 14, 2024
@dr2chase
Copy link
Contributor

@matloob @rsc @samthanawalla
PTAL? I can't tell if this is a bug, a documentation problem leading to pilot error, or WAI.

@dr2chase dr2chase added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 18, 2024
@matloob matloob added the GoCommand cmd/go label Dec 18, 2024
@matloob
Copy link
Contributor

matloob commented Dec 18, 2024

cc @bradfitz

I think this is a bug.

If we get an EOF (or any other error) reading the stdout of the cacheprog, we usually print an error if there are requests we haven't received responses for... unless we are mid-close. If we're mid-close, we quietly drop the error. The problem with that is that we then close all the channels we'd send the responses to, and the select waiting for the response from the close request's channel has no viable branches leading to our deadlock.

There are a couple options for how to deal with this

  • We could keep the quiet exit and just do nothing if we don't get a response to the close response, which could be done by closing the prog cache's context when the EOF is received,
  • We could stop quietly dropping the error mid-close if there are requests we haven't gotten responses to, and complain that we exited "mid-Close" with pending requests.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

4 participants