code coverage: enable -covermode=atomic for test coverage

[jakdept]

Given the example go code:

package example

// filename example.go

package example

func example(letter string) string {
	switch letter {
	case "a":
		return "apple"
	case "b":
		return "banana"
	case "c":
		return "carrot"
	case "d":
		return "date"
	case "e":
		return "eggplant"
	case "f":
		return "fig"
	default:
		return "nevermind"
	}
}

package example

//filename example_test.go

import "testing"

func TestExample(t *testing.T) {
	for in, out := range map[string]string{
		"a": "apple",
		"b": "banana",
		"c": "carrot",
		"d": "date",
		"e": "eggplant",
		"f": "fig",
		"":  "nevermind",
	} {
		t.Run("input-"+in, func(t *testing.T) {
			t.Parallel()
			result := example(in)
			if result != out {
				t.Errorf("sent %s and got %s expected %s", in, result, out)
			}
		})
	}
}

This can then be tested from CLI with go test and you can see coverage with something like:

go test -covermode=atomic -coverprofile=${TMPDIR}/gotc.cover ./...
go tool cover -html=${TMPDIR}/gotc.cover

Because this test uses testing.T.Parallel(), if you don't run go test in atomic mode, it will produce a coverage report that does not track all parallel test branches.

VScode also displays this coverage in the editor. When doing so, it logs that it's running the following command:

Running tool: /usr/local/go/bin/go test -timeout 30s -coverprofile=<some-temp-dir>/go-code-cover

Notably, it does not allow you to configure the -covermode flag, and per https://golang.org/pkg/cmd/go/internal/test/ the default is set, which does not properly follow parallel tests.

Of note, it appears that line 16755 of dist/goMain.js is building the go test command from above, and vscode-go is running the command (from what I can tell), there's just no -covermode flag option.

It'd be nice to enable parallel testing code coverage, as golang supports it, by adding a -covermode flag, and making it a configurable option? Possibly also add a configurable option to package.json for go.testCoverMode and I'd suggest atomic as a default option as well? I'm just rubbish with typescript, so modifying the extension is likely beyond me.

Screenshots or recordings

Tests showing incorrect coverage (due to lack of -covermode flag and parallel test):

Test showing correct coverage (but without parallel testing):

General environment information:

• Run code -v or code-insiders -v to get version of VS Code or VS Code Insiders

╰─> code -v
1.48.2
a0479759d6e9ea56afa657e454193f72aef85bd0
x64

• Check your installed extensions to get the version of the VS Code Go extension

╰─> code --list-extensions --show-versions|grep golang
golang.go@0.16.1

• Run go version to get version of Go

╰─> go version
go version go1.15 darwin/amd64

• Run go env to get the go development environment details

╰─> go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jhayhurst/Library/Caches/go-build"
GOENV="/Users/jhayhurst/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/jhayhurst/pkg/mod"
GONOPROXY="git.liquidweb.com"
GONOSUMDB="git.liquidweb.com"
GOOS="darwin"
GOPATH="/Users/jhayhurst"
GOPRIVATE="git.liquidweb.com"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/w3/4010chss5832ggzxpfwj54qh0000gn/T/go-build620958345=/tmp/go-build -gno-record-gcc-switches -fno-common"

• Share the Go related settings you have added/edited

	"go.formatTool": "goreturns",
	"go.buildOnSave": "package",
	"go.lintOnSave": "package",
	"go.lintTool": "golangci-lint",
	"go.lintFlags": [
		"--disable-all",
		"--enable=deadcode",
		"--enable=errcheck",
		"--enable=gosimple",
		"--enable=govet",
		"--enable=golint",
		"--enable=ineffassign",
		"--enable=staticcheck",
		"--enable=structcheck",
		"--enable=typecheck",
		"--enable=unused",
		"--enable=varcheck",
		"--enable=interfacer",
		"--enable=gosec",
		"--enable=dupl",
		"--enable=goconst",
		"--enable=goimports",
		"--enable=misspell",
		"--enable=unparam",
	],
	"go.vetOnSave": "package",
	"go.coverOnSave": true,
	"go.toolsEnvVars": {
		"CGO_ENABLED": 1
	},
	"go.useLanguageServer": true,

[hyangah]

@pjweinbgo added a new setting go.coverMode recently and that feature is available in the nightly version of this extension. Please try the nightly version (installation instruction). Feedback is greatly appreciated.

[jakdept]

The nightly extension looks to correct the issue, but I usually stick with stable tools when possible.

I see the release cadence for the nightly version and pre-releases in docs/nightly.md - is there a defined expected release window for 0.17.x of this extension? Is that documented somewhere and I'm just missing it?

[hyangah]

@jakdept I tentatively set the deadline for v0.17.0 milestone to be mid September. I hoped to run monthly new feature releases, but there were not many new features for 0.17.x, so it was a bit delayed.