Remote debugging to Docker container gives empty values for all variables

[Strum355]

Go version: 1.12.6
vscode-go version: 0.11.0
Delve version: commit a7c2d837d5fe920fb7a08812cebd6981d0a09035

launch.json

{
	"version": "0.2.0",
    "configurations": [
        {
            "name": "Debug in Docker Container",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "cwd": "${workspaceFolder}/cmd/<repo_name>",
            "remotePath": "/go/src/github.com/<org>/<repo_name>/cmd/<repo_name>",
            "port": 3456,
            "host": "127.0.0.1",
            "showLog": true,
            "trace": "verbose",
            "logOutput": "debugger"
        }
    ]
}

delve command in Docker container
dlv debug ./cmd/<repo_name> -l 0.0.0.0:3456 --headless=true --api-version=2 --log

VSCode debugger:

Delve CLI:

$ dlv connect 127.0.0.1:3456 --wd /home/noah/<org>/<repo_name>/cmd/<repo_name> --api-version 2

Type 'help' for list of commands.
(dlv) b /go/src/github.com/<org>/<repo_name>/cmd/<repo_name>/main.go:54
Breakpoint 1 set at 0x10bd507 for main.main() /go/src/github.com/<org>/<repo_name>/cmd/<repo_name>/main.go:54
(dlv) c
> main.main() /go/src/github.com/<org>/<repo_name>/cmd/<repo_name>/main.go:54 (hits goroutine(1):1 total:1) (PC: 0x10bd507)
(dlv) locals
buildID = "local-dev"
e = ("*github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo")(0xc000174000)
(dlv) print e
*github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo {
        StdLogger: *log.Logger {
                mu: (*sync.Mutex)(0xc000222500),
                prefix: "echo: ",
                flag: 0,
                out: io.Writer(*os.File) ...,
                buf: []uint8 len: 0, cap: 0, nil,},
        colorer: *github.com/<org>/<repo_name>/vendor/github.com/labstack/gommon/color.Color {
                output: io.Writer(*os.File) ...,
                disabled: true,},
        premiddleware: []github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.MiddlewareFunc len: 0, cap: 0, nil,
        middleware: []github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.MiddlewareFunc len: 1, cap: 1, [github.com/<org>/<repo_name>/vendor/github.com/labstack/echo/middleware.RequestIDWithConfig.func1],
        maxParam: *0,
        router: *github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Router {
                tree: *(*"github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.node")(0xc0001761c0),
                routes: map[string]*github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Route [],
                echo: *(*"github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo")(0xc000174000),},
        notFoundHandler: nil,
        pool: sync.Pool {
                noCopy: sync.noCopy {},
                local: unsafe.Pointer(0x0),
                localSize: 0,
                New: github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.New.func1,},
        Server: *net/http.Server {
                Addr: "",
                Handler: net/http.Handler(*github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo) ...,
                TLSConfig: *crypto/tls.Config nil,
                ReadTimeout: 0,
                ReadHeaderTimeout: 0,
                WriteTimeout: 0,
                IdleTimeout: 0,
                MaxHeaderBytes: 0,
                TLSNextProto: map[string]func(*net/http.Server, *crypto/tls.Conn, net/http.Handler) nil,
                ConnState: nil,
                ErrorLog: *log.Logger nil,
                disableKeepAlives: 0,
                inShutdown: 0,
                nextProtoOnce: (*sync.Once)(0xc00040afe0),
                nextProtoErr: error nil,
                mu: (*sync.Mutex)(0xc00040b000),
                listeners: map[*net.Listener]struct {} nil,
                activeConn: map[*net/http.conn]struct {} nil,
                doneChan: chan struct {} {},
                onShutdown: []func() len: 0, cap: 0, nil,},
        TLSServer: *net/http.Server {
                Addr: "",
                Handler: net/http.Handler(*github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo) ...,
                TLSConfig: *crypto/tls.Config nil,
                ReadTimeout: 0,
                ReadHeaderTimeout: 0,
                WriteTimeout: 0,
                IdleTimeout: 0,
                MaxHeaderBytes: 0,
                TLSNextProto: map[string]func(*net/http.Server, *crypto/tls.Conn, net/http.Handler) nil,
                ConnState: nil,
                ErrorLog: *log.Logger nil,
                disableKeepAlives: 0,
                inShutdown: 0,
                nextProtoOnce: (*sync.Once)(0xc00040b0b0),
                nextProtoErr: error nil,
                mu: (*sync.Mutex)(0xc00040b0d0),
                listeners: map[*net.Listener]struct {} nil,
                activeConn: map[*net/http.conn]struct {} nil,
                doneChan: chan struct {} {},
                onShutdown: []func() len: 0, cap: 0, nil,},
        Listener: net.Listener nil,
        TLSListener: net.Listener nil,
        AutoTLSManager: github.com/<org>/<repo_name>/vendor/golang.org/x/crypto/acme/autocert.Manager {
                Prompt: github.com/<org>/<repo_name>/vendor/golang.org/x/crypto/acme/autocert.AcceptTOS,
                Cache: github.com/<org>/<repo_name>/vendor/golang.org/x/crypto/acme/autocert.Cache nil,
                HostPolicy: nil,
                RenewBefore: 0,
                Client: *github.com/<org>/<repo_name>/vendor/golang.org/x/crypto/acme.Client nil,
                Email: "",
                ForceRSA: false,
                ExtraExtensions: []crypto/x509/pkix.Extension len: 0, cap: 0, nil,
                clientMu: (*sync.Mutex)(0xc000174100),
                client: *github.com/<org>/<repo_name>/vendor/golang.org/x/crypto/acme.Client nil,
                stateMu: (*sync.Mutex)(0xc000174110),
                state: map[github.com/<org>/<repo_name>/vendor/golang.org/x/crypto/acme/autocert.certKey]*github.com/<org>/<repo_name>/vendor/golang.org/x/crypto/acme/autocert.certState nil,
                renewalMu: (*sync.Mutex)(0xc000174120),
                renewal: map[github.com/<org>/<repo_name>/vendor/golang.org/x/crypto/acme/autocert.certKey]*github.com/<org>/<repo_name>/vendor/golang.org/x/crypto/acme/autocert.domainRenewal nil,
                tokensMu: (*sync.RWMutex)(0xc000174130),
                tryHTTP01: false,
                httpTokens: map[string][]uint8 nil,
                certTokens: map[string]*crypto/tls.Certificate nil,
                nowFunc: nil,},
        DisableHTTP2: false,
        Debug: false,
        HideBanner: true,
        HidePort: false,
        HTTPErrorHandler: github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.(*Echo).DefaultHTTPErrorHandler-fm,
        Binder: github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Binder(*github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.DefaultBinder) *{},
        Validator: github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Validator nil,
        Renderer: github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Renderer nil,
        Logger: github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Logger(*github.com/<org>/<repo_name>/vendor/github.com/labstack/gommon/log.Logger) *{
                prefix: "echo",
                level: ERROR (4),
                skip: 2,
                output: io.Writer(*os.File) ...,
                template: *(*"github.com/<org>/<repo_name>/vendor/github.com/valyala/fasttemplate.Template")(0xc00014a000),
                levels: []string len: 8, cap: 8, ["-","DEBUG","INFO","WARN","ERROR","","PANIC","FATAL"],
                color: *(*"github.com/<org>/<repo_name>/vendor/github.com/labstack/gommon/color.Color")(0xc0001ea700),
                bufferPool: (*sync.Pool)(0xc0003c5058),
                mutex: (*sync.Mutex)(0xc0003c5070),},}
(dlv) 

VSCode Debug Console

ScopesRequest
locals [{"name":"buildID","addr":824635686480,"onlyAddr":false,"type":"string","realType":"string","flags":0,"kind":24,"value":"","len":9,"cap":0,"children":[],"base":20013314,"unreadable":"","LocationExpr":"[block] DW_OP_fbreg -0x148 ","DeclLine":53},{"name":"e","addr":824635686376,"onlyAddr":false,"type":"*github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo","realType":"*github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo","flags":0,"kind":22,"value":"","len":1,"cap":0,"children":[{"name":"","addr":824635244544,"onlyAddr":true,"type":"github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo","realType":"github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo","flags":0,"kind":25,"value":"","len":0,"cap":0,"children":[],"base":0,"unreadable":"","LocationExpr":"","DeclLine":0}],"base":0,"unreadable":"","LocationExpr":"[block] DW_OP_fbreg -0x1b0 ","DeclLine":46}]
functionArgs []
global vars [{"name":"statictmp_0","addr":21899856,"onlyAddr":false,"type":"string","realType":"string","flags":0,"kind":24,"value":"","len":17,"cap":0,"children":[],"base":20042789,"unreadable":"","LocationExpr":"[block] DW_OP_addr 0x14e2a50 ","DeclLine":0}]
To client: {"seq":0,"type":"response","request_seq":12,"command":"scopes","success":true,"body":{"scopes":[{"name":"Local","variablesReference":1000,"expensive":false},{"name":"Global","variablesReference":1001,"expensive":false}]}}
ScopesResponse
From client: variables({"variablesReference":1000})
VariablesRequest
To client: {"seq":0,"type":"response","request_seq":13,"command":"variables","success":true,"body":{"variables":[{"name":"buildID","value":"\"\"","evaluateName":"buildID","variablesReference":0},{"name":"e","value":"<*github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo>(0xc000174000)","evaluateName":"e","variablesReference":1002}]}}
VariablesResponse [
 {
  "name": "buildID",
  "value": "\"\"",
  "evaluateName": "buildID",
  "variablesReference": 0
 },
 {
  "name": "e",
  "value": "<*github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo>(0xc000174000)",
  "evaluateName": "e",
  "variablesReference": 1002
 }
]
From client: variables({"variablesReference":1002})
VariablesRequest
To client: {"seq":0,"type":"response","request_seq":14,"command":"variables","success":true,"body":{"variables":[{"name":"","value":"<github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo>","evaluateName":"e","variablesReference":0}]}}
VariablesResponse [
 {
  "name": "",
  "value": "<github.com/<org>/<repo_name>/vendor/github.com/labstack/echo.Echo>",
  "evaluateName": "e",
  "variablesReference": 0
 }
]
From client: variables({"variablesReference":1001})
VariablesRequest
To client: {"seq":0,"type":"response","request_seq":15,"command":"variables","success":true,"body":{"variables":[{"name":"statictmp_0","value":"\"\"","variablesReference":0}]}}
VariablesResponse [
 {
  "name": "statictmp_0",
  "value": "\"\"",
  "variablesReference": 0
 }
]

[ob1dev]

@Strum355 I've got the same issue today but attaching to remote pid on a local machine.

Start DLV like this:

PS> dlv debug --headless --listen=:2345 --api-version=2

In VS Code, attach using the following config:

"name": "Attach to Process",
"type": "go",
"request": "launch",
"mode": "remote",
"remotePath": "${workspaceRoot}/main.go",
"port": 2345,
"host": "127.0.0.1",
"program": "${workspaceRoot}/main.go",
"env": {},
"apiVersion": 2,
"dlvLoadConfig": {
    "followPointers": true,
    "maxVariableRecurse": 1,
    "maxStringLen": 1000,
    "maxArrayValues": 1000,
    "maxStructFields": -1
}

As far as I can see, it started happening after VS Code automatically had installed Go extension v0.11.0. If I downgrade to v0.10.2 it works fine.

[Strum355]

Can confirm v0.10.2 works, so theres a pretty big regression in v0.11.0 :/ v0.11.0 seems to be down as an update to the debugging side of the extension so that seems likely

[ramya-rao-a]

@Strum355 @olegburov Is this only when attaching to a headless delve instance? Are the variables showing up fine in the case of starting the program as part of debugging itself? i.e when request is set to launch and mode is debug?

6a77627 is the only change around debugging that was made in v0.11.0. None of the change was around how the variables are displayed though.

@Strum355 In your logs, under "VariablesResponse", the value for buildID shows as empty string. When you change to v0.10.2, the logs show the right value at that point?

[ramya-rao-a]

@olegburov Are you using the same debug configuration both in v0.10.2 and v0.11.0?

[ob1dev]

@ramya-rao-a, yes I used the same between v0.10.2 and v0.11.0. Please see my example ☝️. Should it be different for v0.11.0?

[ramya-rao-a]

I was just asking to narrow down on what are the changes between the 2 attempts you had made other than changing the Go extension version.

I am able to repro the same and see the empty values being returned from delve. Will dig deeper into what changed to get this difference between the 2 versions

[ramya-rao-a]

I found the issue. While refactoring the code around starting delve in 6a77627, we had an early exit in case of remote debugging (as the extension doesn't have to start delve in this case). The part where we set the dlvLoadConfig was missed for remote scenarios due to the early exit

I have pushed a fix for it, can either of you try it out by using the latest beta version of this extension?

[ob1dev]

@ramya-rao-a, thanks a lot for looking into this. With v0.11.1-beta.3 it starts working for me.

[ramya-rao-a]

Thanks for confirming @olegburov!

This issue is now fixed in the latest update (0.11.1) to the Go extension

[rv4242]

I think I'm still seeing a problem with 0.11.1
I'm debugging some httpexpect tests. See the screenshot.
elements[0].data.name.data == ""
yet e.data.name.data has a value (and e is elements[0])

Note this isn't remote, so maybe I should open a new bug.

[rv4242]

And I see the same behaviour in 0.10.2.

[rv4242]

OK, it looks like the issue goes away if I set maxVariableRecurse to 3