Log pkg/errors stack traces

[MichaelSnowden]

What changed?
I upgraded our zap version from v1.24.0 to v1.26.0, which contains support for pkg/errors. See this issue and this commit.

Why?
Before this change, our logs would only contain the stack trace from where the logger itself was invoked, not from the source of where the error was generated or wrapped. This provided very little useful information.

How did you test it?
I ran a custom build of server which overwrote some deep code in history to return an error. I then set up docker-compose to output logs -> promtail -> loki -> grafana. Then, I queried Grafana to verify that the error log contained an "errorVerbose" field with the stack trace from where my error was generated. As you can see from the below image, the stack trace does appear under this field, and if you turn on JSON parsing and newline escaping, you can both see it rendered correctly, and you can copy-paste the stack trace.

The stack trace from Grafana:

oopsie woopsie
main.(*faultyShardEngine).StartWorkflowExecution
	/Users/mikey/src/temporalio/temporal/.scratches/main.go:39
go.temporal.io/server/service/history.(*Handler).StartWorkflowExecution
	/Users/mikey/src/temporalio/temporal/service/history/handler.go:595
go.temporal.io/server/api/historyservice/v1._HistoryService_StartWorkflowExecution_Handler.func1
	/Users/mikey/src/temporalio/temporal/api/historyservice/v1/service.pb.go:1300
...

Potential risks
The stack traces are pretty deep because of all our gRPC interceptors. However, we can definitely fix that later if we want by filtering the pkg/errors.StackTrace. I'd rather do that in a follow-up after getting support for this initial change first, though.

Is hotfix candidate?
No.

[stephanos]

❤️