Gaia-7001 consensus failure and seg fault at height 24570.

[joe-bowman]

Summary of Bug

At (or shortly after) height 24570, I received log messages stating CONSENSUS FAILURE!!! accompanied by the following stack trace:

{"log":"panic: runtime error: invalid memory address or nil pointer dereference\n","stream":"stderr","time":"2018-07-22T21:13:03.399133484Z"}
{"log":"[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x61c1a9]\n","stream":"stderr","time":"2018-07-22T21:13:03.399225074Z"}
{"log":"\n","stream":"stderr","time":"2018-07-22T21:13:03.399231744Z"}
{"log":"goroutine 1 [running]:\n","stream":"stderr","time":"2018-07-22T21:13:03.399234873Z"}
{"log":"math/big.(*Rat).Add(0xc42007a800, 0x0, 0xc42007a540, 0x0)\n","stream":"stderr","time":"2018-07-22T21:13:03.399237845Z"}
{"log":"\u0009/usr/local/go/src/math/big/rat.go:475 +0x29\n","stream":"stderr","time":"2018-07-22T21:13:03.399253834Z"}
{"log":"github.com/cosmos/cosmos-sdk/types.Rat.Add(0x0, 0xc42007a540, 0xc4209b6f80)\n","stream":"stderr","time":"2018-07-22T21:13:03.399322344Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/types/rational.go:144 +0x4f\n","stream":"stderr","time":"2018-07-22T21:13:03.399336778Z"}
{"log":"github.com/cosmos/cosmos-sdk/x/gov.tally.func2(0x0, 0xfcdf60, 0xc42007a780, 0xc42007a780)\n","stream":"stderr","time":"2018-07-22T21:13:03.399519397Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/x/gov/tally.go:52 +0x111\n","stream":"stderr","time":"2018-07-22T21:13:03.399528438Z"}
{"log":"github.com/cosmos/cosmos-sdk/x/stake/keeper.Keeper.IterateDelegations(0xfc6f60, 0xc420b584d0, 0xc4200d44d0, 0xfc6f60, 0xc420b584b0, 0xf52210, 0xc4200d44d0, 0x4, 0xfce660, 0xc422393140, ...)\n","stream":"stderr","time":"2018-07-22T21:13:03.399541622Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/x/stake/keeper/sdk_types.go:94 +0x30e\n","stream":"stderr","time":"2018-07-22T21:13:03.399545612Z"}
{"log":"github.com/cosmos/cosmos-sdk/x/gov.tally(0xfce660, 0xc422393140, 0xc422335540, 0x9, 0xfc6f60, 0xc420b584b0, 0xf52210, 0xc4200d44d0, 0xfd36c0, 0xc4200b36c0, ...)\n","stream":"stderr","time":"2018-07-22T21:13:03.399549084Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/x/gov/tally.go:50 +0x711\n","stream":"stderr","time":"2018-07-22T21:13:03.399552668Z"}
{"log":"github.com/cosmos/cosmos-sdk/x/gov.EndBlocker(0xfce660, 0xc422393140, 0xc422335540, 0x9, 0xfc6f60, 0xc420b584b0, 0xf52210, 0xc4200d44d0, 0xfd36c0, 0xc4200b36c0, ...)\n","stream":"stderr","time":"2018-07-22T21:13:03.399556098Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/x/gov/handler.go:118 +0x895\n","stream":"stderr","time":"2018-07-22T21:13:03.399559864Z"}
{"log":"github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker(0xc420a4c340, 0xfce660, 0xc422393140, 0xc422335540, 0x9, 0x5ffa, 0x0, 0x0, 0x0, 0x0, ...)\n","stream":"stderr","time":"2018-07-22T21:13:03.399609141Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:143 +0x140\n","stream":"stderr","time":"2018-07-22T21:13:03.399630336Z"}
{"log":"github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).EndBlocker-fm(0xfce660, 0xc422393140, 0xc422335540, 0x9, 0x5ffa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n","stream":"stderr","time":"2018-07-22T21:13:03.399634151Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:104 +0xa4\n","stream":"stderr","time":"2018-07-22T21:13:03.39963791Z"}
{"log":"github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc420a38240, 0x5ffa, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n","stream":"stderr","time":"2018-07-22T21:13:03.399641492Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:657 +0xc6\n","stream":"stderr","time":"2018-07-22T21:13:03.399645065Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client.(*localClient).EndBlockSync(0xc4200f9080, 0x5ffa, 0xc4223769e0, 0x0, 0x0)\n","stream":"stderr","time":"2018-07-22T21:13:03.399648441Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/abci/client/local_client.go:213 +0x85\n","stream":"stderr","time":"2018-07-22T21:13:03.39982724Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*appConnConsensus).EndBlockSync(0xc420039520, 0x5ffa, 0x14, 0x20, 0xc420aad8b0)\n","stream":"stderr","time":"2018-07-22T21:13:03.399835295Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/app_conn.go:77 +0x3d\n","stream":"stderr","time":"2018-07-22T21:13:03.399839106Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.execBlockOnProxyApp(0xfcec60, 0xc422099160, 0xfd31e0, 0xc420039520, 0xc422c42910, 0xc422320150, 0xfd7300, 0xc421b88030, 0x1, 0xc422322340, ...)\n","stream":"stderr","time":"2018-07-22T21:13:03.399849419Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:210 +0x7b5\n","stream":"stderr","time":"2018-07-22T21:13:03.399853243Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc4243bc340, 0xc420aad3a0, 0x9, 0x5ff9, 0x7d7, 0xc422322300, 0x14, 0x14, 0x1, 0xc422322340, ...)\n","stream":"stderr","time":"2018-07-22T21:13:03.399863559Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/execution.go:76 +0x12f\n","stream":"stderr","time":"2018-07-22T21:13:03.399867547Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(0xc420a4b400, 0xc420aad3a0, 0x9, 0x5ff9, 0x7d7, 0xc422322300, 0x14, 0x14, 0x1, 0xc422322340, ...)\n","stream":"stderr","time":"2018-07-22T21:13:03.399871076Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/replay.go:414 +0x23a\n","stream":"stderr","time":"2018-07-22T21:13:03.399874898Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(0xc420a4b400, 0xc420aad3a0, 0x9, 0x5ff9, 0x7d7, 0xc422322300, 0x14, 0x14, 0x1, 0xc422322340, ...)\n","stream":"stderr","time":"2018-07-22T21:13:03.3998995Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/replay.go:345 +0x805\n","stream":"stderr","time":"2018-07-22T21:13:03.399904273Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc420a4b400, 0xfd7d80, 0xc420a50300, 0x4, 0xfcec60)\n","stream":"stderr","time":"2018-07-22T21:13:03.399907887Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/replay.go:246 +0x3e5\n","stream":"stderr","time":"2018-07-22T21:13:03.399911462Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy.(*multiAppConn).OnStart(0xc420a50300, 0xc42209a7e0, 0x15)\n","stream":"stderr","time":"2018-07-22T21:13:03.399939008Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/proxy/multi_app_conn.go:108 +0x527\n","stream":"stderr","time":"2018-07-22T21:13:03.39994443Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/libs/common.(*BaseService).Start(0xc420a50300, 0x0, 0x0)\n","stream":"stderr","time":"2018-07-22T21:13:03.399948041Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/libs/common/service.go:130 +0x3bd\n","stream":"stderr","time":"2018-07-22T21:13:03.399951295Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/node.NewNode(0xc420acb8c0, 0xfcfea0, 0xc4200f74d0, 0xfc3b40, 0xc4223252a0, 0xc420b59ac0, 0xf51c08, 0xf51c10, 0xfcec60, 0xc4223252e0, ...)\n","stream":"stderr","time":"2018-07-22T21:13:03.399954409Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/node/node.go:187 +0x693\n","stream":"stderr","time":"2018-07-22T21:13:03.399958396Z"}
{"log":"github.com/cosmos/cosmos-sdk/server.startInProcess(0xc42000d500, 0xc42000d540, 0x1d, 0x0, 0x0)\n","stream":"stderr","time":"2018-07-22T21:13:03.399961688Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/server/start.go:98 +0x2fa\n","stream":"stderr","time":"2018-07-22T21:13:03.399965115Z"}
{"log":"github.com/cosmos/cosmos-sdk/server.StartCmd.func1(0xc420a8c000, 0x17076b8, 0x0, 0x0, 0x0, 0x0)\n","stream":"stderr","time":"2018-07-22T21:13:03.399968785Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/server/start.go:38 +0xaa\n","stream":"stderr","time":"2018-07-22T21:13:03.399972103Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra.(*Command).execute(0xc420a8c000, 0x17076b8, 0x0, 0x0, 0xc420a8c000, 0x17076b8)\n","stream":"stderr","time":"2018-07-22T21:13:03.399975599Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra/command.go:698 +0x46d\n","stream":"stderr","time":"2018-07-22T21:13:03.400197806Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420a42fc0, 0xcf9260, 0xc420017e01, 0xc42000d700)\n","stream":"stderr","time":"2018-07-22T21:13:03.40021554Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra/command.go:783 +0x2e4\n","stream":"stderr","time":"2018-07-22T21:13:03.40021954Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420a42fc0, 0xc42000d700, 0xc420017ed8)\n","stream":"stderr","time":"2018-07-22T21:13:03.400222856Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra/command.go:736 +0x2b\n","stream":"stderr","time":"2018-07-22T21:13:03.400226373Z"}
{"log":"github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/libs/cli.Executor.Execute(0xc420a42fc0, 0xf52800, 0x2, 0xc420036870)\n","stream":"stderr","time":"2018-07-22T21:13:03.400230295Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/libs/cli/setup.go:89 +0x4e\n","stream":"stderr","time":"2018-07-22T21:13:03.40023397Z"}
{"log":"main.main()\n","stream":"stderr","time":"2018-07-22T21:13:03.400237642Z"}
{"log":"\u0009/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/cmd/gaiad/main.go:38 +0x214\n","stream":"stderr","time":"2018-07-22T21:13:03.400241057Z"}

Steps to Reproduce

Occurred on Validator node during usual usage (fully synced, on or after height 24570)

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[Ivshti]

Since sdk.Rat is a pointer (

cosmos-sdk/types/rational.go

Line 21 in 8d3d3d3

*big.Rat `json:"rat"`

) shouldn't this line here https://github.com/cosmos/cosmos-sdk/blob/develop/x/gov/tally.go#L58 first check if results[vote.Option] is initialized?

[hendrikhofstadt]

@Ivshti It is not necessary in this case as the result map is being initialized with all possible options:

cosmos-sdk/x/gov/tally.go

Lines 17 to 21 in 032d92f

	results := make(map[VoteOption]sdk.Rat)
	results[OptionYes] = sdk.ZeroRat()
	results[OptionAbstain] = sdk.ZeroRat()
	results[OptionNo] = sdk.ZeroRat()
	results[OptionNoWithVeto] = sdk.ZeroRat()

[Ivshti]

well yeah, but VoteOption is a byte, but from a brief look in the keeper I couldn't figure out how an invalid value can get in there