influx db revoke crashes vault

[sandromehic]

Describe the bug
I'm using vault with nomad to run influxdb job and provide credentials to various other jobs (telegraf, etc.) through vault influxdb secrets engine. When I stop the nomad job, it revokes the given lease, after which the vault crashes.

I'm not sure if this is vault or nomad bug, but either way, I find my vault instance restarted and sealed because of this. I have started to see this behaviour after updating vault from 1.1.0 to 1.1.2 and nomad from 0.8.7 to 0.9.1.

Vault Logs
It looks like vault revokes the influxdb lease correctly ( database/creds/monitor ), then fails to revoke two postgres leases (one is the role that is no longer existing, another a role that has some table ownerships, so it can't be revoked), after which it crashes. From the stacktrace it seems like it has some issue with influxdb lease, even tough it logged a successful revoke.

vault[29570]: 2019-05-15T09:54:27.196+0200 [INFO]  expiration: revoked lease: lease_id=database/creds/monitor/7qXaVgsw4GMZ2vsNhyDUPORi
vault[29570]: 2019-05-15T09:54:27.201+0200 [ERROR] expiration: failed to revoke lease: lease_id=database/creds/leaser_admin/H3JlBA2Pr2J4VcnrpCxswgnK error="failed to revoke entry: resp: (*logical.Res
vault[29570]: 2019-05-15T09:54:27.201+0200 [ERROR] expiration: failed to revoke lease: lease_id=database/creds/kms_admin/pi9DKryO8S1JyWzwCI0FVepO error="failed to revoke entry: resp: (*logical.Respon
vault[29570]: panic: runtime error: invalid memory address or nil pointer dereference
vault[29570]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x13f6a37]
vault[29570]: goroutine 3692 [running]:
vault[29570]: github.com/hashicorp/vault/vendor/github.com/influxdata/influxdb/client/v2.(*Response).Error(0x0, 0x0, 0xc0010a31c0)
vault[29570]:         /gopath/src/github.com/hashicorp/vault/vendor/github.com/influxdata/influxdb/client/v2/client.go:477 +0x37
vault[29570]: github.com/hashicorp/vault/plugins/database/influxdb.(*Influxdb).RevokeUser(0xc001581cc0, 0x2f1a100, 0xc001244de0, 0xc0018967e0, 0x6b, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
vault[29570]:         /gopath/src/github.com/hashicorp/vault/plugins/database/influxdb/influxdb.go:194 +0x50f
vault[29570]: github.com/hashicorp/vault/builtin/logical/database/dbplugin.(*DatabaseErrorSanitizerMiddleware).RevokeUser(0xc0001e9e00, 0x2f1a100, 0xc001244de0, 0xc0018967e0, 0x6b, 0x0, 0x0, 0x0, 0x0
vault[29570]:         /gopath/src/github.com/hashicorp/vault/builtin/logical/database/dbplugin/databasemiddleware.go:236 +0xa3
vault[29570]: github.com/hashicorp/vault/builtin/logical/database/dbplugin.(*databaseMetricsMiddleware).RevokeUser(0xc001581ce0, 0x2f1a100, 0xc001244de0, 0xc0018967e0, 0x6b, 0x0, 0x0, 0x0, 0x0, 0x0, 
vault[29570]:         /gopath/src/github.com/hashicorp/vault/builtin/logical/database/dbplugin/databasemiddleware.go:148 +0x1fc
vault[29570]: github.com/hashicorp/vault/builtin/logical/database.(*databaseBackend).secretCredsRevoke.func1(0x2f1a100, 0xc001244de0, 0xc000f52a00, 0xc00074efd0, 0x0, 0x0, 0x0)
vault[29570]:         /gopath/src/github.com/hashicorp/vault/builtin/logical/database/secret_creds.go:139 +0x47b
vault[29570]: github.com/hashicorp/vault/logical/framework.(*Secret).HandleRevoke(0xc0001e9aa0, 0x2f1a100, 0xc001244de0, 0xc000f52a00, 0xc00078d738, 0x1, 0x28)
vault[29570]:         /gopath/src/github.com/hashicorp/vault/logical/framework/secret.go:87 +0x94
vault[29570]: github.com/hashicorp/vault/logical/framework.(*Backend).handleRevokeRenew(0xc00075cdd0, 0x2f1a100, 0xc001244de0, 0xc000f52a00, 0x393761376be6417b, 0x3530632d37303862, 0xc0008c98a0)
vault[29570]:         /gopath/src/github.com/hashicorp/vault/logical/framework/backend.go:415 +0x1da
vault[29570]: github.com/hashicorp/vault/logical/framework.(*Backend).HandleRequest(0xc00075cdd0, 0x2f1a100, 0xc001244de0, 0xc000f52a00, 0x0, 0x0, 0x0)
vault[29570]:         /gopath/src/github.com/hashicorp/vault/logical/framework/backend.go:170 +0x845
vault[29570]: github.com/hashicorp/vault/vault.(*Router).routeCommon(0xc000795c80, 0x2f1a100, 0xc001244de0, 0xc000f52a00, 0x40c100, 0x0, 0x2af0000, 0x0, 0x0)
vault[29570]:         /gopath/src/github.com/hashicorp/vault/vault/router.go:665 +0x926
vault[29570]: github.com/hashicorp/vault/vault.(*Router).Route(0xc000795c80, 0x2f1a100, 0xc001244de0, 0xc000f52a00, 0xc001244de0, 0xc0008c9d48, 0x1f89071)
vault[29570]:         /gopath/src/github.com/hashicorp/vault/vault/router.go:465 +0x4e
vault[29570]: github.com/hashicorp/vault/vault.(*ExpirationManager).revokeEntry(0xc00078c480, 0x2f1a100, 0xc00123c0c0, 0xc0010f2640, 0x2f, 0xc0010f2640)
vault[29570]:         /gopath/src/github.com/hashicorp/vault/vault/expiration.go:1363 +0x284
vault[29570]: github.com/hashicorp/vault/vault.(*ExpirationManager).revokeCommon(0xc00078c480, 0x2f1a100, 0xc00123c0c0, 0xc001570540, 0x2f, 0xbf2f0f89203a0000, 0x0, 0x0)
vault[29570]:         /gopath/src/github.com/hashicorp/vault/vault/expiration.go:626 +0x4b3
vault[29570]: github.com/hashicorp/vault/vault.(*ExpirationManager).Revoke(0xc00078c480, 0x2f1a100, 0xc00123c0c0, 0xc001570540, 0x2f, 0x0, 0x0)
vault[29570]:         /gopath/src/github.com/hashicorp/vault/vault/expiration.go:573 +0x11c
vault[29570]: github.com/hashicorp/vault/vault.expireLeaseStrategyRevoke(0x2f1a040, 0xc000724040, 0xc00078c480, 0xc0010f34a0)
vault[29570]:         /gopath/src/github.com/hashicorp/vault/vault/expiration.go:122 +0x2c4
vault[29570]: github.com/hashicorp/vault/vault.(*ExpirationManager).updatePendingInternal.func1()
vault[29570]:         /gopath/src/github.com/hashicorp/vault/vault/expiration.go:1323 +0x52
vault[29570]: created by time.goFunc
vault[29570]:         /goroot/src/time/sleep.go:172 +0x44
systemd[1]: vault.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
systemd[1]: vault.service: Failed with result 'exit-code'.

[catsby]

Hello - when you mention "When I stop the nomad job, it revokes the given lease, after which the vault crashes.", could you clarify what specifically is revoking the lease? Is the Nomad job doing it at termination?

[catsby]

Opened #8282 to patch this

[catsby]

#8282 was merged, thanks for reporting this!