/pki/tidy returns errors indicating that certificates are nil on some pki mounts

[protochron]

Environment:

• Vault Version: 0.7.2
• Operating System/Architecture: Ubuntu 14.04.5

Vault Config File:

{
  "backend": {
    "file": {
      "path": "/opt/vault/data"
    }
  },
  "listener": {
    "tcp": {
      "tls_key_file": "/etc/ssl/private/vault.key",
      "tls_cert_file": "/etc/ssl/certs/vault.crt",
      "address": "0.0.0.0:8200"
    }
  }
}

Startup Log Output:
Partial output:

2018/03/20 15:40:36.159619 [INFO ] rollback: starting rollback manager
2018/03/20 15:40:36.168195 [INFO ] expiration: restoring leases
2018/03/20 15:44:26.374946 [INFO ] expire: leases restored: restored_lease_count=769535
2018/03/20 15:44:26.381820 [INFO ] core: post-unseal setup complete

Expected Behavior:
Running curl -X POST --header '...' https://${vault-server-address}:8200/v1/pki/tidy -d '{"safety_buffer": "1440h", "tidy_cert_store": true}' should successfully clean up old certificates.

Actual Behavior:

It looks like there's a single pki mount that is unable to tidy up all of the certificates. We keep seeing an error

{"errors":["1 error occurred:\n\n* certificate entry for serial 55:93:c6:ba:32:07:92:94:84:52:09:75:94:1d:de:55:6e:49:8f:b6 is nil"]}

Having read the code, I can't figure out why we'd be seeing an issue where certificates are nil, especially when you can read the cert with the cli or by curling the pki mount after the tidy operation fails. All I can think of is that Vault is not actually able to read all of the certificates from the store or the large amount of leases that we have is somehow interfering.

Steps to Reproduce:
See above

Important Factoids:

References:

[jefferai]

I must admit I'm pretty stumped -- as you saw in the code, this should only happen if the actual underlying bytes for the value at that location in storage are empty. I think probably in this case we could warn and continue rather than error out, but if this function is seeing nil bytes, and fetching is seeing the full certs...I'm not sure. What storage are you using? Any chance that you're getting intermittent errors from it?

[jefferai]

See #4214 - if you have better ideas for how to handle this, without understanding why your entry is nil in the first place, let me know.

[jefferai]

Closing due to #4214, if you prefer another approach I can re-open.

[protochron]

@jefferai sounds good! If I think of any other way to handle it then I'll let you know