You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Nomad fingerprinters can be static, periodic, or "reloadable" (update on SIGHUP). The only two periodic fingerprints are for Consul and Vault. One could also argue that these two fingerprints are unusual in that they fingerprint something other than the host's "own" environment, and this can cause flapping fingerprints if the availability of those services isn't 100%.
So in Nomad 1.4.0 we shipped #14673 which changed the fingerprinters so that we wouldn't change the fingerprint if the up/down status of Consul or Vault changed. But this still leaves the rest of the fingerprint like the version. For Vault, this can make Vault cluster upgrades perilous for large Nomad clusters. If your Vault cluster is behind a DNS name, your Nomad clients will fingerprint different Vault IPs behind that name as you upgrade, causing the fingerprint to change each time. Multiply this by many many clients and you have a storm of Node.Register requests and the resulting Raft writes and evals to handle.
Instead, let's move the power to decide when to re-fingerprint Vault and Consul in the cluster administrator's hands. Make these fingerprinters reloadable rather than periodic. We can combine this work with #24048 to make reloading configuration easier as well.
(See also #23526#18327 for other examples of fingerprints that could be reloadable.)
The text was updated successfully, but these errors were encountered:
Nomad fingerprinters can be static, periodic, or "reloadable" (update on SIGHUP). The only two periodic fingerprints are for Consul and Vault. One could also argue that these two fingerprints are unusual in that they fingerprint something other than the host's "own" environment, and this can cause flapping fingerprints if the availability of those services isn't 100%.
So in Nomad 1.4.0 we shipped #14673 which changed the fingerprinters so that we wouldn't change the fingerprint if the up/down status of Consul or Vault changed. But this still leaves the rest of the fingerprint like the version. For Vault, this can make Vault cluster upgrades perilous for large Nomad clusters. If your Vault cluster is behind a DNS name, your Nomad clients will fingerprint different Vault IPs behind that name as you upgrade, causing the fingerprint to change each time. Multiply this by many many clients and you have a storm of
Node.Register
requests and the resulting Raft writes and evals to handle.Instead, let's move the power to decide when to re-fingerprint Vault and Consul in the cluster administrator's hands. Make these fingerprinters reloadable rather than periodic. We can combine this work with #24048 to make reloading configuration easier as well.
(See also #23526 #18327 for other examples of fingerprints that could be reloadable.)
The text was updated successfully, but these errors were encountered: