Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

make Consul/Vault fingerprints reloadable, not periodic #24049

Open
tgross opened this issue Sep 23, 2024 · 0 comments
Open

make Consul/Vault fingerprints reloadable, not periodic #24049

tgross opened this issue Sep 23, 2024 · 0 comments

Comments

@tgross
Copy link
Member

tgross commented Sep 23, 2024

Nomad fingerprinters can be static, periodic, or "reloadable" (update on SIGHUP). The only two periodic fingerprints are for Consul and Vault. One could also argue that these two fingerprints are unusual in that they fingerprint something other than the host's "own" environment, and this can cause flapping fingerprints if the availability of those services isn't 100%.

So in Nomad 1.4.0 we shipped #14673 which changed the fingerprinters so that we wouldn't change the fingerprint if the up/down status of Consul or Vault changed. But this still leaves the rest of the fingerprint like the version. For Vault, this can make Vault cluster upgrades perilous for large Nomad clusters. If your Vault cluster is behind a DNS name, your Nomad clients will fingerprint different Vault IPs behind that name as you upgrade, causing the fingerprint to change each time. Multiply this by many many clients and you have a storm of Node.Register requests and the resulting Raft writes and evals to handle.

Instead, let's move the power to decide when to re-fingerprint Vault and Consul in the cluster administrator's hands. Make these fingerprinters reloadable rather than periodic. We can combine this work with #24048 to make reloading configuration easier as well.

(See also #23526 #18327 for other examples of fingerprints that could be reloadable.)

# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

No branches or pull requests

1 participant