gazctl: shard pruning should consider all hints #350
Conversation
If a shard is in a persistent state where it immediately becomes FAILED after recovery, it's very possible for the oldest recovered backup hints to actually be _newer_ than primary hints. Then, if we prune without considering the primary hints, the shard is unable to recover from its log. Fortunately this is easily remedied -- delete the primary hints while leaving backup recovery hints in place -- but better to not do it in the first place.
| if hints != nil && len(hints.LiveNodes) > 0 { | ||
| foldHintsIntoSegments(*hints, logSegmentSets) | ||
| } else { | ||
| skipRecoveryLogs[recoveryLog] = true |
There was a problem hiding this comment.
Not something that's changed here, but this just made me curious. If there's non-nil hints, but they don't have any LiveNodes, could that indicate that all the files have been deleted, and thus we should prune that journal still? Not proposing that we handle that corner case, but more just checking my understanding.
There was a problem hiding this comment.
Hmm, that sounds conceptually right but I don't recall if hinted log replay can function if there are literally no LiveNodes. Possibly it handles it by jumping to the head of the log, or possibly it errors out because it wouldn't have an (offset, sequenceNo, checksum) tuple to resume reading from.
But if all data is also deleted, that can also be problematic because there must be written & persisted data that's old enough for the reader's offset to be eligible to jump to the log head.
(To my knowledge there's no actual scenario where LiveNodes is intentionally empty.)
If a shard is in a persistent state where it immediately becomes FAILED after recovery, it's very possible for the oldest recovered backup hints to actually be newer than primary hints.
Then, if we prune without considering the primary hints, the shard is unable to recover from its log.
Fortunately this is easily remedied -- delete the primary hints while leaving backup recovery hints in place -- but better to not do it in the first place.
This change is