Clarification on watch when server (or store) is partitioned

[embano1]

After reading up on some corner cases with etcd client behavior (connections, streams, etc.) I came across the following option that has been introduced in etcd a while ago:

// WithRequireLeader requires client requests to only succeed
// when the cluster has a leader.
func WithRequireLeader(ctx context.Context) context.Context {...}

(Source)

From the official etcd docs on client behavior:

Client-side keepalive ping still does not reason about network partitions. Streaming request may get stuck with a partitioned node. Advanced health checking service need to be implemented to understand the cluster membership (see etcd#8673 for more detail).
Source: clientv3-grpc1.23: Balancer Limitation

I do not see this option being used in the API server when Watch() is established.

Is there code in the API server that deals with these cases (etcd/API server node paritioned) and disconnects clients (REST, SDK) appropriately? Or are we at risk of having dangling (hanging) consumer-side client watches due to not using this option in the etcd client?

Related discussions:

clientv3: clarify "WithRequireLeader" for network partition
Network-partition aware health service

cc/ @dims @jingyih @jpbetz @sttts

[embano1]

Opened on k/k
kubernetes/kubernetes#89488