Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

unexpected election when one of followers is network partition with leader #9020

Closed
lhy1024 opened this issue Jan 26, 2025 · 0 comments · Fixed by #9021
Closed

unexpected election when one of followers is network partition with leader #9020

lhy1024 opened this issue Jan 26, 2025 · 0 comments · Fixed by #9021
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. severity/major type/bug The issue is confirmed as a bug.

Comments

@lhy1024
Copy link
Contributor

lhy1024 commented Jan 26, 2025

Enhancement Task

An unexpected election occurs when one of the followers experiences a network partition from the leader.

If the etcd client is using a follower endpoint and encounters a failure when attempting to save a timestamp, the allocator_manager will reset the leader.

[2025/01/21 20:39:49.149 +08:00] [WARN] [lease.go:187] ["lease keep alive failed"] [purpose="leader election"] [start=2025/01/21 20:39:46.149 +08:00] [error="context deadline exceeded"]
[2025/01/21 20:39:53.032 +08:00] [WARN] [etcd_kv.go:180] ["txn runs too slow"] [response=null] [cost=5.721552451s] [error="rpc error: code = Unavailable desc = error reading from server: read tcp [10.200.26.212:34736](http://10.200.26.212:34736/)->[10.200.27.59:2379](http://10.200.27.59:2379/): read: connection timed out"]
[2025/01/21 20:39:53.032 +08:00] [WARN] [tso.go:333] ["save timestamp failed"] [] [timestamp-path=timestamp] [error="rpc error: code = Unavailable desc = error reading from server: read tcp [10.200.26.212:34736](http://10.200.26.212:34736/)->[10.200.27.59:2379](http://10.200.27.59:2379/): read: connection timed out"]
[2025/01/21 20:39:53.032 +08:00] [WARN] [allocator_manager.go:289] ["failed to update allocator's timestamp"] [] [name=tc-pd-1] [error="rpc error: code = Unavailable desc = error reading from server: read tcp [10.200.26.212:34736](http://10.200.26.212:34736/)->[10.200.27.59:2379](http://10.200.27.59:2379/): read: connection timed out"]

In fact, manual retry is needed, next request succeeds with the new endpoint, according to etcd-io/etcd#8711

@lhy1024 lhy1024 added the type/enhancement The issue or PR belongs to an enhancement. label Jan 26, 2025
@ti-chi-bot ti-chi-bot bot closed this as completed in #9021 Feb 5, 2025
ti-chi-bot bot added a commit that referenced this issue Feb 5, 2025
close #9020

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@lhy1024 lhy1024 added type/bug The issue is confirmed as a bug. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. severity/major and removed type/enhancement The issue or PR belongs to an enhancement. labels Feb 11, 2025
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 11, 2025
close tikv#9020

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 11, 2025
close tikv#9020

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 11, 2025
close tikv#9020

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Feb 11, 2025
close tikv#9020

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot bot pushed a commit that referenced this issue Feb 13, 2025
close #9020

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <liuhanyang@pingcap.com>
Co-authored-by: lhy1024 <admin@liudos.us>
ti-chi-bot bot pushed a commit that referenced this issue Feb 13, 2025
close #9020

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <liuhanyang@pingcap.com>
Co-authored-by: lhy1024 <admin@liudos.us>
ti-chi-bot bot added a commit that referenced this issue Feb 13, 2025
close #9020

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <liuhanyang@pingcap.com>
Co-authored-by: lhy1024 <admin@liudos.us>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue Feb 13, 2025
close #9020

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <admin@liudos.us>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot pushed a commit that referenced this issue Feb 13, 2025
close #9020

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <liuhanyang@pingcap.com>
Co-authored-by: lhy1024 <admin@liudos.us>
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant