TestSplitKeyspaceGroup is unstable

[lhy1024]

Flaky Test

Which jobs are failing

1. TestKeyspaceGroupTestSuite/TestSplitKeyspaceGroup

2023-11-16T02:38:53.5396266Z --- FAIL: TestKeyspaceGroupTestSuite (14.23s)
2023-11-16T02:38:53.5396899Z     --- FAIL: TestKeyspaceGroupTestSuite/TestSplitKeyspaceGroup (7.93s)
2023-11-16T02:38:53.5397445Z         testutil.go:172: 
2023-11-16T02:38:53.5398211Z             	Error Trace:	/home/runner/work/pd/pd/tests/server/apiv2/handlers/testutil.go:172
2023-11-16T02:38:53.5399581Z             	            				/home/runner/work/pd/pd/tests/server/apiv2/handlers/tso_keyspace_group_test.go:170
2023-11-16T02:38:53.5400259Z             	Error:      	Not equal: 
2023-11-16T02:38:53.5400716Z             	            	expected: 200
2023-11-16T02:38:53.5401171Z             	            	actual  : 503
2023-11-16T02:38:53.5401784Z             	Test:       	TestKeyspaceGroupTestSuite/TestSplitKeyspaceGroup
2023-11-16T02:38:53.5402210Z FAIL

https://github.com/lhy1024/pd/actions/runs/6885616709/job/18730031255

2. TestTSOKeyspaceGroupManager/TestTSOKeyspaceGroupSplit

2023-11-15T21:59:05.8905539Z     testutil.go:172: 
2023-11-15T21:59:05.8906055Z         	Error Trace:	/home/runner/work/pd/pd/tests/server/apiv2/handlers/testutil.go:172
2023-11-15T21:59:05.8906941Z         	            				/home/runner/work/pd/pd/tests/integrations/mcs/tso/keyspace_group_manager_test.go:274
2023-11-15T21:59:05.8907520Z         	            				/opt/hostedtoolcache/go/1.21.3/x64/src/runtime/asm_amd64.s:1650
2023-11-15T21:59:05.8907689Z         	Error:      	Not equal: 
2023-11-15T21:59:05.8907899Z         	            	expected: 200
2023-11-15T21:59:05.8908106Z         	            	actual  : 500
2023-11-15T21:59:05.8908468Z         	Test:       	TestTSOKeyspaceGroupManager/TestTSOKeyspaceGroupSplit

https://github.com/lhy1024/pd/actions/runs/6883321480/job/18723641024

CI link

Reason for failure (if possible)

Anything else

[lhy1024]

it print a lot of log

2023-12-08T10:40:42.6376898Z [2023/12/08 10:33:47.238 +00:00] [ERROR] [tso_dispatcher.go:498] ["[tso] getTS error"] [dc-location=global] [stream-addr=http://127.0.0.1:35421] [error="[PD:client:ErrClientGetTSO]get TSO failed, after processing requests"]
2023-12-08T10:40:42.6377270Z [2023/12/08 10:33:47.238 +00:00] [INFO] [tso_dispatcher.go:441] ["[tso] tso stream is not ready"] [dc=global]

https://github.com/tikv/pd/actions/runs/7140266211/job/19445195668