Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

lightning report "received message larger than max (18552234 vs. 4194304)" during duplicate resolution #56114

Closed
D3Hunter opened this issue Sep 18, 2024 · 2 comments · Fixed by #56771
Closed

lightning report "received message larger than max (18552234 vs. 4194304)" during duplicate resolution #56114

D3Hunter opened this issue Sep 18, 2024 · 2 comments · Fixed by #56771
Assignees
@fishiu
Labels
affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. component/lightning This issue is related to Lightning of TiDB. good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. severity/major type/bug The issue is confirmed as a bug.

Comments

@D3Hunter
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

currently, we use default max-message-size which is 4M, see

tidb/pkg/lightning/backend/local/local.go

Line 144 in bda79ca

func (f *importClientFactoryImpl) makeConn(ctx context.Context, storeID uint64) (*grpc.ClientConn, error) {

2. What did you expect to see? (Required)

no error

3. What did you see instead (Required)

err

4. What is your TiDB version? (Required)

8.1.0
@D3Hunter D3Hunter added type/bug The issue is confirmed as a bug. severity/major component/lightning This issue is related to Lightning of TiDB. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. labels Sep 18, 2024
@ti-chi-bot ti-chi-bot bot added may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 labels Sep 18, 2024
@D3Hunter
Copy link
Contributor Author

tikv send max 4k KV on each response. each KV is about 4K+ in the issue case
https://github.com/tikv/tikv/blob/076b79c54b9e7656f9119865d4ed75bec3bd58d6/src/import/duplicate_detect.rs#L16

@D3Hunter D3Hunter added good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. and removed good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. labels Sep 18, 2024
@fishiu
Copy link
Contributor

fishiu commented Sep 24, 2024

/assign

@fishiu fishiu mentioned this issue Oct 22, 2024
Merged
13 tasks
@lance6716 lance6716 added affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. labels Oct 29, 2024
@ti-chi-bot ti-chi-bot bot added affects-8.5 This bug affects the 8.5.x(LTS) versions. and removed may-affects-6.1 may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.5 labels Oct 29, 2024
@D3Hunter D3Hunter removed affects-5.3 This bug affects 5.3.x versions. affects-5.4 This bug affects the 5.4.x(LTS) versions. labels Nov 5, 2024
@ti-chi-bot ti-chi-bot bot closed this as completed in b522e98 Nov 6, 2024
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this issue Nov 11, 2024
@fishiu @ti-chi-bot
lightning: unlimited grpc MaxCallRecvMsgSize for TikvImporter (pingca…
07e3783 
…p#56771)

close pingcap#56114
@ti-chi-bot ti-chi-bot mentioned this issue Nov 11, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Nov 11, 2024
@ti-chi-bot
lightning: unlimited grpc MaxCallRecvMsgSize for TikvImporter (#56771) (
1e0d6e3 
#57263)

close #56114
@ti-chi-bot ti-chi-bot mentioned this issue Nov 13, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Nov 19, 2024
@ti-chi-bot
lightning: unlimited grpc MaxCallRecvMsgSize for TikvImporter (#56771) (
238f8a8 
#57346)

close #56114
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this issue Dec 3, 2024
@fishiu @ti-chi-bot
lightning: unlimited grpc MaxCallRecvMsgSize for TikvImporter (pingca…
729f136 
…p#56771)

close pingcap#56114
@ti-chi-bot ti-chi-bot mentioned this issue Dec 3, 2024
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Dec 11, 2024
@ti-chi-bot
lightning: unlimited grpc MaxCallRecvMsgSize for TikvImporter (#56771) (
7da37ba 
#57920)

close #56114
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@fishiu fishiu

Labels
affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. component/lightning This issue is related to Lightning of TiDB. good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

lightning: unlimited grpc MaxCallRecvMsgSize for TikvImporter fishiu/tidb
3 participants
@lance6716 @D3Hunter @fishiu