Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

TestLabelerRuleTTL is flaky #8687

Closed
okJiang opened this issue Oct 10, 2024 · 4 comments · Fixed by #8723
Closed

TestLabelerRuleTTL is flaky #8687

okJiang opened this issue Oct 10, 2024 · 4 comments · Fixed by #8723
Labels
type/ci The issue is related to CI.

Comments

@okJiang
Copy link
Member

okJiang commented Oct 10, 2024

Flaky Test

Which jobs are failing


[FAIL]  pkg/schedule/labeler TestLabelerRuleTTL
=== RUN   TestLabelerRuleTTL
==================
WARNING: DATA RACE
Read at 0x00c000433d10 by goroutine [135](https://github.com/tikv/pd/actions/runs/11271034543/job/31343067657?pr=8685#step:4:136):
  runtime.mapaccess1_faststr()
      /opt/hostedtoolcache/go/1.23.2/x64/src/runtime/map_faststr.go:13 +0x0
  github.com/tikv/pd/pkg/schedule/labeler.checkRuleInMemoryAndStorage()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:405 +0x90
  github.com/tikv/pd/pkg/schedule/labeler.TestLabelerRuleTTL()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:393 +0x1891
  github.com/pingcap/failpoint.parseTerm()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:149 +0x233
  github.com/pingcap/failpoint.parse()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:126 +0xa5
  github.com/pingcap/failpoint.newTerms()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/terms.go:98 +0x3e
  github.com/pingcap/failpoint.(*Failpoint).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoint.go:54 +0x3e
  github.com/pingcap/failpoint.(*Failpoints).Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:105 +0x296
  github.com/pingcap/failpoint.Enable()
      /home/runner/go/pkg/mod/github.com/pingcap/failpoint@v0.0.0-20210918120811-547c13e3eb00/failpoints.go:225 +0x174a
  github.com/tikv/pd/pkg/schedule/labeler.TestLabelerRuleTTL()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:383 +0x174b
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1743 +0x44

Previous write at 0x00c000433d10 by goroutine [137](https://github.com/tikv/pd/actions/runs/11271034543/job/31343067657?pr=8685#step:4:138):
  runtime.mapdelete_faststr()
      /opt/hostedtoolcache/go/1.23.2/x64/src/runtime/map_faststr.go:321 +0x0
  github.com/tikv/pd/pkg/schedule/labeler.(*RegionLabeler).DeleteLabelRuleLocked()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:265 +0x1e8
  github.com/tikv/pd/pkg/schedule/labeler.(*RegionLabeler).checkAndClearExpiredLabels()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:92 +0x417
  github.com/tikv/pd/pkg/schedule/labeler.(*RegionLabeler).doGC()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:67 +0x165
  github.com/tikv/pd/pkg/schedule/labeler.NewRegionLabeler.gowrap1()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:55 +0x44

Goroutine 135 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1743 +0x825
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:2168 +0x85
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1690 +0x226
  testing.runTests()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:2166 +0x8be
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:2034 +0xf17
  main.main()
      _testmain.go:73 +0x164

Goroutine 137 (running) created at:
  github.com/tikv/pd/pkg/schedule/labeler.NewRegionLabeler()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler.go:55 +0x297
  github.com/tikv/pd/pkg/schedule/labeler.TestLabelerRuleTTL()
      /home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:344 +0x48e
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.23.2/x64/src/testing/testing.go:1743 +0x44
==================
    labeler_test.go:405: 
        	Error Trace:	/home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:405
        	            				/home/runner/work/pd/pd/pkg/schedule/labeler/labeler_test.go:393
        	Error:      	Not equal: 
        	            	expected: true
        	            	actual  : false
        	Test:       	TestLabelerRuleTTL
    testing.go:[139](https://github.com/tikv/pd/actions/runs/11271034543/job/31343067657?pr=8685#step:4:140)9: race detected during execution of test
--- FAIL: TestLabelerRuleTTL (0.10s)

CI link

https://github.com/tikv/pd/actions/runs/11271034543/job/31343067657?pr=8685

Reason for failure (if possible)

Anything else

@okJiang okJiang added the type/ci The issue is related to CI. label Oct 10, 2024
@okJiang
Copy link
Member Author

okJiang commented Oct 11, 2024

@okJiang
Copy link
Member Author

okJiang commented Oct 11, 2024

@okJiang
Copy link
Member Author

okJiang commented Oct 15, 2024

@okJiang
Copy link
Member Author

okJiang commented Oct 16, 2024

labeler, err := NewRegionLabeler(context.Background(), store, time.Millisecond*10)

gcInterval(10ms) is so short that it has been GC before L393(check the status) and after L387(update its ttl).

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
type/ci The issue is related to CI.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant