TAP testcase 'simple expires test' sometimes failed (flaky?)

[ligurio]

https://github.com/tarantool/expirationd/runs/3910102548?check_suite_focus=true

TAP version 13
1..10
    # simple expires test
    1..4
    ok - checking start time
    ok - checking task name
    ok - checking restart count
    not ok - Test task executed and moved to archive
make: *** [Makefile:39: test] Error 1
      ---
      filename: ./test.lua
      trace:
      - line: 296
        source: '@./test.lua'
        filename: ./test.lua
        what: Lua
        namewhat: local
        name: fun
        src: ./test.lua
      - line: 218
        source: '@builtin/tap.lua'
        filename: builtin/tap.lua
        what: Lua
        namewhat: method
        name: test
        src: builtin/tap.lua
      - line: 0
        source: '@./test.lua'
        filename: ./test.lua
        what: main
        namewhat: 
        src: ./test.lua
      line: 0
      expected: 7
      got: 8
      ...
    # simple expires test: end
not ok - failed subtests
  ---
  filename: ./test.lua
  trace:
  - line: 245
    source: '@builtin/tap.lua'
    filename: builtin/tap.lua
    what: Lua
    namewhat: method
    name: test
    src: builtin/tap.lua
  - line: 0
    source: '@./test.lua'
    filename: ./test.lua
    what: main
    namewhat: 
    src: ./test.lua
  planned: 4
  failed: 1
  line: 0
  ...
    # execution error test
    1..2
    ok - checking restart count
    ok - Error task executed
    # execution error test: end

Tarantool version 2.8

How-To reproduce:

$ INDEX_TYPE='HASH' ./test.lua

[ligurio]

One more:

INDEX_TYPE='HASH' ./test.lua
TAP version 13
1..10
    # simple expires test
    1..4
    ok - checking start time
    ok - checking task name
    ok - checking restart count
    not ok - Test task executed and moved to archive
      ---
      filename: ./test.lua
make: *** [Makefile:43: test-coverage] Error 1
      trace:
      - line: 296
        source: '@./test.lua'
        filename: ./test.lua
        what: Lua
        namewhat: local
        name: fun
        src: ./test.lua
      - line: 222
        source: '@builtin/tap.lua'
        filename: builtin/tap.lua
        what: Lua
        namewhat: method
        name: test
        src: builtin/tap.lua
      - line: 0
        source: '@./test.lua'
        filename: ./test.lua
        what: main
        namewhat: 
        src: ./test.lua
      line: 0
      expected: 7
      got: 8
      ...
    # simple expires test: end
not ok - failed subtests
  ---
  filename: ./test.lua
  trace:
  - line: 249
    source: '@builtin/tap.lua'
    filename: builtin/tap.lua
    what: Lua
    namewhat: method
    name: test
    src: builtin/tap.lua
  - line: 0
    source: '@./test.lua'
    filename: ./test.lua
    what: main
    namewhat: 
    src: ./test.lua
  planned: 4
  failed: 1
  line: 0
  ...
    # execution error test
    1..2
    ok - checking restart count
    ok - Error task executed
    # execution error test: end
ok - execution error test

https://github.com/tarantool/expirationd/runs/3969338282?check_suite_focus=true

[ylobankov]

It is still reproduced. We've got a failure recently for INDEX_TYPE='TREE' ./test.lua:

INDEX_TYPE='TREE' ./test.lua
TAP version 13
1..10
    # simple expires test
    1..4
    ok - checking start time
    ok - checking task name
    ok - checking restart count
    not ok - Test task executed and moved to archive
      ---
      filename: ./test.lua
      trace:
      - line: 296
        source: '@./test.lua'
        filename: ./test.lua
        what: Lua
        namewhat: local
        name: fun
        src: ./test.lua
      - line: 218
        source: '@builtin/tap.lua'
        filename: builtin/tap.lua
        what: Lua
        namewhat: method
        name: test
        src: builtin/tap.lua
      - line: 0
        source: '@./test.lua'
        filename: ./test.lua
        what: main
        namewhat: 
        src: ./test.lua
      line: 0
      expected: true
      got: false
      ...
    # simple expires test: end
not ok - failed subtests
  ---
  filename: ./test.lua
  trace:
  - line: 245
    source: '@builtin/tap.lua'
    filename: builtin/tap.lua
    what: Lua
    namewhat: method
    name: test
    src: builtin/tap.lua
  - line: 0
    source: '@./test.lua'
    filename: ./test.lua
    what: main
    namewhat: 
    src: ./test.lua
  planned: 4
  failed: 1
  line: 0
  ...
    # execution error test
    1..2
    ok - checking restart count
    ok - Error task executed
make: *** [Makefile:41: test] Error 1
    # execution error test: end

The full log can be found here.

[ligurio]

It is not clear from comment why it is a flaky fail. Flaky test changes status (Pass ⇾ Fail or vice versa) without changes in test environment, system under test and test itself. Integration tests runs on every new Tarantool version, so at least new changes in Tarantool may cause this fail in expirationd.

Yaroslav, is it possible to obtain statistics about number of passed/failed integration tests for expirationd in Tarantool CI?

[Totktonada]

Got it too on unchanged expirationd master in testing against tarantool 2.4, see 132c854. The last release of tarantool 2.4 was in Oct, 2020.