test: Timed out waiting for Vinyl memory quota

[Totktonada]

Reported by @kyukhin, found by tarantool's integration tests in CI.

expirationd 1.1.1-44-g838c2d1.

The list of failed tests:

• expirationd_stats.engine:"vinyl".index_type:"TREE".test_stats_expired_count
• iterator_type.engine:"vinyl".index_type:"TREE".test_tree_index_all
• iterator_type.engine:"vinyl".index_type:"TREE".test_tree_index_eq
• iterator_type.engine:"vinyl".index_type:"TREE".test_tree_index_gt
• iterator_type.engine:"vinyl".index_type:"TREE".test_tree_index_req
• iterator_type.engine:"vinyl".index_type:"TREE".test_tree_index_ge
• iterator_type.engine:"vinyl".index_type:"TREE".test_tree_index_lt
• iterator_type.engine:"vinyl".index_type:"TREE".test_tree_index_le
• process_while.engine:"vinyl".index_type:"TREE".test_tree_index
• start_key.engine:"vinyl".index_type:"TREE".test_tree_index

The error reported:

Timed out waiting for Vinyl memory quota

Full logs: logs.txt.

(The job itself is here, but it will expire after several months.)

The job is run on 2.10.0-beta2-22-g1b46becbe with one irrelevant commit upward. I reproduced it locally on 2.10.0-beta2-5-gdc19be406. It seems, it didn't behave in such way before, but maybe it is not something that we can highlight as a regression in tarantool. At least, if I do the following:

diff --git a/test/helper.lua b/test/helper.lua
index ed2dcc1..4a5821f 100644
--- a/test/helper.lua
+++ b/test/helper.lua
@@ -200,7 +200,7 @@ t.before_suite(function()
         memtx_dir  = t.datadir,
         vinyl_dir  = t.datadir,
         -- vinyl_memory is changed to test test_mvcc_vinyl_tx_conflict.
-        vinyl_memory = 1024,
+        --vinyl_memory = 1024,
     }
 
     local tree_code = [[function(tuple)
diff --git a/test/unit/atomic_iteration_test.lua b/test/unit/atomic_iteration_test.lua
index fec7a78..6a58888 100644
--- a/test/unit/atomic_iteration_test.lua
+++ b/test/unit/atomic_iteration_test.lua
@@ -132,6 +132,7 @@ end
 
 -- it's not check tarantool or vinyl as engine
 -- just check expirationd task continue work after conflicts
+--[[
 function g.test_mvcc_vinyl_tx_conflict(cg)
     t.skip_if(cg.params.engine ~= 'vinyl', 'Unsupported engine')
 
@@ -169,6 +170,7 @@ function g.test_mvcc_vinyl_tx_conflict(cg)
     t.assert_equals(box.stat.vinyl().tx.transactions, 0)
     task:kill()
 end
+]]--
 
 -- Create a task that use atomic_iteration and check that task is gone after
 -- kill.

The tests passed. Maybe it is just too tight quota.

(However it would be interesting to bisect it by tarantool commits.)

[Totktonada]

Hm. Got different results on the same version: 2.10.0-beta2-5-gdc19be406. When I use tarantool installed using emerge (on Gentoo), I got fail. But for tarantool built from git sources directly I got successful testing.

Failed configuration

$ tarantool --version
Tarantool 2.10.0-beta2-5-gdc19be406
Target: Linux-x86_64-Debug
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr -DENABLE_BACKTRACE=yes
Compiler: /usr/bin/x86_64-pc-linux-gnu-gcc /usr/bin/x86_64-pc-linux-gnu-g++
C_FLAGS:-march=native -O2 -ggdb -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -mavx -std=c11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-gnu-alignof-expression -fno-gnu89-inline -Wno-cast-function-type -Werror
CXX_FLAGS:-march=native -O2 -ggdb -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -mavx -std=c++11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-invalid-offsetof -Wno-gnu-alignof-expression -Wno-cast-function-type -Werror

Ebuild is here.

$ grep -R dev-db/tarantool /etc/portage/
/etc/portage/package.accept_keywords/tarantool:dev-db/tarantool **
/etc/portage/package.use/tarantool:dev-db/tarantool debug
/etc/portage/package.env/debug.env.list:dev-db/tarantool debug.conf
$ cat /etc/portage/env/debug.conf 
# > If nostrip is in your default FEATURES, splitdebug won't do anything! 
# https://wiki.gentoo.org/wiki/Debugging
CFLAGS="${CFLAGS} -ggdb"
CXXFLAGS="${CXXFLAGS} -ggdb"
FEATURES="${FEATURES} splitdebug compressdebug installsources -nostrip"
USE="debug"

USE flags: backtrace debug system-libcurl system-libyaml system-zstd -feedback-daemon -gcov -gprof -systemd -test CPU_FLAGS_X86="avx sse2".

Successful configuration

Tarantool 2.10.0-beta2-5-gdc19be406
Target: Linux-x86_64-Debug
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr/local -DENABLE_BACKTRACE=ON
Compiler: /usr/bin/cc /usr/bin/c++
C_FLAGS: -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-gnu-alignof-expression -fno-gnu89-inline -Wno-cast-function-type -Werror 
CXX_FLAGS: -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c++11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-invalid-offsetof -Wno-gnu-alignof-expression -Wno-cast-function-type -Werror 

Build command:

$ cmake . -DCMAKE_BUILD_TYPE=Debug -DENABLE_BACKTRACE=ON -DENABLE_DIST=ON -DENABLE_FEEDBACK_DAEMON=OFF -DENABLE_BUNDLED_LIBCURL=OFF && make -j

A side problem

Don't know, whether it is relevant, but I have different results for the same tarantool version (and the version installed from ebuild is bad again) on one of LuaJIT's tests:

$ ./src/tarantool third_party/luajit/test/LuaJIT-tests/lib/ffi/bit64.lua
$ echo $?
0
$ tarantool third_party/luajit/test/LuaJIT-tests/lib/ffi/bit64.lua
LuajitError: third_party/luajit/test/LuaJIT-tests/lib/ffi/bit64.lua:49: assertion failed!
fatal error, exiting the event loop

Thoughts

Anyway, we meet it in the integration testing of tarantool, so my configuration is not the only one, where the problem (with expirationd tests) is reproduced.

[Totktonada]

I see a correlation with tarantool/tarantool#6787. When I build tarantool to reproduce the luajit's issue, I also see the vinyl quota error on expirationd tests.

Anyway, I propose to just increase the quota in expirationd tests. Let's re-verify whether the difference in vinyl behaviour is result of difference in luajit behaviour in the scope of tarantool's issue.

[Totktonada]

Regarding the fail in tarantool's integration testing on Ubuntu 20.04 (Focal). AFAIS, Kirill restarted the testing on the same commit (without any change) and all tests passed. It seems the vinyl's difference in behaviour is unstable (disregarding whether it is related to luajit's bit ops problem).

I downloaded Ubuntu package from the integration testing (the second run, which was successful):
tarantool-ubuntu-focal-267f988ab4ed43646e18a8107ccb7c8cba2eeec5.zip, run Ubuntu in docker and made several runs of the testing. Kinda this:

$ docker run -it ubuntu:focal
<..next steps are in the docker container..>
<..install tarantool deps and unzip..>
$ unzip tarantool-ubuntu-focal-267f988ab4ed43646e18a8107ccb7c8cba2eeec5.zip
$ apt-get install tarantool_2.10.0~beta2.37.dev-1_amd64.deb tarantool-common_2.10.0~beta2.37.dev-1_all.deb tarantool-dev_2.10.0~beta2.37.dev-1_amd64.deb
$ git clone https://github.com/tarantool/expirationd.git
$ cd expirationd
$ make deps
$ for i in $(seq 1 100); do .rocks/bin/luatest -v --coverage || break; echo -e "\\n\\n====\\nSuccess: ${i}\\n====\\n\\n"; done

It fails on the 9th iteration.

I also tried it without --coverage and it fails too (on 3rd iteration). So luacov is not relevant here.

However the test case re luajit's bit ops is not reproduced on this tarantool build. Maybe vinyl's difference/unstability in behaviour may have different causes.

[oleg-jukovec]

Anyway, I propose to just increase the quota in expirationd tests. Let's re-verify whether the difference in vinyl behaviour is result of difference in luajit behaviour in the scope of tarantool's issue.

The quota won't be needed after #120