Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[CRASH] dictRehash on eviction #876

Open
keithchew opened this issue Oct 13, 2024 · 0 comments
Open

[CRASH] dictRehash on eviction #876

keithchew opened this issue Oct 13, 2024 · 0 comments

Comments

@keithchew
Copy link

keithchew commented Oct 13, 2024

Testing on async_flash branch, but bug is likely present earlier. When memory is under pressure, a crash happens during the eviction process:

=== KEYDB BUG REPORT START: Cut & paste starting from here ===
7:101:S 13 Oct 2024 04:33:16.151 # KeyDB 6.3.4 crashed by signal: 11, si_code: 1
7:101:S 13 Oct 2024 04:33:16.151 # Accessing address: 0x8065700082
7:101:S 13 Oct 2024 04:33:16.151 # Crashed running the instruction at: 0x55a96cc7b836

------ STACK TRACE ------
EIP:
/opt/KeyDB/bin/keydb-server *:6379(dictRehash+0x86) [0x55a96cc7b836]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f7de4eb2420]
/opt/KeyDB/bin/keydb-server *:6379(dictRehash+0x86) [0x55a96cc7b836]
/opt/KeyDB/bin/keydb-server *:6379(dictGetSomeKeys+0x81) [0x55a96cc7e261]
/opt/KeyDB/bin/keydb-server *:6379(evictionPoolPopulate(int, redisDb*, bool, evictionPoolEntry*)+0x9e) [0x55a96cd7d1fe]
/opt/KeyDB/bin/keydb-server *:6379(performEvictions(bool)+0xf3f) [0x55a96cd7f4ff]
/opt/KeyDB/bin/keydb-server *:6379(processCommand(client*, int)+0xf5c) [0x55a96cc8b8dc]
/opt/KeyDB/bin/keydb-server *:6379(processCommandAndResetClient(client*, int)+0x69) [0x55a96cca9cc9]
/opt/KeyDB/bin/keydb-server *:6379(processInputBuffer(client*, bool, int)+0x254) [0x55a96ccafe94]
/opt/KeyDB/bin/keydb-server *:6379(processClients()+0xe6) [0x55a96ccb0036]
/opt/KeyDB/bin/keydb-server *:6379(beforeSleep(aeEventLoop*)+0x2db) [0x55a96cc8701b]
/opt/KeyDB/bin/keydb-server *:6379(aeProcessEvents+0x400) [0x55a96cc78680]
/opt/KeyDB/bin/keydb-server *:6379(aeMain+0x3e) [0x55a96cc78e6e]
/opt/KeyDB/bin/keydb-server *:6379(workerThreadMain(void*)+0x12b) [0x55a96cc923db]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f7de4ea6609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f7de4dc9133]

Note that the server has this option:

activerehashing no

So, both code flows in active rehash and async rehash have been eliminated. I wonder if there could be a fundamental issue with the main dict logic.

Will try to narrow this down further, but thought I did post this first just in case there are others who have come across the same issue.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant