[Bug] Pulsar RocksDB metadata driver causes JVM to crash in unit tests #19250
Comments
lhotari
added a commit
to lhotari/pulsar
that referenced
this issue
Jan 17, 2023
- helps investigate issues when there's a JVM crash such as apache#19250
4 tasks
2 tasks
2 tasks
2 tasks
|
Here's another crash: https://gist.github.com/lhotari/00101de825a7858db17dc3f1d5d88291 . This happened in https://github.com/apache/pulsar/actions/runs/4074068864/jobs/7018986238 |
|
The issue had no activity for 30 days, mark with Stale label. |
|
Another one: in https://github.com/apache/pulsar/actions/runs/4604751972/jobs/8136228746?pr=19945#step:11:1553 |
|
The issue had no activity for 30 days, mark with Stale label. |
|
Another one: full file: https://gist.github.com/lhotari/6f33d841b7aa41f0d602d6494ed585b6 |
|
The issue had no activity for 30 days, mark with Stale label. |
|
Recent crash: https://gist.github.com/lhotari/6c66d0dc55a5104d4364cdbafe26d2e9 |
|
Another recent crash: https://gist.github.com/lhotari/9dc43fae65b12e9b812cdefc845ee96d |
|
Yet another crash: https://gist.github.com/lhotari/177f186d9f73297fbfbce8b026824365 |
4 tasks
|
Yet another crash: https://gist.github.com/lhotari/cb00532b4888acfda9053a5b4d47afee |
|
@merlimat @hangc0276 Do we have any plans to update RocksDB version in Bookkeeper? We are currently on 7.9.2 and it seems that 7.10.2 is the newest 7.x: https://github.com/facebook/rocksdb/releases/tag/v7.10.2 . Newest release is 8.9.1: https://github.com/facebook/rocksdb/releases . |
|
Yet another crash, this time in broker tests. full dump https://gist.github.com/lhotari/f35745b737e7107f13c8becea80f9da4 |
|
A recent crash with Java 17.0.10 (it contains fix for JDK-8314024): https://gist.githubusercontent.com/lhotari/c21fbfdae7f3c59985480397570f9783/raw/7a73251dbd5864ebf6b5045ba1a76c5248f28033/hs_err_pid18047.log |
|
Another recent crash: |
|
@lhotari, thank you for your ongoing efforts in documenting this issue. Unfortunately, the CI artifact has expired: https://github.com/apache/pulsar/actions/runs/7299672614 In RocksDB, attempting to operate on a closed database instance leads to a crash. This behavior is documented in a related issue: facebook/rocksdb#5234 Below is a code snippet to demonstrate how to reproduce the crash: import org.junit.rules.TemporaryFolder;
import org.rocksdb.Options;
import org.rocksdb.RocksDB;
import org.rocksdb.RocksDBException;
import org.rocksdb.RocksIterator;
import java.io.IOException;
class Scratch {
static {
RocksDB.loadLibrary();
}
public static void main(String[] args) throws IOException, RocksDBException {
TemporaryFolder tempFolder = new TemporaryFolder();
tempFolder.create();
String path = tempFolder.newFolder().toString();
Options options = new Options();
options.setCreateIfMissing(true);
RocksDB db = RocksDB.open(options, path);
byte[] key = {0x1};
byte[] value = {0x2};
db.put(key, value);
RocksIterator it = db.newIterator();
db.close();
it.value(); // crashes the JVM
}
} |
@Shawyeok yes, that's most likely the case. It's probably an issue in the shutdown sequence in tests. The scheduled jobs that might access the db should be closed before the db. |
4 tasks
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Search before asking
Version
334c3a5
Minimal reproduce step
none
What did you expect to see?
no JVM crashes
What did you see instead?
There's a crash in calling org.rocksdb.RocksDB.get
Anything else?
It happened in
org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationClientTest.https://ge.apache.org/s/x2u5wjemaiim6/failure?focused-exception-line=0-9#1
Are you willing to submit a PR?
The text was updated successfully, but these errors were encountered: