[Bug]: Standalone SIGABRT: tantivy OpenReadError (FileDoesNotExist("meta.json"))

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master-ci
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): go sdk
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

pr's go-sdk ci test run failed:

• https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20HA%20CI(go-sdk)/detail/PR-39579/3/pipeline/

• server logs:

[2025-01-24T10:45:39Z INFO  tantivy::directory::file_watcher] Meta file "/var/lib/milvus/data/indexnode/index_files/455531513553853686/1/meta.json" was modified |  
-- | --
  |   | [2025-01-24T10:45:39Z INFO  tantivy::directory::file_watcher] Meta file "/var/lib/milvus/data/indexnode/index_files/455531513553853646/1/meta.json" was modified |  
  |   | [2025-01-24T10:45:39Z ERROR tantivy::reader] Error while loading searcher after commit was detected. OpenReadError(FileDoesNotExist("meta.json")) |  
  |   | terminate called after throwing an instance of 'milvus::SegcoreError' |  
  |   | what():   => remove local directory:/var/lib/milvus/data/indexnode/index_files/455531513553853646/1/ failed, error: Directory not empty, files: /var/lib/milvus/data/indexnode/index_files/455531513553853646/1/.tantivy-meta.lock at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:228 |  
  |   |   |  
  |   | SIGABRT: abort |  
  |   | PC=0x7f2df9e419fc m=662 sigcode=18446744073709551610 |  
  |   | signal arrived during cgo execution |  
  |   |   |  
  |   | goroutine 245337 gp=0xc001dbbdc0 m=662 mp=0xc0209df808 [syscall, locked to thread]: |  
  |   | non-Go function |  
  |   | pc=0x7f2df9e419fc |  
  |   | non-Go function |  
  |   | pc=0x7f2df9ded475 |  
  |   | non-Go function |  
  |   | pc=0x7f2df9dd37f2 |  
  |   | non-Go function |  
  |   | pc=0x7f2df9c21b9d |  
  |   | non-Go function |  
  |   | pc=0x7f2df9c2d20b

• server pods:

ms-39579-3-go-pr-etcd-0                                     1/1     Running            0               15m     10.104.29.87    4am-node35   <none>           <none>

2025-01-24T10:47:45Z {container="step-check-status"} ms-39579-3-go-pr-milvus-standalone-5ccf8b647b-dnpqk         1/1     Running            3 (2m ago)      15m     10.104.31.192   4am-node34   <none>           <none>

2025-01-24T10:47:45Z {container="step-check-status"} ms-39579-3-go-pr-minio-85454bb9f-znjpp                      1/1     Running            0               15m     10.104.29.85    4am-node35   <none>           <none>

Loki logs

Expected Behavior

No response

Steps To Reproduce

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

it seems to be a file not found?

[xiaofan-luan]

/assign @SpadeA-Tang
please help on it

[SpadeA-Tang]

It seems there's a race condition between LocalChunkManager::RemoveDir and Tantivy InnerIndexReader::reload.
Tantivy has a watcher on "meta.json" and when the file changed, tantivy will reload the index reader.
So, my conjecture is that:
t1: Tantivy: meta.json was changed -> file watcher detected it, so a index reader reload will be triggered
t2: Segcore: LocalChunkManager::RemoveDir is called concurrently
t3: Tantivy: for index reader reload, it acquires "tantivy-meta.lock" (creating file named "tantivy-meta.lock"). And at this time, "meta.json" is deleted by LocalChunkManager, so it reports logs:
[2025-01-24T10:45:39Z ERROR tantivy::reader] Error while loading searcher after commit was detected. OpenReadError(FileDoesNotExist("meta.json"))
t4: Segcore: return the error "Directory not empty" due to concurrently file creation ("tantivy-meta.lock") I guess, which triggers the panic.

If my conjuecture is right, the problem goes to why LocalChunkManager chooses to remove the directory while the index reader is still alive.

[xiaofan-luan]

@SpadeA-Tang
maybe related to #39471

[SpadeA-Tang]

what's the commit hash of the panic cluster? @ThreadDao

[SpadeA-Tang]

I just noticed that the panic info is after the fix #39471. I think the root cause should be similar with that.

[ThreadDao]

@SpadeA-Tang I will try again after this PR is merged #39253

[ThreadDao]

It seems that it has not reappeared