[Bug]: Insert and index building are slow

[bigsheeper]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:master&2.5&2.4
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

When the number of segments is high, some operations in DataCoord metadata hold locks for extended periods, leading to lock starvation and subsequently slowing down insertion and index building.

Expected Behavior

No response

Steps To Reproduce

Milvus Log

No response

Anything else?

No response

[dosubot]

I found a similar closed issue titled "[Bug]: [benchmark] slow insertion with low datanodes workload" ^[1]. However, no solution was found in the comments of that issue.

Regarding the current issue, the SetStoredIndexFileSizeMetric function in the Milvus codebase uses a read lock (RLock) to ensure thread-safe access to the indexMeta and collections data structures while calculating the stored index file size metric. This locking mechanism allows multiple readers to access the data concurrently but can lead to lock starvation if write operations are blocked for extended periods ^[2][3].

To address the issue of lock starvation, consider reviewing the operations that hold locks for extended periods and explore potential optimizations or restructuring to minimize lock contention. Additionally, examining the locking patterns and ensuring that locks are held for the shortest duration necessary can help mitigate the problem.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[bigsheeper]

related to #37630